开启一个单独进程运行某个python函数 作者:xieaotian发表于2008-08-21 15:20:31
Description:
This is a simple function that runs another function in a different process by forking a new process which runs the function and waiting for the result in the parent. This can be useful for releasing resources used by the function such as memory.
Source: Text Source
#!/usr/bin/env python
import os, cPickle
def run_in_separate_process(f, *args, **kwds):
pread, pwrite = os.pipe()
pid = os.fork()
if pid > 0:
os.close(pwrite)
with os.fdopen(pread, 'rb') as f:
status, result = cPickle.load(f)
os.waitpid(pid, 0)
if status == 0:
return result
else:
raise result
else:
os.close(pread)
try:
result = f(*args, **kwds)
status = 0
except Exception, exc:
result = exc
status = 1
with os.fdopen(pwrite, 'wb') as f:
try:
cPickle.dump((status,result), f, cPickle.HIGHEST_PROTOCOL)
except cPickle.PicklingError, exc:
cPickle.dump((2,exc), f, cPickle.HIGHEST_PROTOCOL)
f.close()
os._exit(0)
#an example of use
def treble(x):
return 3 * x
def main():
#calling directly
print treble(4)
#calling in separate process
print run_in_separate_process(treble, 4)
Discussion:
Frequently, one might write code such like:
for x in alist: result = do_work(params)
where do_work consumes a lot of memory which is not useful for the lifetime of the program. I actually wrote this function when I was doing large amounts of computations (using numpy) and using large number of temporary arrays. A good way of actually reclaiming the memory is forking a child process, doing the computation in the child process, and returning the results to the parent. This pattern was mentioned in [1].
The function run_in_separate_process encodes this pattern. It handles exceptions as well, partially. Basically the child process returns a status code as well as a result. If the status is 0, then the function returned successfully and its result is returned. If the status is 1, then the function raised an exception, which will be raised in the parent. If the status is 2, then the function has returned successfully but the result is not picklable, an exception is raised.
Exceptions such as SystemExit and KeyboardInterrupt in the child are not checked. As the function stands, it will result in an EOFError in the parent.
This function works on Linux and Mac OS X. It should work on all "reasonable" systems.
Thanks to Alex Martelli for confirming the basic approach and supplying the code for pipe communication. His message and the (short) thread [2] contain more explanation.