15

I have some subprocesses (using multiprocessing) and when they stop, each of them need do some final work. Something like the following, which did not work though... 

import multiprocessing
import atexit

def final():
    print "final work"

def worker():
    print 'Doing some work'
    atexit.register(final)

if __name__ == '__main__':
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

So how could I do this?

jkdev
  • 11,360
  • 15
  • 54
  • 77
rongdong.bai
  • 471
  • 1
  • 6
  • 16
  • Check http://stackoverflow.com/questions/1231599/python-multiprocessing-exit-elegantly-how which is similar to what you are trying to accomplish. – Selçuk Cihan Dec 29 '15 at 07:52

2 Answers2

14

Processes started via multiprocessing do not exit normally and atexit functions get never called. You have to run cleanup stuff manually via a try/finally block, for example:

def worker():
    try:
        print('Doing some work')
    finally:
        final()

Or, if you really want to use atexit (see below for the drawbacks):

def worker():
    try:
        print('Doing some work')
    finally:
        atexit._run_exitfuncs()

Why don't subprocesses exit normally?

Technically, because they quit via os._exit(), skipping any cleanup job (including atexit functions, __del__() and weakref finalizers).

The reason behind this decision is that cleanup stuff risks being executed at the wrong time, in the wrong process. Most of the code that uses atexit (or other mechanisms) expects the code to be executed only once and only when the main interpreter exits (for example, one may use atexit to remove a temporary directory). By executing atexit functions every time a subprocess quits, you break this expectation and bad things may happen.

So, even though you can run atexit functions via atexit._run_exitfuncs(), avoid doing so, both because the function is undocumented and because you may not be the only user of atexit (other third party libraries may use it, and you may introduce bugs in their code).

Andrea Corbellini
  • 17,339
  • 3
  • 53
  • 69
  • 9
    This was an absurd decision. atexit functions dont expect to be called from "main". They expect to be called in the process in which they were registered. so atexit merely needs to store the pid of registration in order to know whether they can be called. Any modules which use atexit are now rendered useless inside a multiprocessing job... forcing me to use Popen instead and pickle their own state. I actually forked multiprocessing just to put the proper cleanups back in. – Erik Aronesty Dec 15 '17 at 14:15
  • 1
    I do agree with you very much Erik. I put together a better replacement for atexit, maybe you can take a look? https://github.com/kuralabs/multiexit – Havok Apr 07 '18 at 05:20
0

You may use this atexit replacement that is multiprocess aware.

It will call only the functions registered by the process itself and not any other inherited my a parent process or anything.

https://github.com/kuralabs/multiexit

Havok
  • 5,776
  • 1
  • 35
  • 44