python多进程下运行类方法的坑
多进程运行类方法:
- 通常情况下我们习惯运行多进程的方法是导入multiprocessing包,进而创建多进程并运行start方法进行启动。这个操作直接运行某个函数是没有错的,例如:
def like_cheese(): pid = os.getpid() print(pid) for i in range(10): print(i) time.sleep(2) if i == 5: cmd = "kill -9 %d" % int(pid) os.system(cmd) if __name__ == __main__: pid = os.getpid() print(pid) while True: prpcess = multiprocessing.Process(target=like_cheese) prpcess.start() prpcess.join() if not prpcess.is_alive(): os.system(rpython3 ./xiao.py)
这种情况下是可以正常运行的。但是下面这个例子就不能正常运行了:
class FbPageRollback: def __init__(self): pass def run(self): for i in range(10): print(i) if __name__ == __main__: rollback_obj = FbPageRollback() multiprocessing.Process(target=rollback_obj.run).start()
此时报如下错误:
Traceback (most recent call last): File "F:/PythonProject/globalegrow/spider/fb_page_rollback.py", line 620, in <module> multiprocessing.Process(target=rollback_obj.run).start() File "C:UsersxiaomengAppDataLocalProgramsPythonPython36libmultiprocessingprocess.py", line 105, in start self._popen = self._Popen(self) File "C:UsersxiaomengAppDataLocalProgramsPythonPython36libmultiprocessingcontext.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:UsersxiaomengAppDataLocalProgramsPythonPython36libmultiprocessingcontext.py", line 322, in _Popen return Popen(process_obj) File "C:UsersxiaomengAppDataLocalProgramsPythonPython36libmultiprocessingpopen_spawn_win32.py", line 65, in __init__ reduction.dump(process_obj, to_child) File "C:UsersxiaomengAppDataLocalProgramsPythonPython36libmultiprocessing eduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cant pickle module objects
查询得知是在序列化和反序列化的过程中出现错误,也就是说此Python多进程不能直接在用上述的start直接开启多进程运行类方法,那有其它办法可以解决吗?答案是肯定的。我们可以在类的外部写一个函数,通过运行外部函数来运行类方法,其次通过下面这个方法来启动类方法。
p = multiprocessing.Process(target=self.multi_process) p.run()
感兴趣的可以了解一些多进程启动的两种方法,start与run,会发现其实start最终还是通过run启动多进程。