请教 concurrent.futures 的多进程 ProcessPoolExecutor, sumit 提交任务,如何正确把字典作为参数传入? - V2EX
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
pppguest3962

请教 concurrent.futures 的多进程 ProcessPoolExecutor, sumit 提交任务,如何正确把字典作为参数传入?

  •  
  •   pppguest3962 Sep 22, 2020 1977 views
    This topic created in 2058 days ag, the information mentioned may be changed or developed.

    只是个伪代码例子, 原代码量大,只能弄了个结构上一致的简单的例子,
    procFun 在没套入到多进程的时候,工作得很好,是一个闭包,不会与其它什么的造成干涉。。。

    BABTaskDict = {'TkNum':32, 'TkString':'test String'} def procFun(taskInfo): taskNumber = taskInfo.get('TkNum') taskString = taskInfo.get('TkString') ... ... pass with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor: to_do = [] #executor.submit 返回 future 实例 future = executor.submit(procFun, BABTaskDict ) # 方式 1 # 方式 2 future = executor.submit(procFun, *BABTaskDict ) # 方式 3 future = executor.submit(procFun, **BABTaskDict ) to_do.append(future) 

    用了方式 1 的方法,会有如下提示:

    Traceback (most recent call last): File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 234, in _feed obj = _ForkingPickler.dumps(obj) File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects 

    用了方式 2 的方法,future.result()有如下:

     File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 432, in result return self.__get_result() File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 384, in __get_result raise self._exception TypeError: procFun() takes 1 positional argument but 17 were given 

    用了方式 3 的方法,与方式 1 一样,
    是不是姿势不对?
    没其它选择,info 字典内容必须传进去,散写逐个参数,非常麻烦的。。。

    3 replies    2020-09-22 19:41:07 +08:00
    xiaolinjia
        1
    xiaolinjia  
       Sep 22, 2020
    方式 1 的写法没问题,报那错是因为多进程在 windows 下要用 pickle 模块将环境的所有对象序列化,再 copy 一份给其他进程。所以是 procFun 函数里面有什么对象是不能 pickle 序列化吧。
    VYSE
        2
    VYSE  
       Sep 22, 2020
    taskInfo 这个 dict 里存了 thread lock, 所以不能序列化传递到子进程
    pppguest3962
        3
    pppguest3962  
    OP
       Sep 22, 2020
    醍醐灌顶,谢谢两位,找到原因了,多线程 /进程环境不同,我在在 taskInfo 里,就把 queen 队列的地址传进去使用了,
    ltQueen = queue.LifoQueue()
    taskInfo = {'ltQueen': ltQueen}
    其实很希望能在子进程里,能做 put 任务到主线程 /进程的队列 ltQueen 的操作,我看看试试能不能解决这个问题。。。,如果不成功,还是会发帖请教~~
    About     Help     Advertise     Blog     API     FAQ     Solana     3297 Online   Highest 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 52ms UTC 14:08 PVG 22:08 LAX 07:08 JFK 10:08
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86