Python 多进程+协程的问题 - V2EX
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
bfpiaoran

Python 多进程+协程的问题

  •  
  •   bfpiaoran Aug 22, 2018 4207 views
    This topic created in 2820 days ago, the information mentioned may be changed o developed.

    最近在研究子域名爆破 试用了 dnspython 这个第三方库
    但是踩到了很多坑,程序算是能跑起来了,但是感觉速度不是十分理想

    因为要做扫描的列表 于是使用了 multiprocessing.Manager 来管理进程之间的共享数据

    但是 gevent 如果打了猴子补丁的话会有冲突,于是代码改成 monkey.patch_all(thread=False, socket=False, select=False) 但是跑起来速度很慢 那种慢看起来是 dns 查询的慢,想请问有没有什么优化方法啊

    11 replies    2018-08-23 19:49:37 +08:00
    lfzyx
        1
    lfzyx  
       Aug 22, 2018
    用 asyncio 啊
    Trim21
        2
    Trim21  
       Aug 22, 2018 via Android
    asyncio aiodns
    bfpiaoran
        3
    bfpiaoran  
    OP
       Aug 22, 2018
    @Trim21 我试一试
    matrace
        4
    matrace  
       Aug 22, 2018 via iPhone
    你把 socket 设置为 false,就相当于是多进程了
    bfpiaoran
        5
    bfpiaoran  
    OP
       Aug 22, 2018
    @w9ay 但是 socket True 会有冲突 emm 再仔细研究研究
    lolizeppelin
        6
    lolizeppelin  
       Aug 22, 2018 via Android
    抄 openstack 的多进城 service 启动部分代码
    好好看看么写多进程代码的
    bfpiaoran
        7
    bfpiaoran  
    OP
       Aug 22, 2018 via Android
    我去抄一抄
    panyanyany
        8
    panyanyany  
       Aug 22, 2018
    插句题外话,凡是涉及多进程 /多线程的,我都首先考虑 Golang ……
    lieh222
        9
    lieh222  
       Aug 23, 2018
    multiprocessing.Manager 貌似是通过 multiprocessing.pipe 实现的,原理跟 rpc 一样,性能缺失低,不如用 mmap 吧
    whoami9894
        10
    whoami9894  
       Aug 23, 2018
    这种网络 I/O 为主的,多线程就够用了吧

    #9
    manager 是本地 server 提供的共享,它可以在子进程间共享;不是通过 pipe,pipe 只能在父子进程间共享
    bfpiaoran
        11
    bfpiaoran  
    OP
       Aug 23, 2018
    @whoami9894 刚试了下 asyncio 那速度是真的爽啊 。。。。
    About     Help     Advertise     Blog     API     FAQ     Solana     5808 Online   Highest 6679       Select Language
    创意工作者们的社区
    World is poweed by solitude
    VERSION: 3.9.8.5 75ms UTC 02:47 PVG 10:47 LAX 19:47 JFK 22:47
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86