Uwsgi 进程内存一直飙升(占到几个 G) - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
请不要在回答技术问题时复制粘贴 AI 生成的内容
vegetableChick
V2EX    程序员

Uwsgi 进程内存一直飙升(占到几个 G)

  •  
  •   vegetableChick 2021-03-19 16:35:32 +08:00 2363 次点击
    这是一个创建于 1677 天前的主题,其中的信息可能已经有所发展或是发生改变。

    没找到原因, 大佬们帮忙看看

    项目使用 Django + uwsgi + Nginx

    uwsgi

    [uwsgi] pythOnpath=/usr/local/server chdir=/home/server env=DJANGO_SETTINGS_MODULE=conf.settings module=server.wsgi master=True pidfile=logs/server.pid vacuum=True max-requests=1000 enable-threads=true processes = 4 threads=8 listen=1024 daemOnize=logs/wsgi.log http=0.0.0.0:16020 buffer-size=32000 

    有一个从服务端获取 excel 文件的请求,请求全量 40 多万条数 响应时间过长, 然后浏览器得到Nginx 504 但是一个子进程的内存一直彪升

    top -p 866 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 866 soe 20 0 7059m 5.3g 5740 S 100.8 33.9 4:17.34 uwsgi --ini /home/smb/work/soe_server 

    基本代码

    from openpyxl.writer.excel import save_virtual_workbook from django.http import HttpResponse ... class ExcelTableObj(object): def __init__(self, file_name=None): if file_name: self.file_name = file_name self.wb = load_workbook(file_name) else: self.wb = Workbook() def create_new_sheet(self, title='Sheet1'): new_ws = self.wb.create_sheet(title=title) def write_to_sheet(self, sheetname, datas, filename): ws = self.wb[sheetname] for data in datas: ws.append(data) self.wb.save(filename) def update_sheet_name(self, sheetname): ws = self.wb.active ws.title = sheetname def append_data_to_sheet(self, sheetname, data): ws = self.wb[sheetname] ws.append(data) def save_file(self, file_name): self.wb.save(file_name) self.wb.close() def get_upload_file_data(self, name=None): if name: ws = self.wb.get_sheet_by_name(name) else: ws = self.wb.worksheets[0] rows = ws.max_row cols = ws.max_column file_data = [] fields = [] for i in range(1, cols+1): cell = ws.cell(row=1, column=i) if cell.value: fields.append(cell.value.lower().strip()) for row in range(2, rows + 1): row_data = {} for j in range(len(fields)): value = ws.cell(row=row, column=j+1).value if value: row_data[fields[j]] = str(value).strip() if row_data: file_data.append(row_data) return file_data def get_sheet_maxrow(self, name): ws = self.wb.get_sheet_by_name(name) rows = ws.max_row return rows def _get_download_data(datas): for data in queryset : ... item = [str(data.account_id), ILLEGAL_CHARACTERS_RE.sub(r'', data.account_name) if data.account_name else data.account_name, type, fb_aac_conf.FB_ACCOUNT_STATUS[data.account_status], data.submitter, data.submit_time, data.confirmor, data.confirm_time, fb_aac_conf.BATCH_STATUS[data.status], data.reason, data.entity_name, data.payment_name, data.sale, data.ae_note, urgent ] yield item queryset = MyModel.objects.filter(...) # about `450k` rows datas = _get_download_data(queryset) excel = ExcelTableObj() excel.update_sheet_name(sheetname) excel.append_data_to_sheet(sheetname, title) excel.write_to_sheet(sheetname, datas, filename) excel.save_file(filename) respOnse= HttpResponse(save_virtual_workbook(excel.wb), content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet') response['Content-Disposition'] = 'attachment; filename={}'.format(filename) 

    请大家帮忙分洗一下问题原因。感谢

    13 条回复    2021-03-20 00:01:17 +08:00
    ch2
        1
    ch2  
       2021-03-19 17:49:01 +08:00
    openpyxl 导出 excel 非常耗性能的,40W 条卡住很正常
    BeautifulSoap
        2
    BeautifulSoap  
       2021-03-19 17:52:56 +08:00
    openpyxl 别说导出了,大点的 excel 读一下就要十来秒
    chenqh
        3
    chenqh  
       2021-03-19 18:04:02 +08:00
    导出 excel 一般不是生成一个地址,然后前端自己去下载吗?
    no1xsyzy
        4
    no1xsyzy  
       2021-03-19 18:17:51 +08:00
    ……nginx 504 了以后子进程没退出吧(
    根据官方的 benchmark 放大一下,40w 条需要大约 4 分钟?
    ryd994
        5
    ryd994  
       2021-03-19 18:25:18 +08:00 via Android
    为啥导出 xls,如果只是数据的话 CSV 不就行了
    excel 可以直接打开
    vegetableChick
        6
    vegetableChick  
    OP
       2021-03-19 18:33:59 +08:00
    @ch2 卡住可以理解, 但是为什么内存占用会一直往上升啊
    vegetableChick
        7
    vegetableChick  
    OP
       2021-03-19 18:34:26 +08:00
    @BeautifulSoap 。。为什么内存会吃这么多 而且一直彪
    vegetableChick
        8
    vegetableChick  
    OP
       2021-03-19 18:35:04 +08:00
    @no1xsyzy 为什么进程内存一致彪升啊
    superrichman
        9
    superrichman  
       2021-03-19 21:17:01 +08:00
    这几十万的数据在内存里被你复制了不知道多少次,用完了也不主动去释放,openpyxl 也没开 write_only 模式,内存不炸有鬼了,自己 debug 一步一步看吧


    还有

    ```
    def _get_download_data(datas):
    for data in queryset :

    ...

    queryset = MyModel.objects.filter(...)
    datas = _get_download_data(queryset)
    ```

    你确定函数是这么传参数的?
    BeautifulSoap
        10
    BeautifulSoap  
       2021-03-19 21:47:58 +08:00
    不是 ls 的内容我还没注意到,作为犯过同样错误的人提醒下 lz,data 的复数形式依旧是 data,不是 datas
    ericls
        11
    ericls  
       2021-03-19 21:54:16 +08:00 via iPhone
    @BeautifulSoap data 的单数是 datum.
    BeautifulSoap
        12
    BeautifulSoap  
       2021-03-19 22:17:39 +08:00
    @ericls 语言是一直在动态变化的,datum 实际用得并不多,data 同时做单数已经成了英语世界一个大家都接受的用法了
    FindHao
        13
    FindHao  
       2021-03-20 00:01:17 +08:00 via Android
    我也有个辣鸡代码是这样的,最后的解决方案是暴力写了 crontab 重启自己的服务
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     3309 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 25ms UTC 11:05 PVG 19:05 LAX 04:05 JFK 07:05
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86