Digital Ocean 新加坡最近总是 out of memory 崩溃 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
kosenpai
V2EX    问与答

Digital Ocean 新加坡最近总是 out of memory 崩溃

  •  
  •   kosenpai 2016-05-17 22:20:20 +08:00 2736 次点击
    这是一个创建于 3457 天前的主题,其中的信息可能已经有所发展或是发生改变。

    自用爬梯,一直稳定运行了一年多,最近 VPS 运行半天就完全死掉, ssh 也登不上去。看 syslog 发现崩溃之前都是 out of memory 。但是看内存记录发现崩溃前内存占用完全正常,不正常的部分是 CPU 占用在崩溃前异常高。下面是崩溃的一段,感觉就是这样被搞死的。请问大家这是怎么了?

    May 17 06:16:01 PIMSserver kernel: [18538.680826] sh invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0 May 17 06:16:01 PIMSserver kernel: [18538.680852] sh cpuset=/ mems_allowed=0 May 17 06:16:01 PIMSserver kernel: [18538.680875] CPU: 0 PID: 19134 Comm: sh Tainted: P OX 3.13.0-48-generic #80-Ubuntu May 17 06:16:01 PIMSserver kernel: [18538.680878] Hardware name: DigitalOcean Droplet, BIOS 20160425 04/25/2016 May 17 06:16:01 PIMSserver kernel: [18538.680882] 0000000000000000 ffff880018913978 ffffffff81721506 ffff880019dc8000 May 17 06:16:01 PIMSserver kernel: [18538.680896] ffff880018913a00 ffffffff8171bdc1 ffffffff81067996 ffff8800189139d8 May 17 06:16:01 PIMSserver kernel: [18538.680900] ffffffff810c75dc 0000000000000001 ffff88001fffae28 0000000000000000 May 17 06:16:01 PIMSserver kernel: [18538.680904] Call Trace: May 17 06:16:01 PIMSserver kernel: [18538.680979] [<ffffffff81721506>] dump_stack+0x45/0x56 May 17 06:16:01 PIMSserver kernel: [18538.680991] [<ffffffff8171bdc1>] dump_header+0x7f/0x1f1 May 17 06:16:01 PIMSserver kernel: [18538.681025] [<ffffffff81067996>] ? put_online_cpus+0x56/0x80 May 17 06:16:01 PIMSserver kernel: [18538.681045] [<ffffffff810c75dc>] ? rcu_oom_notify+0xcc/0xf0 May 17 06:16:01 PIMSserver kernel: [18538.681067] [<ffffffff811529be>] oom_kill_process+0x1ce/0x330 May 17 06:16:01 PIMSserver kernel: [18538.681097] [<ffffffff812d7225>] ? security_capable_noaudit+0x15/0x20 May 17 06:16:01 PIMSserver kernel: [18538.681102] [<ffffffff811530f4>] out_of_memory+0x414/0x450 May 17 06:16:01 PIMSserver kernel: [18538.681112] [<ffffffff81159460>] __alloc_pages_nodemask+0xa60/0xb80 May 17 06:16:01 PIMSserver kernel: [18538.681135] [<ffffffff81197ad3>] alloc_pages_current+0xa3/0x160 May 17 06:16:01 PIMSserver kernel: [18538.681146] [<ffffffff81179064>] __pmd_alloc+0x24/0x110 May 17 06:16:01 PIMSserver kernel: [18538.681152] [<ffffffff8117a825>] handle_mm_fault+0xd75/0xf10 May 17 06:16:01 PIMSserver kernel: [18538.681183] [<ffffffff811787f3>] ? follow_page_mask+0x73/0x5b0 May 17 06:16:01 PIMSserver kernel: [18538.681190] [<ffffffff811787be>] ? follow_page_mask+0x3e/0x5b0 May 17 06:16:01 PIMSserver kernel: [18538.681195] [<ffffffff8117ab26>] __get_user_pages+0x166/0x5e0 May 17 06:16:01 PIMSserver kernel: [18538.681199] [<ffffffff8117aff2>] get_user_pages+0x52/0x60 May 17 06:16:01 PIMSserver kernel: [18538.681214] [<ffffffff811c4246>] copy_strings.isra.17+0x256/0x2e0 May 17 06:16:01 PIMSserver kernel: [18538.681219] [<ffffffff811c4304>] copy_strings_kernel+0x34/0x40 May 17 06:16:01 PIMSserver kernel: [18538.681224] [<ffffffff811c586c>] do_execve_common.isra.22+0x4fc/0x7e0 May 17 06:16:01 PIMSserver kernel: [18538.681229] [<ffffffff811c5de6>] SyS_execve+0x36/0x50 May 17 06:16:01 PIMSserver kernel: [18538.681239] [<ffffffff81732569>] stub_execve+0x69/0xa0 May 17 06:16:01 PIMSserver kernel: [18538.681242] Mem-Info: May 17 06:16:01 PIMSserver kernel: [18538.681250] Node 0 DMA per-cpu: May 17 06:16:01 PIMSserver kernel: [18538.681257] CPU 0: hi: 0, btch: 1 usd: 0 May 17 06:16:01 PIMSserver kernel: [18538.681259] Node 0 DMA32 per-cpu: May 17 06:16:01 PIMSserver kernel: [18538.681262] CPU 0: hi: 186, btch: 31 usd: 160 May 17 06:16:01 PIMSserver kernel: [18538.681268] active_anon:866 inactive_anon:890 isolated_anon:0 May 17 06:16:01 PIMSserver kernel: [18538.681268] active_file:37 inactive_file:46 isolated_file:0 May 17 06:16:01 PIMSserver kernel: [18538.681268] unevictable:0 dirty:1 writeback:0 unstable:0 May 17 06:16:01 PIMSserver kernel: [18538.681268] free:1471 slab_reclaimable:2333 slab_unreclaimable:4106 May 17 06:16:01 PIMSserver kernel: [18538.681268] mapped:24 shmem:5 pagetables:1581 bounce:0 May 17 06:16:01 PIMSserver kernel: [18538.681268] free_cma:0 May 17 06:16:01 PIMSserver kernel: [18538.681274] Node 0 DMA free:1928kB min:88kB low:108kB high:132kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:96kB slab_unreclaimable:404kB kernel_stack:8kB pagetables:144kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6801 all_unreclaimable? yes May 17 06:16:01 PIMSserver kernel: [18538.681288] lowmem_reserve[]: 0 471 471 471 May 17 06:16:01 PIMSserver kernel: [18538.681296] Node 0 DMA32 free:3956kB min:2732kB low:3412kB high:4096kB active_anon:3464kB inactive_anon:3560kB active_file:132kB inactive_file:172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:507896kB managed:485892kB mlocked:0kB dirty:4kB writeback:0kB mapped:80kB shmem:20kB slab_reclaimable:9236kB slab_unreclaimable:16020kB kernel_stack:992kB pagetables:6180kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11036 all_unreclaimable? yes May 17 06:16:01 PIMSserver kernel: [18538.681303] lowmem_reserve[]: 0 0 0 0 May 17 06:16:01 PIMSserver kernel: [18538.681307] Node 0 DMA: 20*4kB (U) 29*8kB (UM) 35*16kB (UM) 15*32kB (UMR) 5*64kB (UMR) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB May 17 06:16:01 PIMSserver kernel: [18538.681334] Node 0 DMA32: 537*4kB (UEM) 208*8kB (UM) 1*16kB (M) 0*32kB 0*64kB 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3956kB May 17 06:16:01 PIMSserver kernel: [18538.681353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB May 17 06:16:01 PIMSserver kernel: [18538.681355] 554 total pagecache pages May 17 06:16:01 PIMSserver kernel: [18538.681357] 453 pages in swap cache May 17 06:16:01 PIMSserver kernel: [18538.681360] Swap cache stats: add 526366, delete 525913, find 169308/273582 May 17 06:16:01 PIMSserver kernel: [18538.681362] Free swap = 4110736kB May 17 06:16:01 PIMSserver kernel: [18538.681363] Total swap = 4194300kB May 17 06:16:01 PIMSserver kernel: [18538.681365] 130972 pages RAM May 17 06:16:01 PIMSserver kernel: [18538.681367] 0 pages HighMem/MovableOnly 
    7 条回复    2016-05-20 09:24:05 +08:00
    meiriyitie
        1
    meiriyitie  
       2016-05-17 22:27:51 +08:00
    重装系统大法? 也许 do 想让你升级到 10 刀哈哈
    tvallday
        2
    tvallday  
       2016-05-17 22:49:46 +08:00 via iPad
    Swap space 太小了?整个大的?其实 10 刀的应该就不会出现 memory 不够的情况了,节省很多无谓的时间。
    ksky
        3
    ksky  
       2016-05-17 22:56:13 +08:00
    装 Mysql 了吗。内存不够不是 apache 就是 mysql 的问题。显然是有人在弄你的网站。
    kosenpai
        4
    kosenpai  
    OP
       2016-05-18 08:57:39 +08:00
    @meiriyitie 现在正在试新建一个,看看是不是还是有问题。
    kosenpai
        5
    kosenpai  
    OP
       2016-05-18 09:04:49 +08:00
    @tvallday 应该不是内存的问题,从图里可以看到崩溃的时候 CPU 爆炸,但是内存还是正常的。写了 tickets 问 DO ,但是他们也不给建议……


    kosenpai
        6
    kosenpai  
    OP
       2016-05-18 09:08:14 +08:00
    @ksky mysql 和 apache 是刚装的,之前只有 PPTP 和 ss 。崩溃在 mysql 安装之前就开始发生了。现在每天都崩溃一次,弄的我也崩溃了。不过谢谢提醒,也许有人通过 PPTP 来攻击我呢?我去查查 log 。
    kosenpai
        7
    kosenpai  
    OP
       2016-05-20 09:24:05 +08:00
    我查了 log 发现每条都和 Supervisor 有关,我都忘了什么时候装了这个。现在卸载掉之后稳定运行一天了。再看一段时间有没有问题。
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     1159 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 27ms UTC 17:44 PVG 01:44 LAX 09:44 JFK 12:44
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86