求解, 今日热榜数据是怎么批量爬取的?动态网页用 Puppeteer 不仅性能慢还超时报错

问题

各位老哥求解, 如能帮忙解决问题, 口令红包私人感谢.

• 单独解决 私发:￥ 8.88 ￥ 16.88
• 多人解决 群发:￥ 20
• 感谢方式: 支付宝口令红包

谢谢各位.

1. 今日热榜数据是怎么批量爬取的

好奇今日热榜这些热榜站是如何进行批量爬取的

2. 怎么爬取动态网页

cheerio 抓取静态网页, Puppeteer 批量爬取, 性能好慢, 频繁超时报错找不到原因

page.goto 设置多长时间都超时, 30s 60s 90s

本地 windows 运行又没有问题, 远程服务器 vps 小鸡动不动报超时

报错

数据处理过程错误: TimeoutError: Navigation timeout of 30000 ms exceeded at new Deferred (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/util/Deferred.js:59:34) at Deferred.create (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/util/Deferred.js:21:16) at new LifecycleWatcher (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/LifecycleWatcher.js:66:60) at CdpFrame.goto (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Frame.js:143:29) at CdpFrame.<anonymous> (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/util/decorators.js:98:27) at CdpPage.goto (/usr/local/node_modules/puppeteer-core/lib/cjs/puppeteer/api/Page.js:588:43) at fetchData (file:///usr/local/script/%E8%B4%A2%E7%BB%8F%E7%83%AD%E6%A6%9C.js:51:18) at async executeProcess (file:///usr/local/script/%E8%B4%A2%E7%BB%8F%E7%83%AD%E6%A6%9C.js:108:24) 

代码逻辑

async function fetchData(page, name, url, hrefSelector) { const maxRetries = 3; // Maximum number of attempts let attempts = 0; while (attempts < maxRetries) { try { attempts++; await page.goto(url, { timeout: 1000 * 30 }); await page.waitForSelector(hrefSelector, { timeout: 1000 * 30 }); const results = await page.$$eval(hrefSelector, anchors => anchors.map(anchor => ({ href: anchor.href, text: anchor.textContent.trim() })) ); const trade_date = getCurrentDateTime(); return { name, news: results, trade_date }; } catch (error) { if (attempts < maxRetries) { console.warn(`获取数据报错 ${url}. Retry attempt ${attempts}...`); await delay(2000); // Wait for 2 seconds before retrying } else { console.error(`获取数据报错 ${url} after ${attempts} attempts:`, error); throw error; } } } } 

使用插件

• xvfb
• puppeteer-extra
• puppeteer-extra-plugin-stealth
• puppeteer-extra-plugin-anonymize-ua

import puppeteer from 'puppeteer-extra'; import StealthPlugin from 'puppeteer-extra-plugin-stealth'; import AnonymizeUaPlugin from 'puppeteer-extra-plugin-anonymize-ua' puppeteer.use(StealthPlugin()); puppeteer.use(AnonymizeUaPlugin()); 

puppeteer 启动参数

// 启动浏览器和页面 const browser = await puppeteer.launch({ args: [ "--disable-setuid-sandbox", "--no-sandbox", "--disable-gpu", "--no-first-run", "--disable-dev-shm-usage", "--single-process" ], headless: true }); console.log('启动浏览器 √'); const page = await browser.newPage(); // 设置拦截请求，屏蔽不必要的资源请求 await page.setRequestInterception(true); page.on('request', (request) => { const resourceType = request.resourceType(); if (['image', 'stylesheet', 'font'].includes(resourceType)) { request.abort(); } else { request.continue(); } }); // 抓取数据 const allCOntents= []; for (const data of config) { const cOntents= await fetchData(page, data.name, data.url, data.hrefSelector); allContents.push(contents); } 

爬取易报错的网站

{ name: '第一财经', url: 'https://www.yicai.com/news/', hrefSelector: '#newsRank div:nth-child(1) > ul > li a' }, { name: '金融界', url: 'https://stock.jrj.com.cn/', hrefSelector: 'ul.opportunity-list > li a' }, { name: '八阕', url: 'https://news.popyard.space/cgi-mod/threads.cgi?lan=cn&r=0&cid=11&t=all', hrefSelector: 'div#page_1 > table b > a' } 

本地 Windows 爬取第一财经也报错..

系统

Extra IPv4None RAM2.5 GB RAM (Included) CPU Cores2 CPU Cores (Included) Operating SystemDebian 12 64 Bit (Recommended Min. 2 GB RAM) LocationSan Jose, CA (Test IP: 192.210.207.88)