推荐学习书目 Learn Python the Hard Way Python Sites PyPI - Python Package Index http://diveintopython.org/toc/index.html Pocoo 值得关注的项目 PyPy Celery Jinja2 Read the Docs gevent pyenv virtualenv Stackless Python Beautiful Soup 结巴中文分词 Green Unicorn Sentry Shovel Pyflakes pytest Python 编程 pep8 Checker Styles PEP 8 Google Python Style Guide Code Style from The Hitchhiker's Guide
这是一个创建于 3319 天前的主题，其中的信息可能已经有所发展或是发生改变。
请问如何爬取 crunchbase 的站点。
目前用的是python的requests，但不知道要在headers中加上什么内容。
第 1 条附言 2016-09-16 13:26:02 +08:00
我的测试代码是：
# encoding=utf-8
import requests
headers = {
'Host':'www.crunchbase.com',
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0",
"Upgrade-Insecure-Requests":"1",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
}
url="https://www.crunchbase.com/sitemap/organizations/4a5eb6a619f10f89ea9cccd15c6fa38f-eb176350a92046f2425523ca58f58ce6"
try:
page1 = requests.get(
url=url,headers=headers,
)
print page1.content
print "--------------------"
print page1.headers
print "--------------------"
print page1.request.headers
print "--------------------"
print page1.status_code
except Exception, e:
print e
# encoding=utf-8
import requests
headers = {
'Host':'www.crunchbase.com',
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0",
"Upgrade-Insecure-Requests":"1",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
}
url="https://www.crunchbase.com/sitemap/organizations/4a5eb6a619f10f89ea9cccd15c6fa38f-eb176350a92046f2425523ca58f58ce6"
try:
page1 = requests.get(
url=url,headers=headers,
)
print page1.content
print "--------------------"
print page1.headers
print "--------------------"
print page1.request.headers
print "--------------------"
print page1.status_code
except Exception, e:
print e
6 条回复 2016-09-24 14:37:40 +08:00
 | 1 BiggerLonger 2016-09-16 14:15:00 +08:00 via Android  1 看一下器的 header |
 | 2 caspartse 2016-09-16 14:42:59 +08:00  1 page1.headers 是 Response headers ， page1.request.headers 是 Request headers 。  Copy Request headers 就可以了。 |
 | 3 makeapp OP 谢谢各位指点，我试了很多 headers ，我一直怀疑是 headers 的问题，最后发现是阿里云的 ip 的问题。我换 ip 后就好了。 |
 | 6 googlefans 2016-09-24 14:37:40 +08:00 爬后的网站是什么 |
Select Language