菜鸟初学Python网页数据抓取01

菜鸟学习python网页数据抓取，以Bing为用户代理，搜索“知网”，抓取到响应后数据界面。

初步学习后的代码如下：

import requests

# 1 explict Url
url = https://cn.bing.com/search
# 2.1 ua pretending
headers = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111
    .0.1661.41
# 2.2 pakage params into dictionary
kw = input(please enter a word:)
param = {
    query : kw
}
# 2.3 excuting request for the resource in url
# 3 and get response data from servers
response = requests.get(url = url, params = param, headers =headers)
# 4 sustainably store data
page_text = response.text
fileName = kw + .html
with open(fileName,w, encoding = utf-8) as fp:
    fp.write(page_text)
print(fileName,sucessfully storing!!!)

运行后出现报错：

经了解后，发现items是 dic 的属性，而语句中的 ‘headers’是字符串形式，因此出现报错。

headers = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111
    .0.1661.41

尝试解决方式如下，将headers封装进 dic ：

ua = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111
    .0.1661.41
headers = {
    headers:ua
}

问题解决，运行结果和抓取数据如下：

菜鸟学习python网页数据抓取，以Bing为用户代理，搜索“知网”，抓取到响应后数据界面。 初步学习后的代码如下： import requests # 1 explict Url url = https://cn.bing.com/search # 2.1 ua pretending headers = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111 .0.1661.41 # 2.2 pakage params into dictionary kw = input(please enter a word:) param = { query : kw } # 2.3 excuting request for the resource in url # 3 and get response data from servers response = requests.get(url = url, params = param, headers =headers) # 4 sustainably store data page_text = response.text fileName = kw + .html with open(fileName,w, encoding = utf-8) as fp: fp.write(page_text) print(fileName,sucessfully storing!!!) 运行后出现报错： 经了解后，发现items是 dic 的属性，而语句中的 ‘headers’是字符串形式，因此出现报错。 headers = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111 .0.1661.41 尝试解决方式如下，将headers封装进 dic ： ua = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111 .0.1661.41 headers = {     headers:ua } 问题解决，运行结果和抓取数据如下：