爬全景网壁纸(反爬机制Referer)
url以及参数:
https://www.quanjing.com/Handler/SearchUrl.ashx?t=1509&callback=searchresult&q=%E5%A3%81%E7%BA%B8&stype=1&pagesize=100&pagenum=1&imageType=2&imageColor=&brand=&imageSType=&fr=1&sortFlag=1&imageUType=&btype=&authid=&_=1597160644966 t: 1509 callback: searchresult q: 壁纸 stype: 1 pagesize: 100 pagenum: 1 imageType: 2 imageColor: brand: imageSType: fr: 1 sortFlag: 1 imageUType: btype: authid: _: 1597160644966
爬取全景网壁纸图片,该网站增加了反爬机制Referer,在url参数上添加了回调函数callback=searchresult 破解方法:在请求头添加Referer,具体代码如下:
#添加Referer,用来破解Referer反爬机制
header={
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 Edg/84.0.522.52,
Referer: https://www.quanjing.com/search.aspx?q=%E5%A3%81%E7%BA%B8
}
respone=requests.get(url=url1,headers=header)
添加了异步下载,全部代码如下:
执行结果如下:
