爬全景网壁纸(反爬机制Referer)

url以及参数:

https://www.quanjing.com/Handler/SearchUrl.ashx?t=1509&callback=searchresult&q=%E5%A3%81%E7%BA%B8&stype=1&pagesize=100&pagenum=1&imageType=2&imageColor=&brand=&imageSType=&fr=1&sortFlag=1&imageUType=&btype=&authid=&_=1597160644966
t: 1509
callback: searchresult
q: 壁纸
stype: 1
pagesize: 100
pagenum: 1
imageType: 2
imageColor: 
brand: 
imageSType: 
fr: 1
sortFlag: 1
imageUType: 
btype: 
authid: 
_: 1597160644966

爬取全景网壁纸图片,该网站增加了反爬机制Referer,在url参数上添加了回调函数callback=searchresult 破解方法:在请求头添加Referer,具体代码如下:

#添加Referer,用来破解Referer反爬机制
header={
          
   
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 Edg/84.0.522.52,
    Referer: https://www.quanjing.com/search.aspx?q=%E5%A3%81%E7%BA%B8
}
respone=requests.get(url=url1,headers=header)

添加了异步下载,全部代码如下:

执行结果如下:

经验分享 程序员 微信小程序 职场和发展