Python爬虫学习-反爬虫机制与应对方法

在爬取数据的过程中遇到以下几种反爬手段： 1 .浏览器反爬检测 2 .IP访问限制 对应的解决措施有： 1 .修改hearders。

headers = {
    authority: ditu.amap.com,
    cache-control: max-age=0,
    upgrade-insecure-requests: 1,
    user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36,
    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,
    accept-encoding: gzip, deflate, br,
    accept-language: zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7,
    cookie: guid=2dcf-1337-74dc-b27b; key=bfe31f4e0fb231d29e1d3ce951e2c780; UM_distinctid=1691af7cd44330-03573b8a5b15a7-1333063-100200-1691af7cd455ff; cna=3qWbFB2RkCcCAXMi8f/QNVwq; CNZZDATA1255626299=343925159-1550935456-%7C1550935456; isg=BJqaOgaPkuSm5B5IstqSUWE460C8yx6lqRRx-qQTQi34FzpRjFobtQJl46Mux5Y9; _uab_collina=155093564321243494443932,
}
demo = requests.get(url,headers=headers)

2 .修改requests中的proxies。

proxies = {

    "http": "http://203.174.112.13:3128"  # 代理ip

}
demo = requests.get(url,proxies=proxies)