爬虫入门技巧1——如何获取浏览器cookies绕过反爬虫

抓取雪球的时候遇到一个问题，cookies失效导致爬虫失败 解决方案:

1. 每次手动更新浏览器cookies 【评价】爬一次还好，要是需要更新就扑街了
2. 访问首页获取cookies 【评价】之前也想过，但是没有加header所以还是被ban了， 【要点】获取首页cookies时~~~ 使用不加cookies的 header(仔细想想也是本来就是来骗cookie的…………)

link1="https://xueqiu.com"
    header1={
          
   
        Accept: */*,
        Accept-Encoding: gzip, deflate, br,
        Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,
        Connection: keep-alive,
        #Cookie: 
        Host: xueqiu.com,
        Referer: https://xueqiu.com/,
        User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36,
        X-Requested-With: XMLHttpRequest,
        User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
    }
    r1=s.get(link1,headers=header1)

    # 2、带着cookies 直接访问就好了 获取maxpage 循环
    link2="https://xueqiu.com/statuses/original/timeline.json?user_id="+str(xueqiu_id)+"&page=1"
    maxPage1=json.loads(s.get(link2,headers=header1).content)[maxPage]

tips: 1、通过浏览器给header加引号的工作能浪费半条命，可以写个函数自己加进去

def get_headers(header_raw):
    # 通过原生请求头获取请求头字典
    # :param header_raw: {str} 浏览器请求头
    # :return: {dict} headers
    header_raw = header_raw.strip()  # 处理可能的空字符
    header_raw = header_raw.split("
")  # 分割每行
    header_raw = [line.split(":", 1) for line in header_raw]  # 分割冒号
    header_raw = dict((k.strip(), v.strip()) for k, v in header_raw)  # 处理可能的空字符
    return header_raw
#只要复制出来就好了
hed1="""Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,zh-TW;q=0.8,en;q=0.7
Connection: keep-alive
Content-Length: 29
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: JSESSIONID=00008wXnhoz-YlCGaw_vv2Y_2Bu:-1; BIGipServerPool_licai_webapp=44373258.31011.0000; count=1
Host: www.chinawealth.com.cn
Origin: https://www.chinawealth.com.cn
Referer: https://www.chinawealth.com.cn/zzlc/jsp/lccpDetail.jsp
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
X-Requested-With: XMLHttpRequest"""