用Scrapy爬取新浪微博用户信息,粉丝,关注数

创建项目

scrapy startproject weibospider
目录层级如下:
weibospider
    -weibospider
        -spiders # 爬虫目录,创建爬虫文件
            -__init__.py
            _weibo.py   # 写入爬虫相关信息
        __init__.py
        items.py    # 设置数据存储模板,用于结构化数据,如Django中的models
        middlewares.py  # 设置中间下载键  可以设置请求头和IP池
        pipelines.py   # 一般结构化的数据持久化
        setting.py  # 爬虫运行的相关设置
    main.py   # 设置快捷启动
    scrapy.cfg   # 项目的配置信息,主要为Scrapy命令行工具提供一个基础的配置信息,主要的配置信息在setting.py中

爬取用户信息

items.py页面

class UserItem(scrapy.Item):
    collections = user_info
    # define the fields for your item here like:
    # name = scrapy.Field()
    id = scrapy.Field()
    screen_name = scrapy.Field()
    profile_image_url = scrapy.Field()
    profile_url = scrapy.Field()
    verified_reason = scrapy.Field()
    close_blue_v = scrapy.Field()
    description = scrapy.Field()
    gender = scrapy.Field()
    follow_me = scrapy.Field()
    following = scrapy.Field()
    followers_count = scrapy.Field()
    follow_count = scrapy.Field()
    cover_image_phone = scrapy.Field()
    avatar_hd = scrapy.Field()
    create_time = scrapy.Field()


class UserRelationItem(scrapy.Item):
    collections = user
    id =scrapy.Field()
    fans = scrapy.Field()
    follower = scrapy.Field()

pipelines.py页面,数据持久化操作

middlewares.py 设置请求头和IP池

import random
from scrapy.conf import settings
from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware

class RandomUserAgent(UserAgentMiddleware):
    def process_request(self, request, spider):
        random.choice(settings[USER_AGENT_LIST])
        request.headers.setdefault(bUser-Agent, self.user_agent)

class RandomProxy(object):
    def process_request(self, request, spider):
        random_proxy = random.choice(settings[PROXY])
        request.meta[proxy] = http://%s %  random_proxy

在setting.py中
DOWNLOADER_MIDDLEWARES = {
   # weibospider.middlewares.WeibospiderDownloaderMiddleware: 543,
   weibospider.middlewares.RandomUserAgent: 543,
   weibospider.middlewares.RandomProxy: 544,
}