REF:

爬取对象：

有融网理财项目列表页【履约中】状态下的前10页数据，地址：

编程思路：

1. 寻找分页地址的变动规律 2. 解析网页，获取内容，放入自定义函数中 3. 调用函数，输出分页内容

详细解说：

1. 首先插入用到的库：BeautifulSoup、requests

1 from bs4 import BeautifulSoup
2 import requests

2. 观察地址的变化规律，可以看到，每切换一页时，后面“createTimeDesc-1.html”中的数字1会随着页面的变动而变动，此时我们将地址存放进列表中，后面用format()和for循环来实现多个地址的存储。

1 urls = [https://www.yrw.com/products/list-direct-all-performance-1-createTimeDesc-{}.html.format(str(i)) for i in range(1,11)]
2 print(urls)

此时可以先print下，看地址是否正确，这里range(1,11)是前10个页面的地址。

def get_titles(urls,data = None):
    web_data = requests.get(urls)
    soup = BeautifulSoup(web_data.text, lxml)
    titles = soup.select( h3 > a > em > strong)
    for title in titles:
        data = {
            title: title.get_text()
        }
        print(data)

4. 最后，我们来调用函数。

1 for titles in urls:
2     get_titles(titles)

完整代码：

from bs4 import BeautifulSoup
import requests

urls = [https://www.yrw.com/products/list-direct-all-performance-1-createTimeDesc-{}.html.format(str(i)) for i in range(1,11)]
# print(urls)

def get_titles(urls,data = None):
    web_data = requests.get(urls)
    soup = BeautifulSoup(web_data.text, lxml)
    titles = soup.select( h3 > a > em > strong)
    for title in titles:
        data = {
            title: title.get_text()
        }
        print(data)

for titles in urls:
    get_titles(titles)

运行结果（只展示部分）：

{title: 资产融ZT321期}

{title: 供应链ZT2923期}

{title: 租车融ZT335期}

{title: 供应链ZT2922期}

{title: 供应链ZT2919期}