[FIXED] How to extract data from multiple pages which have date in their url?

Issue

I want to extract content from a website where the link is as follows:

"www.example.com/getpublicreport?date=2021-10-01"

Using Requests what should be the code to extract data from multiple pages where I could navigate using the date in url.

For example if I want to extract data from date - 2019-01-01 till the current data how do I write code using request library to get the data.

Solution

Hi you can use datetime package :)

For example:

import datetime

import requests


def extract_data(start_date, end_date):
    while start_date <= end_date:
        yield requests.get('www.example.com/getpublicreport?date=%s' % start_date.isoformat())
        start_date += datetime.timedelta(days=1)

if __name__ == '__main__':
    for r in extract_data(datetime.date(2019, 01, 01), datetime.date.today()):
        print(r.content)

Answered By - Félix Herbinet