[FIXED] Scraping only first page reviews - Beautiful Soup (Python)

Issue

I need some help scraping all the reviews posted for one product on random website (M&S here). But it scrapes only the 1st page reviews only where it has around 900 reviews on it. And even the URL doesn't change if we go on to next page of reviews). Below is the coding.

And i can extract only the month along with year as date, Is there any way to find out the review posted date.

Any suggestions would be really helpful. Thanks.

Code for scraping Reviews

Solution

The product pages fetch reviews with Javascript POST to https://prod.prod.gql-mesh.gql.mnscorp.net/graphql , a GraphQL backend. The POST body contains the generic GraphQL query and its variables, like product id, limit of reviews per page, and pagination offset.

Use your browser devtools to copy a valid request., then try sending requests for limit=10&offset=0, limit=10&offset=10, limit=10&offset=20, etc.

And by sending a POST request to this backend instead of GET'ing the website, you also get an easy-to-parse response in JSON format. It contains precise fields like "submissionTime": "2023-03-23T09:34:06.000+00:00" for each review.

Answered By - BoppreH