[FIXED] Could I ask how do I know how many pages are in the website? (Web scrapping)

Issue

I get a website (https://www.diabetesdaily.com/forum/threads/had-a-friend-with-type-one.136015/)

And I found there are many forum pages

Want to use the for loop for web scrapping, therefore could I ask, how I get the maximum number of forum pages on this page by BeaurifulSoup? Many thanks.

Solution

You can try something like this:

import requests
from bs4 import BeautifulSoup as bs

url = "https://www.diabetesdaily.com/forum/threads/had-a-friend-with-type-one.136015/"
req = requests.get(url)
soup = bs(req.content, 'html.parser')
navs = soup.find("ul", { "class" : "pageNav-main" }).find_all("li", recursive=False)
print(navs)
print(f'Length: {len(navs)}')

Result

[<li class="pageNav-page pageNav-page--current"><a href="/forum/threads/had-a-friend-with-type-one.136015/">1</a></li>, <li class="pageNav-page pageNav-page--later"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-2">2</a></li>, <li class="pageNav-page pageNav-page--later"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-3">3</a></li>, <li class="pageNav-page"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-4">4</a></li>]
Length: 4

Answered By - Coderio