[FIXED] beautifulsoup not returning complete html of the page in python

Issue

I have taken help to extract data from the package BeautifulSoup, but unfortunately, it does not give me the output of the complete HTM file.

   from bs4 import BeautifulSoup
   import requests

   url = 'https://www.digikala.com/search/category-mobile-phone/?page=1'
   response = requests.get(url)
   print(response.text)
   soup = BeautifulSoup(response.content,"lxml")
   m= soup.find_all("h2", {'class':'ellipsis-2 text-body2-strong 
    color-700'})
   print(m)
   

Solution

This page is loading data from an API. You can see this by inspecting Network tab in Chrome Dev Tools, reloading the page.

The following code will return a json with all items from page 1:

from bs4 import BeautifulSoup
import requests

url = 'https://api.digikala.com/v1/categories/mobile-phone/search/?page=1'
response = requests.get(url).json()
print(response)

Answered By - platipus_on_fire_333