[FIXED] Can't get all html page with Beautiful soup

Issue

I'm trying to get the content of this webpage : https://www.zillow.com/homes/for_rent/1-_beds/?searchQueryState=%7B%22pagination%22%3A%7B%7D%2C%22mapBounds%22%3A%7B%22west%22%3A-122.67022170019531%2C%22east%22%3A-122.19643629980469%2C%22south%22%3A37.615282466144976%2C%22north%22%3A37.93495488175342%7D%2C%22mapZoom%22%3A11%2C%22isMapVisible%22%3Atrue%2C%22filterState%22%3A%7B%22price%22%3A%7B%22max%22%3A872627%7D%2C%22beds%22%3A%7B%22min%22%3A1%7D%2C%22fore%22%3A%7B%22value%22%3Afalse%7D%2C%22mp%22%3A%7B%22max%22%3A3000%7D%2C%22nc%22%3A%7B%22value%22%3Afalse%7D%2C%22fr%22%3A%7B%22value%22%3Atrue%7D%2C%22cmsn%22%3A%7B%22value%22%3Afalse%7D%2C%22fsba%22%3A%7B%22value%22%3Afalse%7D%7D%2C%22isListVisible%22%3Atrue%7D

I can't get all of it. Many elements are empty. I was told that it was the case because it was js code and bs4 can't read js and I had to use selenium instead, but I want to do it with bs4 and I know there is a way to do so. I also was told that it was the case, because I wasn't in the correct iframe, but I doesn't seem to be true. For example if you inspect one of the prices listed (e.g $2,200/mo) you will see that it is contained in a ul list and each apartment listed is a li element of that list. But when I scrape the page with bs it seems that most of these li elements are empty. Also, bear in mind I'm a newbie in web-scraping and in python, so be cool please. Thanks!

Here is the code I'm using to get the page html:

self.response = requests.get(url=URL, headers=headers)
self.html_doc = self.response.text
self.soup = BeautifulSoup(self.html_doc, 'html.parser')

Solution

It appears that I have to use selenium in order to do the work. Thanks all for you participation!

Answered By - younes alaoui