[FIXED] Python BeatifulSoup find all divs returning empty

Issue

I'm trying to get all house listings from a portuguese real estate agency website. I'm using the following small piece of code:

import requests
import bs4
import os

REAL_ESTATE_AGENCY_URL= os.getenv('REAL_ESTATE_AGENCY_URL')

response = requests.get(REAL_ESTATE_AGENCY_URL)
response.raise_for_status()
content = response.text

soup = bs4.BeautifulSoup(content, 'lxml')

all_listings = soup.find_all(name="div", class_="listing-card")
print(all_listings)

The same link I'm using is: this one (It was too long to put in plain text). The print statement only prints an empty array. As you can see on the image below, the div containing all the information has a class of listing-card. I can't understand what am I doing wrong.

Solution

You are returning empty result because the website is entirely depends on JavaScript.BeautifulSoup can't mimic data but You can grab data easily from api calls json response using only requests module.Here is the woriking example.

Script:

import requests
import json
body= {"filters":[{"field":"BusinessTypeID","value":"1","type":0},{"field":"NumberOfBedrooms","biggerThan":2,"type":4},{"field":"ListingTypeID","shouldValues":["11","1"],"type":2},{"field":"Region1ID","value":"78","type":0}],"sort":{"fieldToSort":"ContractDate","order":1}}
headers= {
    'content-type': 'application/json',
   }

api_url = "https://www.remax.pt/Api/Listing/MultiMatchSearch?page=1&searchValue=&size=20"


jsonData = requests.post(api_url, data=json.dumps(body), headers=headers).json()
#print(jsonData)
for item in jsonData['results']:
    price=item['listingPriceText']
    print(price)

Output:

125 000 €
137 500 €
132 500 €
142 500 €
132 500 €
152 500 €
132 500 €
142 500 €
135 000 €
145 000 €
110 000 €
198 000 €
120 000 €
260 000 €
265 000 €
305 000 €
252 500 €
500 000 €
75 000 €
142 000 €

Answered By - F.Hoque