[FIXED] Web Scraping by BeautifulSoup if the data shown only if I click "show details"

Issue

I am trying to scrape data from selling cars website, when I enter the website I see a table of cars (type, price, year), but if I want to know more details about the car I have to click on the car and then it shows more details. How can i scrape data from the those details without Selenium?

import headers
import requests
from bs4 import BeautifulSoup

page_num = 1
url = f"https://www.example.com/vehicles/cars?page={page_num}"

req = requests.get(url, headers=headers.get_headers()).text

soup = BeautifulSoup(req,"html.parser")

def decide_row(soup):
    rows = soup.find_all('div',class_="feeditem table")
    return rows

def decide_details(rows):
    for car in rows:
        car_kilometrage = car.find('div',id='accordion_wide_0')
        print(car_kilometrage)

decide_details(decide_row(soup))

Solution

There are different approaches belonging to your needs:

• extract data from the script section of the source

• collecting item-ids, construct item url and request each item page

  for e in soup.select('div[item-id]'):
      print(f'https://www.yad2.co.il/item/{e.get("item-id")}')
  

Answered By - HedgeHog