[FIXED] Python Scraping empty tag

Issue

I have a problem with scraping some element from a page: https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i

code:

import requests
from bs4 import BeautifulSoup


URL="https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i"
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

title=soup.find(class_="product_cart_title").text
price=soup.find(class_="icon_main_block_price_a")
number=soup.find(class_="product_cart_info").findAll('tr')[1].findAll('td')[1]
description=soup.find(id="tab_a")


print(description)

Problem is when I want to get to: tab_a

And its a problem cause inside

<div align="left" class="product_cart_info" id="charlong_id">
</div>

is empty. How I can get it? I see its about js i think. Maybe there is some delay when the page loads?

Solution

As stated in the comments, the info is loaded via JavaScript, so BeautifulSoup doesn't see it. But you if you look to Chrome/Firefox network tab, you can see where the page is making requests:

import re
import requests
from bs4 import BeautifulSoup

url = 'https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i'
ajax_url = 'https://tuning-tec.com/_template/_show_normal/_show_charlong.php?itemId={}'

soup = BeautifulSoup(requests.get(url).content, 'html.parser')

print(soup.select_one('.product_cart_title').get_text(strip=True))
print(soup.select_one('.icon_main_block_price_a').get_text(strip=True))
print(soup.select_one('td:contains("Symbol") ~ td').get_text(strip=True))

item_id = re.findall(r"ajax_update_stat\('(\d+)'\)", soup.text)[0]
soup2 = BeautifulSoup(requests.get(ajax_url.format(item_id)).content, 'html.parser')

print()

# just print some info:
for tr in soup2.select('tr'):
    print(re.sub(r' {2,}', ' ', tr.select_one('td').get_text(strip=True, separator=' ')))

Prints:

MERCEDES W164 ML M-KLASA 05-07 BLACK LED SEQ
1788.62 PLN
LPMED0

PL
Opis
Lampy 
 soczewkowe ze światłem 
 pozycyjnym LED. Z dynamicznym 
 kierunkowskazem. 100% nowe, w komplecie 
 (lewa i prawa). Homologacja: norma E13 - 
 dopuszczone do ruchu.
Szczegóły
Światła pozycyjne: DIODY Kierunkowskaz: DIODY Światła 
 mijania: H9 w 
 zestawie Światła 
 drogowe: H1 w 
 zestawie Regulacja: elektryczna (silniczek znajduje się w 
 komplecie).
LED TUBE LIGHT Dynamic Turn Signal >>

Answered By - Andrej Kesely