[FIXED] Have empty list, while scrapping information from website

Issue

I have issue, when I scrap information about watches CASIO (price, name and model) everytime I get empty list. Can someone faced with this problem? Link for website I scrap information: https://www.casio.com/us/watches/gshock/

This my code:

from bs4 import BeautifulSoup
import requests



def get_data(url):
    headers = {
    "User-Agent": "header"
    }
    
    req = requests.get(url="https://www.casio.com/us/watches/gshock/", headers=headers)
    
    # with open("/Users/yerkebulan/Desktop/parser/mylessons/lesson7/watch7/data.html", "w") as file:
    #     file.write(req.text)
    
    soup = BeautifulSoup(req.text, "lxml")
    
    list_wathces = soup.find_all("li", class_="cmp-product_panel_list__item")
    
    for item in list_wathces:
        watch_name = item.find("div", class_ ="cmp-product_panel__code").text
        watch_company = item.find("div", class_ = "cmp-product_panel__code")
        print(watch_company.text)
    
    # with open("/Users/yerkebulan/Desktop/parser/mylessons/lesson7/watch7", "w") as file:
    #     file.
    
    
def main():
    get_data("https://www.casio.com/us/watches/casio/")
    
    
if __name__ == "__main__":
    main()

Expected:

[
    {
        "product_article": "GA-700SKE-7AER",
        "product_url": "https://shop.casio.ru/catalog/g-shock-youth/ga-700ske-7aer/",
        "product_price": "12 390"
    },
    {
        "product_article": "GA-2100SKE-7AER",
        "product_url": "https://shop.casio.ru/catalog/g-shock-youth/ga-2100ske-7aer/",
        "product_price": "11 790"
    }
]

It is example

Solution

The data you see on the page is loaded from external URL via JavaScript (the data is in Json form). To load the data use next example:

import json

import requests


api_url = "https://www.casio.com/content/casio/locales/us/en/products/watches/gshock/jcr:content/root/responsivegrid/container/product_panel_list_f.products.json"
data = requests.get(api_url).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for d in data["data"]:
    print(f'{d["productName"]:<30} {d.get("sellingPrice", "-"):<10}')

Prints:

...

GW6900-1                       140.0     
DW6900MS-1                     99.0      
G9000MS-1                      110.0     
G9000-1V                       110.0     
G2900F-1V                      99.95     
DW9052-2V                      74.95     
DW9052-1B                      -         
G100-1BV                       110.0     
DW5600E-1V                     74.95     
DW6900-1V                      84.95     

Answered By - Andrej Kesely