[FIXED] How to scrape pages with "load more" button?

Issue

I try to scrape stock symbols. Unfortunately the page that I would like to scrape requires you to click a "load more" button to get all the data.

Here is my code:

import requests
import pandas as pd
from tvDatafeed import TvDatafeed, Interval
from bs4 import BeautifulSoup

symbols = []

url = "https://th.tradingview.com/markets/stocks-thailand/market-movers-all-stocks/"

r = requests.get(url)
data = r.text
soup = BeautifulSoup(data, 'html.parser')

stock_table = soup.find_all('tr', {'class': 'row-RdUXZpkv'})

for row in stock_table:
    cells = row.find_all('a',{'class': 'tickerName-GrtoTeat'})
    for cell in cells:
        symbols.append(cell.text.strip())

Solution

Seems like the "Load-more" button defines/increases the range of symbols to display.

You can adjust this one (e.g 0-900) and post a request to get the data in JSON format.

import requests

url = "https://scanner.tradingview.com/thailand/scan"

payload = {
    "columns": [
        "name", # << add more fields if needed
    ],
    "ignore_unknown_fields": False,
    "options": {"lang": "th"},
    "range": [0, 900],
    "sort": {"sortBy": "name", "sortOrder": "asc", "nullsFirst": False},
    "preset": "all_stocks"
}

headers = {
    "Accept": "application/json",
    "Referer": "https://th.tradingview.com/",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
                  "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

r = requests.post(url, json=payload, headers=headers)

symbols = [d["d"][0] for d in r.json()["data"]]

Output :

print(symbols)

[
    '24CS',
    '2S',
    '3K-BAT',
    '7UP',
    'A',
    # ...
    'YONG',
    'YUASA',
    'ZAA',
    'ZEN',
    'ZIGA'
]

len(symbols) # 853

Answered By - Timeless