[FIXED] Scrape a JavaScript heavy website with requests+bs4

Issue

I'm trying to scrape some data on a stock market. But I don't like to use selenium. I always trace the fetch requests and use the requests module.

On this site, I found the URLs which I think as responsible for fetching data to the frontend.

But those links can't be accessed directly ex:link. They throw HTTP 405. Why is that?

Solution

You can access the API by making a POST request.

Try this:

import requests
response = requests.post('https://www.cse.lk/api/marketSummery')
json.loads(response.content)

Output

{'id': 30071974,
 'tradeVolume': 5829973145.4,
 'shareVolume': 231803649,
 'tradeDate': 1660549800496}

The other APIs work too, eg. this:

url = " https://www.cse.lk/api/dailyMarketSummery"
response = requests.post(url)
json.loads(response.content)

Output

[[{'id': 12893,
   'tradeDate': 1660501800000,
   'marketTurnover': 5829973000.0,
   'marketTrades': 51086.0,
   'marketDomestic': 50376.0,
   'marketForeign': 710.0,
   'equityTurnover': 5829973000.0,
   'equityDomesticPurchase': 5709995000.0,
   'equityDomesticSales': 5592159700.0,
   'equityForeignPurchase': 119978192.0,
   'equityForeignSales': 237813568.0,
   'volumeOfTurnOverNumber': 231803648.0,
   'volumeOfTurnoverDomestic': 226246512.0,
   'volumeOfTurnoverForeign': 5320827,
   'tradesNo': 51086,
   ...

Answered By - petezurich