[FIXED] Getting different HTML code from python request.get() and chrome inspect element while using same URL

Issue

I am trying to scrape data from this link. I used python requests.get(url) method to extract the page content. I am getting different HTML code if I compare it with the actual source code of google chrome. I want to scarpe data from position section (as mentioned in image). But I am getting HTML of performace section. ALthough url is same for both sections. Is there any way, I can redirect to the position section when I make a python request? Thanks in Advance

Solution

If you turn off JavaScript then click on position, it wouldn't work but if you turn on JS and click on position then it will work. So you are not getting the the position data because that data is loaded dynamically by JS via API as POST method.

Example:

import requests
import json
api_url ='https://www.binance.com/bapi/futures/v1/public/future/leaderboard/getOtherPosition'

headers= {
    "content-type":"application/json",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}


body = {"encryptedUid":"D64DDD2177FA081E3F361F70C703A562","tradeType":"PERPETUAL"}

res = requests.post(api_url,headers=headers,data=json.dumps(body)).json()
data =[]
for item in res['data']['otherPositionRetList']:
    data.append({
        'symbol':item['symbol'],
        'entryPrice':item['entryPrice'],
        'markPrice':item['markPrice']
        })
print(data)

Output:

[{'symbol': 'LUNA2USDT', 'entryPrice': 2.329198591774, 'markPrice': 2.44419197}, {'symbol': '1000LUNCBUSD', 'entryPrice': 0.2868522607698, 'markPrice': 0.27523487}]

Answered By - Fazlul