[FIXED] Can't get JSON response to be printed or stored in variable selenium python

Issue

Using below code as an example to get what i want in res variable

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time


options = Options()
options.add_argument("--disable-blink-features=AutomationControlled")
#options.headless = True

driver = webdriver.Chrome('C:/Users/Amanr/Downloads/chromedriver_win32 v106/chromedriver', options=options)

driver.get("https://www.")
time.sleep(5)
res = driver.execute_script('''window.open('https://',"_blank");''')
main_window = driver.window_handles[1]
driver.switch_to.window(main_window)
ps = driver.page_source
print(ps)

The result i am getting now is via page source but i want it as clear JSON response. it also doesn't have full data as in the link something like this:

{"identifier":"OPTIDXBANKNIFTY27-10-2022CE41200.00","name":"BANKNIFTY","grapthData":[[1666689300000,359.2],[1666689301000,386.35],[1666689302000,393.45],[1666689303000,397.05],[1666689304000,385.8],[1666689305000,383.25],[1666689306000,378.95],[1666689307000,370.45],[1666689308000,369.15],[1666689309000,372.3],[1666689310000,383.75],[1666689311000,391.65],[1666689312000,400.9],[1666689313000,393.85],[1666689314000,402.7],[1666689315000,397.2],[1666689316000,391.5],[1666689317000,391.95],[1666689318000,391.85],[1666689319000,385.4],[1666689350000,380.55],[1666689351000,380.25],[1666689352000,377.15],[1666689353000,376.1],[1666689354000,373.85],[1666689355000,370

Is there any way i can get directly get the response as json.

Solution

The api endpoint you confirmed in comment above returns the following information - see below:

import requests
import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)

big_list = []
url = 'https://www.nseindia.com/api/chart-databyindex?index=OPTIDXBANKNIFTY27-10-2022CE41200.00'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.47'
}
s = requests.Session()
s.headers.update(headers)
s.get('https://www.nseindia.com/option-chain')

r = s.get(url)
for x in r.json()['grapthData']:
    big_list.append(set(x))
df = pd.DataFrame(big_list, columns = ['Timestamp', 'Value'])
print(df)

Result in terminal:

Timestamp   Value
0   1666689300000   359.20
1   1666689301000   386.35
2   1666689302000   393.45
3   1666689303000   397.05
4   1666689304000   385.80
... ... ...
7691    1666711732000   121.85
7692    1666711733000   121.55
7693    1666711734000   121.20
7694    1666711735000   121.15
7695    1666711736000   121.10
7696 rows × 2 columns

I randomly named the columns, that data probably means something else, nonetheless there it is.

Answered By - Barry the Platipus