[FIXED] Failed to let a script produce a JSON response mimicking the XHR requests following Chrome dev tools

Issue

When I open this link in the Chrome browser, hover the cursor over Stocks, which is located at the top left corner, and select Trading Liquidity from the category Most Active, I can see the required data displayed on that page.

I attempted to use the script below to replicate the XHR queries that I can see in the Chrome dev tools, but it results in status_code 400.

import requests

url = 'https://www.barchart.com/proxies/core-api/v1/quotes/get'
link = 'https://www.barchart.com/stocks/most-active/daily-volume-leaders'

headers = {
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.5',
    'Host': 'www.barchart.com',
    'Referer': 'https://www.barchart.com/stocks/most-active/trading-liquidity',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
}

params = {
    'lists': 'stocks.us.trading_liquidity.advances.overall',
    'orderDir': 'desc',
    'fields': 'symbol,symbolName,lastPrice,priceChange,percentChange,averageVolume100d,sharesOutstanding,tradingRatio,tradeTime,symbolCode,symbolType,hasOptions',
    'orderBy': 'tradingRatio',
    'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
    'hasOptions': 'true',
    'page': '1',
    'limit': '100',
    'raw': '1',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    s.headers['X-Xsrf-Token'] = res.cookies['XSRF-TOKEN']
    resp = s.get(url)
    print(resp.status_code)
    print(resp.json())

How can I let the script produce the JSON response?

Solution

Nice question, I needed to do some digging.

The cookies aren't encoded properly, hence you'll need to unquote them:

from urllib.parse import unquote

and then:

s.headers['X-XSRF-Token'] = unquote(res.cookies['XSRF-TOKEN'])

---

Using your code:

import requests
from urllib.parse import unquote

url = 'https://www.barchart.com/proxies/core-api/v1/quotes/get'
link = 'https://www.barchart.com/stocks/most-active/daily-volume-leaders'

headers = {
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.5',
    'Host': 'www.barchart.com',
    'Referer': 'https://www.barchart.com/stocks/most-active/trading-liquidity',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
}

params = {
    'lists': 'stocks.us.trading_liquidity.advances.overall',
    'orderDir': 'desc',
    'fields': 'symbol,symbolName,lastPrice,priceChange,percentChange,averageVolume100d,sharesOutstanding,tradingRatio,tradeTime,symbolCode,symbolType,hasOptions',
    'orderBy': 'tradingRatio',
    'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
    'hasOptions': 'true',
    'page': '1',
    'limit': '100',
    'raw': '1',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    s.headers['X-XSRF-Token'] = unquote(res.cookies['XSRF-TOKEN'])
    resp = s.get(url, params=params)
    print(resp.status_code)
    print(resp.json())

Answered By - MendelG