[FIXED] Parsing JSON data using BeautifulSoup

Issue

I am trying to fetch the data for some of the ETF's from Tipranks website using the code below:

url = 'https://www.tipranks.com/etf/xle/forecast'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'}
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')
soup1 = soup.findAll('script')[6].text.strip()
data = json.loads(soup1)
text = data['mainEntity'][4]['acceptedAnswer']['text']
suggestion = text.split('rating of ')[1].split(' which is')[0]

I get the error: Expecting value: line 1 column 1 (char 0)

Sometimes it works, but this morning it stopped working, and I don't know if it's because I am hammering the server too much or something else.

Solution

printing soup1 gives some thing like this (with no error): if (window.__tipranksPlanId__

Okay, so that's not JSON.

Sounds like the website was updated, and soup.findAll('script')[6] is no longer what you expect.

Answered By - OneCricketeer