[FIXED] Scraping items in what looks like a dictonary or json - bs4

Issue

Does anyone know how to grab historical settles from <Barchart.com>?

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
url = 'https://www.barchart.com/futures/quotes/NGG22'
r = requests.get(url, headers=headers) #get the url
soup = BeautifulSoup(r.text, 'html.parser')
contract = soup.find('div', {'class': 'symbol-name'}).text.strip()
price = soup.find('span', {'data-ng-class': "highlightValue('lastPrice')"})

This isn't working. If I could just get the symbol and lastPrice from this html code, that would be better.

<div class="page-title symbol-header-info  " data-ng-controller="symbolHeaderCtrl" data-ng-init="init({"symbol":"NGG22","symbolName":"Natural Gas","symbolType":2,"lastPrice":"6.265s","priceChange":"+1.988","percentChange":"+46.48%","bidPrice":"N\/A","askPrice":"N\/A","bidSize":"N\/A","askSize":"N\/A","tradeTime":"01\/27\/22","lastPriceExt":"-","priceChangeExt":"-","percentChangeExt":"-","tradeTimeExt":"-","contractName":"Natural Gas (Feb \u002722)","daysToContractExpiration":"-8","symbolCode":"FUT","exchange":"NYMEX","sicIndustry":"N\/A","symbolRoot":"NG","sessionDateDisplayLong":"Thu, Jan 27th, 2022","shouldUpdate":false})"> == $0`

Solution

IIUC:

import json

# Find the right html element and extract its attribute
data = soup.find('div', {'class': 'page-title symbol-header-info'})['data-ng-init']

# Extract the json part of previously extracted attribute
data = data[data.find('{'):data.rfind('}')+1]

# Convert json to dict
data = json.loads(data)

Output:

>>> data['symbol']
'NGG22'

>>> data['lastPrice']
'6.265s'

>>> data
{'symbol': 'NGG22',
 'symbolName': 'Natural Gas',
 'symbolType': 2,
 'lastPrice': '6.265s',
 'priceChange': '+1.988',
 'percentChange': '+46.48%',
 'bidPrice': 'N/A',
 'askPrice': 'N/A',
 'bidSize': 'N/A',
 'askSize': 'N/A',
 'tradeTime': '01/27/22',
 'lastPriceExt': '-',
 'priceChangeExt': '-',
 'percentChangeExt': '-',
 'tradeTimeExt': '-',
 'contractName': "Natural Gas (Feb '22)",
 'daysToContractExpiration': '-8',
 'symbolCode': 'FUT',
 'exchange': 'NYMEX',
 'sicIndustry': 'N/A',
 'symbolRoot': 'NG',
 'sessionDateDisplayLong': 'Thu, Jan 27th, 2022',
 'shouldUpdate': False}

Answered By - Corralien