[FIXED] How to parse JavaScript Json into Python dict type, effeciently

Issue

I am looking for way to read javascript json data loaded into one of a script tag of this page. I have tried various re patterns posted on google and stackoveflow but got nothing.

The Json Formatter shows an Invalid (RFC 8259).

Here is a code

import requests,json
from scrapy.selector import Selector

headers = {'Content-Type': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3'}

url = 'https://www.zocdoc.com/doctor/andrew-fagelman-md-7363?insuranceCarrier=-1&insurancePlan=-1'

response = requests.get(url,headers = headers)

sel = Selector(text = response.text)

profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse(')[-1].split(');\n          window.ZD = {')[0]
    
profile_json = json.loads(profile_data)
    
print(type(profile_json))

The problem seems an invalid json format. The type of profile_json is string while a little amendments in above code shows below error stack

>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse("')[-1].split('");\n          window.ZD = {')[0].replace("\\","")
>>> profile_json = json.loads(profile_data)
Traceback (most recent call last):
  File "/usr/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 41316 (char 41315)

Error in output are highlighted here:

The original HTML contains this (heavily trimmed):

<script>
   ...
   window.__REDUX_STATE__ = JSON.parse("{\"routing\": ...
   \"awards\":[\"Journal of Urology - \\\"Efficacy, Safety, and Use of Viagra in Clinical Practice.\\\"\",\"Critical Care Resident of the Year - 2003\"],
   ...

The same string extracted by scrapy is this:

"awards":[
               "Journal of Urology - ""Efficacy",
               "Safety",
               "and Use of Viagra in Clinical Practice.""",
               "Critical Care Resident of the Year - 2003"
            ],

It appears the backslashes are removed from it, making the JSON invalid.

Solution

I don't know if this is an efficient way of handling the problem but below code resolved my problem.

>>> import js2xml
>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}')
>>> parsed = js2xml.parse(profile_data)
>>> js = json.loads(parsed.xpath("//string[contains(text(),'routing')]/text()")[0])

Answered By - Muhammad Danial