[FIXED] How to grab text inside a script tag with scrapy?

Issue

I need to grab as text the contents of a script tag with a very specific attribute with the scrapy library. Essentialy the BeautifulSoup equivalent of this:

js_content = soup.find("script",type="application/ld+json").get_text()

I tried this, but the result is not quite what I need.

response.css('script').attrib['type']

Solution

CSS:

response.css('script[type="application/ld+json"]::text').get()

xpath:

response.xpath('//script[@type="application/ld+json"]/text()').get()

Basically we're finding a script tag that has an attribute type with a value of application/ld+json and grabbing the text.

Answered By - SuperUser