[FIXED] Using Scrapy to extract live data that shows on the website

Issue

Using a website like https://www.interpol.int/How-we-work/Notices/View-Red-Notices, I am using scrapy shell and I want to extract the Total number of public Red Notices in circulation which on the browser shows 7596. When I use this command on scrapy shell:

response.css('strong#totalResults::text').get()

It returns a result of 0. That's what is shown on HTML. I'm not understanding why this is returned and what do I do to obtain the actual number?

Solution

The website dynamically loads the value and places it in the element with Javascript. A quick look at the source code of the website tell us that it loads it from this url:

https://ws-public.interpol.int/notices/v1/red

Technically you could just make a http request to it and parse the JSON result with python yourself. HOWEVER: many websites/organizations do not like it when you scrape their website or reverse engineer their API's so please be mindful when you do this. (Don't spam it or request data every second etc).

Many websites actually have a public API to do the very thing you're trying to do. I believe I've heard Interpol has one too. (But I'll leave Googling that to you :p).

Answered By - Matt