[FIXED] How to retrieve data from json response with scrapy?

Issue

I am using scrapy with python.

This is my url:

https://www.workingnomads.co/jobsapi/job/_search?sort=expired:asc,premium:desc,pub_date:desc&_source=company,category_name,description,location_base,instructions,id,external_id,slug,title,pub_date,tags,source,apply_url,premium,expired,use_at

My code:

def parse(self, response):
        jsonresponse = json.loads(response.body_as_unicode())
        print("============================================================================================================================")
        print(jsonresponse["hits"]["hits"])

It returns the response in JSON format which looks like this.

How can I get the value of a specific key?

this one is the postman response

i want to retrive apply_url key value.

Solution

You'll want to access:

['hits']['hits'][x]['_source']['apply_url']

Where x is the number of items/nodes under hits. See https://jsoneditoronline.org/#left=cloud.22e871cf105e40a5ba32408f6aa5afeb&right=cloud.e1f56c3bd6824a3692bf3c80285ae727

As you can see, there are 10 items or nodes under hits -> hits. apply_url is under _source for each item.

def parse(self, response):
    jsonresponse = json.loads(response.body_as_unicode())
    print("============================================================================================================================")
    for x, node in enumerate(jsonresponse):
        print(jsonresponse['hits']['hits'][x]['_source']['apply_url'])

For example, print(jsonresponse['hits']['hits'][0]['_source']['apply_url']) would produce:

https://boards.greenhouse.io/mesosphere/jobs/1422922?gh_jid=1422922

Answered By - Amir Asyraf