[FIXED] How to get json response using python?

Issue

I want to get a print a json response from a public url. It is giving me timeout error but when I'm using browser or postman I'm able to see the response.

The url: https://www1.nseindia.com/homepage/Indices1.json

So far I've tried both scrapy and requests

Using scrapy:

class StockSpider(scrapy.Spider):
    name = "pe"
    urls = [
        "https://www1.nseindia.com/homepage/Indices1.json"
    ]

    def start_requests(self):
        for url in self.urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        json_response = json.loads(response.body_as_unicode())
        print("+++++++++++++++++++++ printing json response +++++++++++++++++++")
        print(json_response)

Using requests:

try:
    r = requests.get("https://www1.nseindia.com/homepage/Indices1.json")
    print(r.json())
except:
    print("Timeout occurred")

But in both of the approach I'm getting timeout error.

Solution

You have to add a user-agent header.

import requests

try:
    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
    r = requests.get("https://www1.nseindia.com/homepage/Indices1.json", headers=headers)
    print(r.json())
except Exception as e:
    print(e)
    print("Timeout occurred")

Answered By - Sri