[FIXED] Web Scraping Got Empty Array Values

Issue

Hello All I was trying to scrape a stock website to get stock sector wise info in this website.

I just hit scrapy shell in terminal if the data is acheivable for this table

In the terminal this was my command after I ran scrapy shell "https://nepsealpha.com/" response.xpath("//table[@id='fixTable']//tbody//tr")

but the output I get is empty list = []

I feel content is being rendered with javascript anyway I can do without use of selenium?

Solution

The data you're after comes from an API endpoint.

You can get it and then massage it back to the form of a table or use only parts of it.

Here's how:

import requests

import pandas as pd

api_endpoint = "https://nepsealpha.com/api/smx9841/dashboard_board"

payload = {
    "_token": "K5fwARzoE7j49mIE5hdeZUqeoYgQGUXnsUeS7SG1"
}

headers = {
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest",
}

response = requests.request("POST", api_endpoint, headers=headers, data=payload)
data = response.json()["home_table"]
df = pd.json_normalize(data)
print(df)

Output:

       id    index_name  ... indexvalue.percent_change indexvalue.turn_over_value
0   41538         NEPSE  ...                      0.74              1204409411.62
1   41541       BANKING  ...                      0.51                181973963.3
2   41548       TRADING  ...                      0.75                  121077893
3   41550        HOTELS  ...                      1.05                  9852264.4
4   41547       DEVBANK  ...                      1.16                 52379584.2
5   41543    HYDROPOWER  ...                      1.67                317296705.3
6   41546       FINANCE  ...                      1.18                 39390111.3
7   41542   NONLIFEINSU  ...                      0.97                 42936919.2
8   41544   MANUFACTURE  ...                     -1.06                126684476.5
9   41549        OTHERS  ...                      0.71                 21071125.9
10  41540  MICROFINANCE  ...                       0.5                171446649.5
11  41545      LIFEINSU  ...                      0.46                 52938619.8
12  41551    INVESTMENT  ...                      1.11                 44639401.2

Answered By - baduker