[FIXED] Scrapy - Web scraping returning empty list from crypto exchange site

Issue

I usually have success when I web scrape but having trouble with this one. I'm assuming I'm getting blocked or they have security measures.

I'm trying to get the APY rates from https://www.okcoin.com/earn . You don't need an account to login and should be fairly straight forward so here's my code (the xpath I've provided is the table under Other offers):

from requests_html import HTMLSession
from scrapy import Selector

def url_headers(url):
    """
    Creates headers for each url to be scraped.

    param url: webpage for scraping
    :return:
    """

    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0',
        'Accept': '*/*',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate',
        'Referer': url
    }

    # Get HTML version of document
    sess = HTMLSession()
    res = sess.get(url, headers=headers)

    # Convert to text and extract whole document
    selector = Selector(text=res.content)
    return selector

sel = url_headers("https://www.okcoin.com/earn")

apy = sel.xpath(
    '/html/body/div[2]/div/div/div/div[2]/div/div/div[3]/div/div[3]'
).extract()

print(apy)

Solution

Data is also generating from api calls json response. So You can do that only using requests

    import requests
    
    api_url = 'https://www.okcoin.com/v2/asset/outer/earn/project-currency?t=1646119369275'
    
    resp = requests.get(api_url).json()
    
    for item in resp['data']:
        print(item['rate'])

Output:

265.00%
145.00%
21.41%
21.33%
12.55%
10.00%
9.07%
7.94%
6.22%
5.40%
2.31%
3.19%
1.44%
1.83%
1.36%
1.36%
1.29%
1.05%
0.00%
0.00%

Answered By - F.Hoque