[FIXED] Extract website but they provide wrong output

Issue

I am to try to extract table they will give me the output but they will be wrong this is page link https://hoopshype.com/salaries/players/

from scrapy.http import Request
import scrapy
class PushpaSpider(scrapy.Spider):
    name = 'pushpa'
    page_number = 1
    start_urls = ['https://hoopshype.com/salaries/players/']
    custom_settings = {
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }

    

    def parse(self, response):
       rows = response.xpath("//table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']//thead//tr")
       keys = rows.xpath(".//td/text()").getall()
       keys = [i.strip() for i in keys]

       keys = [i for i in keys if i]
       
       columns=response.xpath("//table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']//tbody//tr")
       for column in columns:
           players=column.xpath('td//text()').getall()
           players = ''.join(players).strip()
           details = dict(zip(keys, players))
           yield details

Solution

Try this:

from scrapy.http import Request
import scrapy
class PushpaSpider(scrapy.Spider):
    name = 'pushpa'
    page_number = 1
    start_urls = ['https://hoopshype.com/salaries/players/']
    custom_settings = {
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }



    def parse(self, response):
       rows = response.xpath("//table/thead/tr")
       keys = rows.xpath(".//td/text()").getall()
       keys = [i.strip() for i in keys]

       keys = [i for i in keys if I]

       columns=response.xpath("//table/tbody/tr")
       for column in columns:
           player_name = [column.xpath('td[@class="name"]/a/text()').get().strip()]
           detail = column.xpath('td/@data-value').getall()
           details = dict(zip(keys, player_name+detail))
           yield details

The data-value saves the numerical values too, so we can use it too. Because I had problems extracting the text() as you had.

Finally, I think that you don't need to specify the table class name (table[@class='hh-salaries-ranking-table hh-salaries-table-sortable responsive']) because the page just has one table.

Answered By - Brenda S.