[FIXED] Scrapy - Comparing Data

Issue

I am very new to scrapy and am at a point in my project where I am unsure how to proceed. My idea is that I want to scrape the first 2 pages of hackernews and print out all articles / titles with points above 300. Based on my limited knowledge the following code is the best way I could figure out how to get the information I want. My end goal is I need to compare the id with the post id to match them, add the points to the corresponding matches and then filter out points less than 300. I am not sure how I can compare the dictionary values that I have been able to scrape. The code is as follows:

import scrapy

class ArticlesSpider(scrapy.Spider):
    name = 'articles'
    start_urls = [
        'https://news.ycombinator.com',
        # 'https://news.ycombinator.com/news?p=2'
    ]

    def parse(self, response):
        link = response.css('tr.athing')
        score = response.css('td.subtext')
        for website in link:
            yield {
                'title': website.css('tr.athing td.title a.storylink::text').get(),
                'link':  website.css('tr.athing td.title a::attr(href)').get(),
                'id': website.css('tr::attr(id)').get(),
            }
        for points in score:
            yield {
                'post_id': points.css('span::attr(id)').get(),
                'points': points.css('span.score::text').get()
            }

Is there a better way to achieve what I want to do?

Solution

The posts and scores lists have same length and order.

In each iteration, check the score point of the corresponding post is >= 300.

import scrapy

class ArticlesSpider(scrapy.Spider):
    name = 'articles'
    start_urls = [
        'https://news.ycombinator.com',
        #'https://news.ycombinator.com/news?p=2'
    ]

    def parse(self, response):
        posts = response.css('tr.athing')
        scores = response.css('td.subtext')
        
        for i in range(len(posts)):
            # get the post
            post = posts[i]
        
            # get the score point of the corresponding post
            score = scores[i]
            score_point = score.css('span.score::text').get()
            # handle some post has no score point
            score_point = int(score_point.split(' ')[0]) if score_point else 0
            
            if score_point >= 300:
                yield {
                    'title': post.css('tr.athing td.title a.storylink::text').get(),
                    'link':  post.css('tr.athing td.title a::attr(href)').get(),
                    'id': post.css('tr::attr(id)').get(),
                    'points': score_point
                }

Posts with score >= 300 are printed:

{'title': 'One man's fight for the right to repair broken MacBooks', 'link': 'https://columbianewsse
rvice.com/2021/05/21/one-mans-fight-for-the-right-to-repair-broken-macbooks/', 'id': '27254719', 'po
ints': 1138}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'Why I prefer making useless stuff', 'link': 'https://web.eecs.utk.edu/~azh/blog/makinguse
lessstuff.html', 'id': '27256867', 'points': 604}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'Why Decentralised Applications Don't Work', 'link': 'https://ingrids.space/posts/why-dist
ributed-systems-dont-work/', 'id': '27259321', 'points': 320}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'Freesound just reached 500K Creative Commons sounds', 'link': 'https://blog.freesound.org
/?p=1340', 'id': '27232297', 'points': 696}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'The Limits to Blockchain Scalability', 'link': 'https://vitalik.ca/general/2021/05/23/sca
ling.html', 'id': '27257641', 'points': 378}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'Teardown of a PC Power Supply', 'link': 'https://www.righto.com/2021/05/teardown-of-pc-po
wer-supply.html', 'id': '27256515', 'points': 351}
2021-05-24 18:14:28 [scrapy.core.scraper] DEBUG: Scraped from <200 https://news.ycombinator.com>
{'title': 'Dorodango: the Japanese art of making shiny mud balls (2019)', 'link': 'https://www.laure
nceking.com/blog/2019/09/26/dorodango-blog/', 'id': '27255755', 'points': 313}

Answered By - Kevin Lee