[FIXED] Why Scrapy doesn't work for this page?

Issue

I am trying to crawl this webpage:

http://www.oddsportal.com/search/results/:69Dxbc61/

This is my code:

import scrapy

class Test2Spider(scrapy.Spider):
    name = "test2"
    allowed_domains = ["oddportal.com"]
    start_urls = (
        'http://www.oddsportal.com/search/results/:69Dxbc61/',
    )

    def parse(self, response):
        for partita in response.css('tr.deactivate'):
            yield {
                'score': partita.css('td.table-score::text').extract_first(),
            }

But I get this:

# scrapy runspider test2.py -o uno.json
2018-04-19 16:45:56 [scrapy] INFO: Scrapy 1.0.3 started (bot: cinvestbacktest)
2018-04-19 16:45:56 [scrapy] INFO: Optional features available: ssl, http11, boto
2018-04-19 16:45:56 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'cinvestbacktest.spiders', 'FEED_URI': 'uno.json', 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'SPIDER_MODULES': ['cinvestbacktest.spiders'], 'BOT_NAME': 'cinvestbacktest', 'FEED_FORMAT': 'json', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage'}
2018-04-19 16:45:56 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2018-04-19 16:45:56 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, RedirectMiddleware, CookiesMiddleware, SplashCookiesMiddleware, SplashMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2018-04-19 16:45:56 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, SplashDeduplicateArgsMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2018-04-19 16:45:56 [scrapy] INFO: Enabled item pipelines: 
2018-04-19 16:45:56 [scrapy] INFO: Spider opened
2018-04-19 16:45:56 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-04-19 16:45:56 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-04-19 16:45:56 [scrapy] DEBUG: Crawled (404) <GET http://www.oddsportal.com/search/results/:69Dxbc61/> (referer: None)
2018-04-19 16:45:56 [scrapy] DEBUG: Ignoring response <404 http://www.oddsportal.com/search/results/:69Dxbc61/>: HTTP status code is not handled or not allowed
2018-04-19 16:45:56 [scrapy] INFO: Closing spider (finished)
2018-04-19 16:45:56 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 241,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 12816,
 'downloader/response_count': 1,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 4, 19, 14, 45, 56, 756377),
 'log_count/DEBUG': 3,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2018, 4, 19, 14, 45, 56, 473849)}
2018-04-19 16:45:56 [scrapy] INFO: Spider closed (finished)

Why?

Solution

It looks like the website gives a 404 error when opened with scrapy, while working correctly when viewed in a browser.

This kind of behavior usually means that there's something wrong with your request's headers.

In this case, it seems that simply setting a different user agent solves the problem.

Answered By - stranac