Issue
When i run this code block, it runs successfully but returns empty result for the name
field. please help me check, what am I missing?
Here's my block code
import scrapy
class TruckspiderSpider(scrapy.Spider):
name = 'truckspider'
allowed_domains = ['www.quicktransportsolutions.com']
start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']
def parse(self, response):
containers = response.css('div.col-md-4')
for container in containers:
yield
{
'name': container.css('a::text').get(),
}
Here is the result I get
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Closing spider (finished)
2023-02-01 19:17:42 [scrapy.extensions.feedexport] INFO: Stored json feed (55 items) in: truck.json
2023-02-01 19:17:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 265,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 4912,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'elapsed_time_seconds': 1.916927,
'feedexport/success_count/FileFeedStorage': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2023, 2, 1, 18, 17, 42, 902380),
'httpcompression/response_bytes': 22809,
'httpcompression/response_count': 1,
'item_scraped_count': 55,
'log_count/DEBUG': 57,
'log_count/INFO': 11,
'log_count/WARNING': 1,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2023, 2, 1, 18, 17, 40, 985453)}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Spider closed (finished)
Solution
@Md. Fazlul Hoque thanks for all your time and responses.
I finally got my way around it. I gave my local machine access to this file: 2023-02-02 10:54:17 [filelock] DEBUG: Attempting to acquire lock 2733195998640 on C:\Users\ChiNedu\AppData\Local\Programs\Python\Python310\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
in my AppData folder and it worked like magic. So happy.
Answered By - Miracle
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.