Tuesday, November 14, 2023

[FIXED] Issues with Python Scrapy script with multiple start urls

November 14, 2023 python, scrapy, selenium-webdriver No comments

Issue

I have a scraper written in Scrapy. It works perfectly fine as long as there is just one url in start_urls. As soon as I add 2 or more start urls, the behavior is erratic. The number of pages it scrapes is wrong, it stops midway etc. Each individual url works fine if entered individually. Any idea on what might be causing this? I have a large number of urls to scrape so can't do this one by one.

import scrapy
from selenium import webdriver
from scrapy.selector import Selector #used to pull urls
from scrapy.http import Request
from time import sleep
from random import random
from datetime import datetime



class UsedaudiSpider(scrapy.Spider):
    '''
    This spider basically crawls a number of audi dealerships and gets a list of all the used vehicles available there
    '''
    name = 'usedaudi'
    allowed_domains = [ 
        'www.audimv.com', 
        ]
    start_urls = [ 
        'https://www.audioxnard.com/used-inventory/index.htm',
        'https://www.audimv.com/used-inventory/index.htm',
        'https://www.montereyaudi.com/used-inventory/index.htm',
        ]

    def __init__(self):
        options = webdriver.ChromeOptions()
        options.add_argument('headless')
        self.driver = webdriver.Chrome(options=options)

    def parse(self, response):
        self.driver.get(response.url)
        sleep(2)
        try:
            but = self.driver.find_element_by_xpath('//button[@aria-label="switch to List view"]')
            but.click()
        except:
            pass
        sleep(1.5)
        for _ in range(10):
            self.driver.execute_script("window.scrollBy(0, 700);")
            sleep(0.25)

        
        sel = Selector(text=self.driver.page_source)

        listings = sel.xpath('//li[@class="box box-border vehicle-card vehicle-card-detailed vehicle-card-horizontal"]')
        for listing in listings:
            yr_brand = listing.xpath('.//span[@class="ddc-font-size-small"]/text()').extract_first()
            if yr_brand is not None:
                year = yr_brand.split(' ')[0]
                brand = yr_brand.split(' ')[1]
            else:
                year=''
                brand=''
            model= listing.xpath('.//h2/a/text()').extract_first()
            if model is not None:
                model = model.strip()
                base_model = model.split(' ')[0]
            else:
                base_model = ''

            link =  listing.xpath('.//h2/a/@href').extract_first()
            url2 = response.urljoin(link)
            price =  listing.xpath('.//span[@class="price-value"]/text()').extract_first()
            miles =  listing.xpath('.//li[@class="odometer"]/text()').extract_first()
            engine = listing.xpath('.//li[@class="engine"]/text()').extract_first()
            awd = listing.xpath('.//li[@class="normalDriveLine"]/text()').extract_first()
            stock = listing.xpath('.//li[@class="stockNumber"]/text()').extract_first()

            #dealership name
            dealership = self.driver.current_url
            dealership = dealership.replace('https://', '').split("/")[0].replace("www.", "").replace(".com", "")
            

            yield {
                'dealership':dealership,
                'year': year,
                'brand': brand,
                'base_model':base_model,
                'model_detail': model,
                'price': price,
                'miles': miles,
                'engine': engine,
                'awd': awd,
                'stock': stock,
                'link': url2,
            }

        #find the next link and click it if it exists
        try:
            next = self.driver.find_element_by_xpath('//li[@class="pagination-next"]/a')
            self.logger.info('NEXT IS:')
            print(next)
        except:
            next = None
            self.logger.info('No more pages to load.')

        if next is not None:
            next_url = next.get_attribute("href")
            self.logger.info('THE NEXT URL IS')
            self.logger.info(next_url)
            yield scrapy.Request(next_url, callback=self.parse)

    def close(self, reason):
        self.driver.quit()

Any help would be much appreciated

Solution

The reason it doesn't work with more than one url is because scrapy is asynchronous and selenium is not, so the two do not play nicely together.

It looks like you can get all the information you are looking for using each of the websites json API's.

For example:

import scrapy

class UsedaudiSpider(scrapy.Spider):
    name = 'usedaudi'

    def start_requests(self):
        urls = {'https://www.audioxnard.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory': 15,
                'https://www.audimv.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory' : 18,
                'https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory': 18}
        for k,v in urls.items():
            yield scrapy.Request(k)
            for i in range(1, 12):
                yield scrapy.Request(k + f"?start={i*v}")

    def parse(self, response):
        for vehicle in response.json()["pageInfo"]["trackingData"]:
            del vehicle["images"]
            yield vehicle

Partial Output:

2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 29, 'highwayFuelEfficiency'
: 33, 'indexPosition': 5, 'modelYear': 2022, 'odometer': 8894, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audimo
ntereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$29,243', 'finalPrice': '$29,243'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '29243.0', 'bodySt
yle': 'SUV', 'chromeId': '419441', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'FWD', 'engine': 'Ecotec Turbo DOHC SIDI w/VVT', 'engineSize': '1.3L', 'exteriorColor': 'Mosaic Black Meta
llic', 'fuelType': 'Gasoline', 'interiorColor': 'Jet Black', 'internetPrice': '29243.0', 'inventoryDate': '05/20/2023', 'inventoryType': 'used', 'link': '/used/Chevrolet/2022-Chevrolet-Trailblazer-3f906
e720a0e0a9a374c72e6c9699b00.htm', 'make': 'Chevrolet', 'model': 'Trailblazer', 'modelCode': '1TS56', 'newOrUsed': 'used', 'salePrice': '$29,243', 'status': 'live', 'stockNumber': 'A6583A', 'transmission
': 'CVT', 'trim': 'Activ', 'uuid': '3f906e720a0e0a9a374c72e6c9699b00', 'vin': 'KL79MVSL8NB047169'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 18, 'highwayFuelEfficiency'
: 24, 'indexPosition': 6, 'modelYear': 2022, 'odometer': 10569, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$49,536', 'finalPrice': '$49,536'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '49536.0', 'autod
ataCaId': 'CAD20FOS102E0', 'bodyStyle': 'SUV', 'chromeId': '423548', 'classification': 'primary', 'doors': '4 door', 'driveLine': '4WD', 'engine': 'EcoBoost V6', 'engineSize': '3L', 'exteriorColor': 'Ag
ate Black Metallic', 'fuelType': 'Gasoline', 'interiorColor': 'Ebony', 'internetPrice': '49536.0', 'inventoryDate': '04/22/2023', 'inventoryType': 'used', 'link': '/used/Ford/2022-Ford-Explorer-afa10f40
0a0e0a920f6a25900990218f.htm', 'make': 'Ford', 'model': 'Explorer', 'modelCode': 'K8G', 'newOrUsed': 'used', 'salePrice': '$49,536', 'status': 'live', 'stockNumber': 'A6496A', 'transmission': '10-Speed
Automatic', 'trim': 'ST', 'uuid': 'afa10f400a0e0a920f6a25900990218f', 'vin': '1FM5K8GC5NGA19118'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'indexPosition': 7, 'modelYear': 2022, 'odometer'
: 11751, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audimontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export':
 '426A54'}, 'pricing': {'salePrice': '$50,000', 'finalPrice': '$50,000'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '50000.0', 'autodataCaId': 'CAD20JES231A0', 'bodyStyle': 'SUV', 'chromeI
d': '422079', 'classification': 'primary', 'doors': '4 door', 'driveLine': '4WD', 'engine': 'I4 DOHC', 'engineSize': '2L', 'exteriorColor': 'Silver Zynith Clearcoat', 'fuelType': 'Hybrid', 'interiorColo
r': 'Black', 'internetPrice': '50000.0', 'inventoryDate': '05/31/2023', 'inventoryType': 'used', 'link': '/used/Jeep/2022-Jeep-Wrangler-785970a00a0e0a9a45e557c380356a6e.htm', 'make': 'Jeep', 'model': 'W
rangler', 'modelCode': 'JLXP74', 'newOrUsed': 'used', 'salePrice': '$50,000', 'status': 'live', 'stockNumber': 'A6639A', 'transmission': '8-Speed Automatic', 'trim': 'Unlimited Sahara 4xe', 'uuid': '785
970a00a0e0a9a45e557c380356a6e', 'vin': '1C4JJXP66NW162997'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 22, 'highwayFuelEfficiency'
: 31, 'indexPosition': 8, 'modelYear': 2022, 'odometer': 31646, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$45,372', 'finalPrice': '$45,372'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '45372.0', 'bodyS
tyle': 'Coupe', 'chromeId': '423697', 'classification': 'primary', 'doors': '2 door', 'driveLine': 'RWD', 'engine': 'I4 Turbocharged', 'engineSize': '2L', 'exteriorColor': 'Polar White', 'fuelType': 'Ga
soline', 'interiorColor': 'Cranberry Red', 'internetPrice': '45372.0', 'inventoryDate': '05/09/2023', 'inventoryType': 'used', 'link': '/used/Mercedes-Benz/2022-Mercedes-Benz-C-Class-06f1e84f0a0e0a923f3
f83f025f1adbf.htm', 'make': 'Mercedes-Benz', 'model': 'C-Class', 'modelCode': 'C300C', 'newOrUsed': 'used', 'salePrice': '$45,372', 'status': 'live', 'stockNumber': 'A6582A', 'transmission': '9-Speed Au
tomatic', 'trim': 'C 300', 'uuid': '06f1e84f0a0e0a923f3f83f025f1adbf', 'vin': 'W1KWJ8DBXNG107627'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 20, 'highwayFuelEfficiency'
: 27, 'indexPosition': 9, 'modelYear': 2019, 'odometer': 34592, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$29,896', 'finalPrice': '$29,896'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '29896.0', 'autod
ataCaId': 'CAC90SUS311F0', 'bodyStyle': 'SUV', 'chromeId': '402693', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'AWD', 'engine': 'Boxer H6 DOHC 24V', 'engineSize': '3.6L', 'exteriorCol
or': 'Magnetite Gray Metallic', 'fuelType': 'Gasoline', 'interiorColor': 'Slate Black', 'internetPrice': '29896.0', 'inventoryDate': '04/03/2023', 'inventoryType': 'used', 'link': '/used/Subaru/2019-Sub
aru-Outback-ce4bb9940a0e0a940b780b48a210da9f.htm', 'make': 'Subaru', 'model': 'Outback', 'modelCode': 'KDK', 'newOrUsed': 'used', 'salePrice': '$29,896', 'status': 'live', 'stockNumber': 'A6540TA', 'tra
nsmission': 'CVT Lineartronic', 'trim': '3.6R', 'uuid': 'ce4bb9940a0e0a940b780b48a210da9f', 'vin': '4S4BSENC0K3322606'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=36>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 120, 'highwayFuelEfficiency
': 112, 'indexPosition': 10, 'modelYear': 2018, 'odometer': 30407, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'au
dimontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': 'PLEASE_CALL'}, 'accountId': 'audimontereypeninsulaaoa', 'bodyStyle': 'Sedan', 'chromeId': '402455', 'class
ification': 'primary', 'doors': '4 door', 'driveLine': 'AWD', 'engine': 'Electric ZEV', 'exteriorColor': 'Gray', 'fuelType': 'Electric', 'inventoryDate': '05/31/2023', 'inventoryType': 'used', 'link': '
/used/Tesla/2018-Tesla-Model+3-785969d10a0e0a90410d65dadc6cfad2.htm', 'make': 'Tesla', 'model': 'Model 3', 'modelCode': 'MODEL3P', 'newOrUsed': 'used', 'salePrice': 'PLEASE_CALL', 'status': 'live', 'sto
ckNumber': 'L1435A', 'transmission': '1-Speed Automatic', 'trim': 'Performance', 'uuid': '785969d10a0e0a90410d65dadc6cfad2', 'vin': '5YJ3E1EB0JF086864'}
2023-06-01 21:48:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=54> (referer: None)

2023-06-01 21:48:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=162> (referer: None
)
2023-06-01 21:48:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18> (referer: None)

2023-06-01 21:48:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=198> (referer: None
)
2023-06-01 21:48:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=180> (referer: None
)
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 23, 'highwayFuelEfficiency'
: 28, 'indexPosition': 1, 'modelYear': 2021, 'odometer': 22334, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': 'PLEASE_CALL'}, 'accountId': 'audimontereypeninsulaaoa', 'autodataCaId': 'CAD10AUS021A0', 'bodyStyle': 'SUV',
'chromeId': '414146', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': 'Turbocharged', 'engineSize': '2L', 'exteriorColor': 'Gray', 'fuelType': 'Gasoline', 'interiorColo
r': 'Q5 45 Tfsi', 'inventoryDate': '05/31/2023', 'inventoryType': 'used', 'link': '/used/Audi/2021-Audi-Q5-78596e950a0e0a90410d65da3a70c139.htm', 'make': 'Audi', 'model': 'Q5', 'modelCode': 'FYGBAY', 'n
ewOrUsed': 'used', 'salePrice': 'PLEASE_CALL', 'status': 'live', 'stockNumber': 'A6514TA', 'transmission': '7-Speed Automatic S tronic', 'uuid': '78596e950a0e0a90410d65da3a70c139', 'vin': 'WA1AAAFY6M213
6321'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 23, 'highwayFuelEfficiency'
: 28, 'indexPosition': 2, 'modelYear': 2022, 'odometer': 5778, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audimo
ntereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$43,671', 'finalPrice': '$43,671'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '43671.0', 'autoda
taCaId': 'CAD20AUS022A0', 'bodyStyle': 'SUV', 'chromeId': '421631', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': 'Turbocharged', 'engineSize': '2L', 'exteriorColor':
 'Daytona Gray Pearl Effect', 'fuelType': 'Gasoline', 'interiorColor': 'Black', 'internetPrice': '43671.0', 'inventoryDate': '04/28/2023', 'inventoryType': 'used', 'link': '/used/Audi/2022-Audi-Q5+Sport
back-1ce360d50a0e087f612d9486ae32dcd1.htm', 'make': 'Audi', 'model': 'Q5 Sportback', 'modelCode': 'FYTCAY', 'newOrUsed': 'used', 'salePrice': '$43,671', 'status': 'live', 'stockNumber': 'L1417', 'transm
ission': '7-Speed Automatic S tronic', 'trim': '45 S Line Premium', 'uuid': '1ce360d50a0e087f612d9486ae32dcd1', 'vin': 'WA14AAFY1N2101331'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 22, 'highwayFuelEfficiency'
: 29, 'indexPosition': 3, 'modelYear': 2023, 'odometer': 10370, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$55,146', 'finalPrice': '$55,146'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '55146.0', 'autod
ataCaId': 'CAD30AUS022B0', 'bodyStyle': 'SUV', 'chromeId': '428302', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': 'Turbocharged', 'engineSize': '2L', 'exteriorColor'
: 'Black', 'fuelType': 'Gasoline', 'interiorColor': 'Black', 'internetPrice': '55146.0', 'inventoryDate': '04/28/2023', 'inventoryType': 'used', 'link': '/used/Audi/2023-Audi-Q5+Sportback-ce4bb76b0a0e0a
991091580f6a99df8d.htm', 'make': 'Audi', 'model': 'Q5 Sportback', 'modelCode': 'FYTCAY', 'newOrUsed': 'used', 'salePrice': '$55,146', 'status': 'live', 'stockNumber': 'L1425', 'transmission': '7-Speed A
utomatic S tronic', 'trim': '45 S Line Prestige', 'uuid': 'ce4bb76b0a0e0a991091580f6a99df8d', 'vin': 'WA16AAFY4P2002286'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'indexPosition': 4, 'modelYear': 2020, 'odometer'
: 29363, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audimontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export':
 '426A54'}, 'pricing': {'salePrice': '$37,934', 'finalPrice': '$37,934'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '37934.0', 'autodataCaId': 'CAD00AUS081A0', 'bodyStyle': 'SUV', 'chromeI
d': '411073', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': '4-Cylinder TFSI', 'engineSize': '2L', 'exteriorColor': 'Ibis White', 'fuelType': 'Gasoline', 'interiorCol
or': 'Black', 'internetPrice': '37934.0', 'inventoryDate': '04/03/2023', 'inventoryType': 'used', 'link': '/used/Audi/2020-Audi-Q5+e-4d6d42860a0e0a9270e381af1fc11a86.htm', 'make': 'Audi', 'model': 'Q5 e
', 'modelCode': 'FYB52Y', 'newOrUsed': 'used', 'salePrice': '$37,934', 'status': 'live', 'stockNumber': 'B2623', 'transmission': '7-Speed Automatic S tronic', 'trim': '55 Premium Plus', 'uuid': '4d6d428
60a0e0a9270e381af1fc11a86', 'vin': 'WA1E2AFY1L2065736'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'indexPosition': 5, 'modelYear': 2020, 'odometer'
: 19136, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audimontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export':
 '426A54'}, 'pricing': {'salePrice': '$41,034', 'finalPrice': '$41,034'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '41034.0', 'autodataCaId': 'CAD00AUS081A0', 'bodyStyle': 'SUV', 'chromeI
d': '411073', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': '4-Cylinder TFSI', 'engineSize': '2L', 'exteriorColor': 'Glacier White Metallic', 'fuelType': 'Gasoline',
'interiorColor': 'Black', 'internetPrice': '41034.0', 'inventoryDate': '05/07/2023', 'inventoryType': 'used', 'link': '/used/Audi/2020-Audi-Q5+e-fca178450a0e0a9353bb2b7fc9f8d5d1.htm', 'make': 'Audi', 'm
odel': 'Q5 e', 'modelCode': 'FYB52Y', 'newOrUsed': 'used', 'salePrice': '$41,034', 'status': 'live', 'stockNumber': 'A6490TA', 'transmission': '7-Speed Automatic S tronic', 'trim': '55 Premium Plus', 'u
uid': 'fca178450a0e0a9353bb2b7fc9f8d5d1', 'vin': 'WA1E2AFY6L2069166'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 19, 'highwayFuelEfficiency'
: 25, 'indexPosition': 6, 'modelYear': 2019, 'odometer': 15631, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$37,000', 'finalPrice': '$37,000'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '37000.0', 'autod
ataCaId': 'CAC90AUS011B0', 'bodyStyle': 'SUV', 'chromeId': '402761', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': '4-Cylinder TFSI', 'engineSize': '2L', 'exteriorCol
or': 'Samurai Gray Metallic', 'fuelType': 'Gasoline', 'interiorColor': 'Black', 'internetPrice': '37000.0', 'inventoryDate': '05/23/2023', 'inventoryType': 'used', 'link': '/used/Audi/2019-Audi-Q7-4f0c9
8a60a0e0a9116239af50d00b2a9.htm', 'make': 'Audi', 'model': 'Q7', 'newOrUsed': 'used', 'salePrice': '$37,000', 'status': 'live', 'stockNumber': 'A6620A', 'transmission': '8-Speed Automatic with Tiptronic
', 'trim': '45 Premium', 'uuid': '4f0c98a60a0e0a9116239af50d00b2a9', 'vin': 'WA1AHAF79KD021488'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 19, 'highwayFuelEfficiency'
: 25, 'indexPosition': 7, 'modelYear': 2019, 'odometer': 42866, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$33,026', 'finalPrice': '$33,026'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '33026.0', 'autod
ataCaId': 'CAC90AUS011A0', 'bodyStyle': 'SUV', 'chromeId': '402762', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': '4-Cylinder TFSI', 'engineSize': '2L', 'exteriorCol
or': 'Galaxy Blue Metallic', 'fuelType': 'Gasoline', 'interiorColor': 'Black', 'internetPrice': '33026.0', 'inventoryDate': '04/28/2023', 'inventoryType': 'used', 'link': '/used/Audi/2019-Audi-Q7-7056cf
330a0e081d028cef233564f345.htm', 'make': 'Audi', 'model': 'Q7', 'newOrUsed': 'used', 'salePrice': '$33,026', 'status': 'live', 'stockNumber': 'K0134L', 'transmission': '8-Speed Automatic with Tiptronic'
, 'trim': '45 Premium Plus', 'uuid': '7056cf330a0e081d028cef233564f345', 'vin': 'WA1LHAF72KD041008'}
2023-06-01 21:48:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.montereyaudi.com/apis/widget/INVENTORY_LISTING_DEFAULT_AUTO_USED:inventory-data-bus1/getInventory?start=18>
{'address': {'accountName': 'Audi Monterey Peninsula', 'city': 'Seaside', 'country': '93955', 'postalCode': '93955', 'state': 'CA'}, 'certified': False, 'cityFuelEfficiency': 19, 'highwayFuelEfficiency'
: 25, 'indexPosition': 8, 'modelYear': 2019, 'odometer': 54321, 'optionCodes': [''], 'dealerCodes': {'dealertrack-post': 'audimontereypeninsul', 'aoa': '426A54', 'dtid': '10778', 'dt-dr-profile': 'audim
ontereypeninsul', 'aoaddc-cdc': '426A54', 'aoa-export': '426A54'}, 'pricing': {'salePrice': '$35,525', 'finalPrice': '$35,525'}, 'accountId': 'audimontereypeninsulaaoa', 'askingPrice': '35525.0', 'autod
ataCaId': 'CAC90AUS011E0', 'bodyStyle': 'SUV', 'chromeId': '402353', 'classification': 'primary', 'doors': '4 door', 'driveLine': 'quattro', 'engine': 'TFSI V6 DOHC', 'engineSize': '3L', 'exteriorColor'
: 'Cobra Beige Metallic', 'fuelType': 'Gasoline', 'interiorColor': 'Pistachio Beige', 'internetPrice': '35525.0', 'inventoryDate': '04/28/2023', 'inventoryType': 'used', 'link': '/used/Audi/2019-Audi-Q7
-720bfd4f0a0e0a94188c27d7c1cf75ae.htm', 'make': 'Audi', 'model': 'Q7', 'modelCode': 'T', 'newOrUsed': 'used', 'salePrice': '$35,525', 'status': 'live', 'stockNumber': 'A6043AL', 'transmission': '8-Speed
 Automatic with Tiptronic', 'trim': '55 Prestige', 'uuid': '720bfd4f0a0e0a94188c27d7c1cf75ae', 'vin': 'WA1VAAF72KD041460'}

Answered By - Alexander

This Answer collected from stackoverflow and tested by PythonFixing community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Tuesday, November 14, 2023

[FIXED] Issues with Python Scrapy script with multiple start urls

Issue

Solution

0 comments:

Post a Comment

Popular Posts

Labels