[FIXED] scrapy : spider not found (name or get no data ? )

Issue

I'm trying to use scrapy to get product name. When I've built the scrapy environment,ant start my first try, It failed. Output shows : KeyError:"spider not found : yahoo " What's wrong with the code ? I hope I could get some tip. Thanks.

import scrapy
from bs4 import BeautifulSoup

class yahooCrawler(scrapy.Spider):
   name = "yahoo"
   start_urls = ['https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F']
   def parse(self,response):
       res = BeautifulSoup(response.body)
       for product in res.find("BaseGridItem__title___2HWui"):
           print(product)
           yield product

enter image description here

Solution

Now your code is working fine. For for loop, you need to call find_all method and element selection was incorrect. Finally, you need to use .text to extract string value as text.

Code:

import scrapy
from bs4 import BeautifulSoup

class yahooCrawler(scrapy.Spider):
   name = "yahoo"
   start_urls = ['https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F']
   def parse(self,response):
       res = BeautifulSoup(response.body)
       for product in res.find_all("span",class_="BaseGridItem__title___2HWui"):
           #print(product.text)
           yield {'product':product.text}    

Output:

{'product': '乙)綠油精 Green Oil 5g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '沙威隆-抗菌潔淨洗手乳250ml【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '凡士林原味瓶裝護脣膏7g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '乙)曼秀雷敦 35g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'AIRWAVES 超涼薄荷無糖口香糖【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '媚點media 自轉式唇線筆 RS-3【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏袖珍面紙10抽30入(Shibainc)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Airwave Super 極酷嗆涼無糖口香糖-紫冰野莓口味【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏壓邊化妝棉100片'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏小蒼蘭淨潤沐浴乳1000ml'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏纖細柔軟牙刷(軟毛)1入'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'HK-462 2入修眉刀L型【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '茶山房 蘭花皂 100G【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏 Collagen 活妍肌密眼部緊緻精華 20ml'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '澎澎MAN香浴乳補充包-茶樹精油700g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '沙威隆抗菌潔淨沐浴乳-天然茶樹精油1000ml【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏身體磨砂膏200g-洋甘菊蘆薈'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '日本JACKS 齒間刷15入3號-(S)【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏潔膚柔濕巾10片3包入-薰衣草'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '美粒果白葡萄汁特加清新蘆薈粒 500ml【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏Q10緊緻柔膚乳液480ml'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Plantur 39植物與咖啡因洗髮露-染燙及受損髮質250ml【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '固力果 巧克力棒 40g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '銀離子除臭噴霧(鞋內專用)【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '蘇菲超熟睡超薄細緻棉柔41.5cm 8片裝【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '3M舒適繃 20片入【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Naturals by Watsons 茶樹舒緩精油 10ml'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏備長炭牙刷(軟毛)1入'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏纖細柔軟牙刷(軟毛)4+1支'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏多用途濕巾90片'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '一匙靈亮彩超濃縮洗衣粉3D增豔力縯粉花朵香氛1.9kg【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '日本EBISU 柯南 3~6歲兒童牙刷【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '康乃馨透氣超薄護墊微香40片4包【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏廚房萬用紙巾112組6捲入(Shibainc)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏超細滑圓線護理牙線棒50支(附隨身盒)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏抽取式衛生紙110抽10包入'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Naturals by Watsons 精萃玫瑰潤唇膏 4.5g (new)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '樂絲朵 立體持捲雙效乳 200ml【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '3M 細滑牙線棒150支附隨身盒【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '阿葵亞淨水賦活涼感洗潤組(屈臣氏獨家)(600ml+600ml)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Miine五角乳膠粉撲/10入【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'GATSBY塑型噴霧45g(金)【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏抗菌潔淨柔濕巾20片(綠圓點)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Eclipse 易口舒 無糖薄荷錠 繽紛野莓 31g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '保力達 葡萄C錠【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '丸三 五層可撕型敷面化妝棉 80片【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '乙)綠油精 10g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏潔膚柔濕巾10片3包入-櫻花'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏抽取式廚房萬用紙巾100抽3包入'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏旅行包面紙100抽3包(鳳梨西瓜)'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '3M Nexcare 深層潔淨抗痘洗面乳  100g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '白蘭氏 旭沛人蔘蜆精 60ml 15瓶裝【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '3M  立體密實袋大型15入【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '3M 立體密實袋中型25入【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '『美好人生』比爾A+幼兒營養穀奶(900g/罐)【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '『美好人生』紅豆紅棗八寶飲粉(600g/2罐)【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '去味大師竹木香-淡雅茉莉【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'Alpecin 咖啡因洗髮露 250ML【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': 'GATSBY 造型髮雕霜(強黏性) 200g【屈臣氏】'}
2021-11-28 15:47:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://tw.mall.yahoo.com/search/product?p=%E5%B1%88%E8%87%A3%E6%B0%8F>
{'product': '屈臣氏圓線護理牙線棒50支4盒'}
2021-11-28 15:47:49 [scrapy.core.engine] INFO: Closing spider (finished)
2021-11-28 15:47:49 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 341,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 60279,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 1.335299,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 11, 28, 9, 47, 49, 895142),
 'httpcompression/response_bytes': 800701,
 'httpcompression/response_count': 1,
 'item_scraped_count': 60,

             

Answered By - Fazlul