[FIXED] trying to scrape youtube using scrapy and get no data

Issue

so i am learning scrapy and would like to get data from a youtube channel

https://www.youtube.com/c/JohnWatsonRooney for example

so i open up the shell and fetch it and for example i would like to get the subcriber count i use response.xpath('//*[@id="subscriber-count"]') i tried with the css selector and many other things nothing seems to be working help !

Solution

The url is entirely dynamic that's why you can't pull the data. You have to use either an automation tool something like selenium or API (Youtube provides API). As you want to grab subscriptions count right now.So I use selenium as an example

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

URL = 'https://www.youtube.com/c/JohnWatsonRooney'
driver.get(URL)
time.sleep(3)

subcriber_count=driver.find_element(By.XPATH,'//*[@id="subscriber-count"]').text
print(subcriber_count)

Output:

A device attached to the system is not functioning. (0x1F)

26.5K subscribers

Answered By - F.Hoque