[FIXED] Unable to retrieve video url in a Twitter's post using Selenium

Issue

I am trying to scrape twitter using Selenium in Python. However, I am unable retrieve the video clip's download/streaming url.

Refer below tweet for instance.

https://twitter.com/Tesla/status/1711184330792579093

Is the video source url directly available in the page for me to retrieve?

Thank you in advance.

Solution

Refer the below code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://twitter.com/Tesla/status/1711184330792579093")
wait = WebDriverWait(driver, 10)
# Accept cookies
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Accept all cookies']"))).click()
# capture and print the href attribute of <a> node
link = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@data-testid='card.layoutLarge.media']//following::a[1]"))).get_attribute("href")
print(link)

Console output:

https://twitter.com/i/broadcasts/1YqxoDNrdYZKv

Process finished with exit code 0

Answered By - Shawn