Issue
I am a social scientist dabbling in web scraping. I have been trying to scrape the links from this Page The links for daily news is arranged within a table structure. This is my code
options = Options()
options.add_argument("--disable-notifications")
driver = webdriver.Chrome(driverLocation,chrome_options=options)
browser = webdriver.Chrome()
url='https://economictimes.indiatimes.com//archive/year-2021,month-1.cms'
browser.get(url)
table = browser.find_element(By.ID,"calender");
rows=table.find_element(By.TAG_NAME, "tr")
td=rows.find_element(By.TAG_NAME, "td")
print(td.get_attribute('href'))
The get_attribute
returns nothing .I have tried a lot of other combinations -by Xpath etc but it's not working for me. Can somebody help me with this?
Solution
In case you meant to scrape all the links under td a tag, You can try the below solution.
Code :
browser.maximize_window()
wait = WebDriverWait(browser, 30)
browser.get("https://economictimes.indiatimes.com/archive/year-2021,month-1.cms")
hrefs = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table#calender td a")))
for href in hrefs:
print(href.get_attribute('href'))
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Output :
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44197.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44198.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44199.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44200.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44201.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44202.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44203.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44204.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44205.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44206.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44207.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44208.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44209.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44210.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44211.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44212.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44213.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44214.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44215.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44216.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44217.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44218.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44219.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44220.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44221.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44222.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44223.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44224.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44225.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44226.cms
https://economictimes.indiatimes.com/archivelist/year-2021,month-1,starttime-44227.cms
Answered By - cruisepandey
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.