[FIXED] python selenium loop through some links

Issue

I have an array of links that I am trying to access to every link and print something from it, then return to the main page and access the second link, then do the same until I finish all links in the array.

What happens is that the first link is the only one that works, like if all the links in the array are gone. I get the error:

File "e:\work\MY CODE\scraping\learn.py", line 25, in theprint link.click()

    from selenium import webdriver
from selenium.webdriver.common import keys
#it make us able to use keybored keys like enter ,esc , etc....
from selenium.webdriver.common.keys import Keys
import time

#make us can wait for event to happen until run the next line of code
from selenium.webdriver.common.by import By
from selenium.webdriver.remote import command
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#get the google chrome driver path
PATH="E:\work\crom\chromedriver.exe"
#pass the pass to selenium webdriver method
driver=webdriver.Chrome(PATH)
#get the link of the site we want
driver.get("https://app.dealroom.co/companies.startups/f/client_focus/anyof_business/company_status/not_closed/company_type/not_government%20nonprofit/employees/anyof_2-10_11-50_51-200/has_website_url/anyof_yes/slug_locations/anyof_france?sort=-revenue")

#wait for the page to load
time.sleep(5)
#get the links i want to get info from
the_links=driver.find_elements_by_class_name("table-list-item")

#function that go the link and print somethin and return to main page
links=[]
the_links=driver.find_elements_by_class_name("table-list-item")
for link in the_links:
      links.append(link.get_attribute('href'))

for link in links:
      driver.get(link)
      website=driver.find_element_by_class_name("item-details-info__url")
      print(website.text)
      driver.back()
      time.sleep(3)
      

Solution

Your code will throw a stale element reference error because when you navigate to the next page, the variable holding any elements of the previous page will become unusable.

So what you need to do is either store all elements in array and then loop through it like this:

links=[]
the_links=driver.find_elements_by_class_name("table-list-item")
for link in the_links:
    links.append(link.get_attribute('href'))

for link in links:
    driver.get(link)
    print("do something on this link")

Or you can use a while loop in your current and after driver.back() again populate the the_links variable.

Answered By - Muhammad Suleman