[FIXED] stale element refernce issue and dom refreshing via explicit/implicit waits

Issue

I had an issue which I was able to resolve with explicit wait. My curiosity comes, what goes inside selenium webdriver that when I used implicitly wait for 300 seconds, it did continue to give stale element reference, but explicit wait with timeout 77sec it works without error. my code is below

for i in range(len(x_indexes)):
    x_indexes = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@data-asin]')))#--here i added explicit wait
    x_data_asin=x_indexes[i].get_attribute('data-asin')#on this line error stale ref was occuring
    if x_data_asin!="":
        
        #clicking to each item for getting iban values ->back page
        a_href_element_of_index=x_block_of_index.find_element(By.XPATH,'.//h2/a')

        a_href_element_of_index.click()
        a_isbn_element=driver.find_element(By.XPATH,'//span[contains(text(),"ISBN")]')
        x_isbn_element_parent=a_isbn_element.find_element(By.XPATH,'..')
        print(x_isbn_element_parent.get_attribute('textContent'))
        #driver.back()
        driver.execute_script('window.history.go(-1)')
        print(a_href_element_of_index,'a',sep='-->')
        #driver.implicitly_wait(300)
        --loop ends

when i loop each item inside x_indexes I click every anchor element, page directs to another page there i pull out data that i want, then driver.back or driver.execute_script('window.history.go(-1)') brings me back to the page where iteration continues in same way. I was having stale reference the element was not attached to page, I tried implicitly waiting till 300 after loop was ending line. The result was same error. So when I tried explicitly waiting wait=WebDriverWait(driver,timeout=77) the error stopped occuring. I wonder the logic behind waiting 300 seconds and 77 what lays behind in webdriver?

Solution

By navigating to another page all collected by Selenium web elements (they are actually references to a physical web elements) become no more valid since the web page is re-built when you open it again.
Now, driver.implicitly_wait is setting a timeout for find_element and find_elements methods only to search for the element(s) according to a given locator. The default implicitly_wait is set to 0. It means that if element is not immediately found on the page, Selenium will throw an exception. By setting implicitly_wait to another value it will continue searching for that element until it finds it or the timeout, the first of them.
But this will not help in case you are iterating over existing list of previously collected elements x_indexes since these references now no more relevant since the web page were re-rendered, as mentioned before.
But if you are using x_indexes = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@data-asin]'))) this will wait on the main page, were you getting back from the recently opened link, to visibility of these elements, collect the list again, and then you will be able to click on the next element.
Also, normally no need to set implicitly_wait or WebDriverWait to more than 20-30 seconds.

Answered By - Prophet