[FIXED] Selenium Webdriver cannot find element which loads later than DOM

Issue

I've been trying to webscrap this page using selenium, and as you may notice on the first load a pop-up appears about agreeing on cookies. This pop-up appears around a second later that the rest of the DOM. The problem is, that even after adding a sleep or WebDriverWait the driver still cannot click on the Accept button from the pop-up.

Inspecting the page gives that the Xpath of the element is /html/body/div/div[2]/div[4]/button[1] so I tried doing it as:

driver.get("https://www.planespotters.net/airlines")
time.sleep(5)
driver.find_element(By.XPATH, '/html/body/div/div[2]/div[4]/button[1]').click()

with no effect. I get that the element does not exist.

I tried it even with:

driver.execute_script('document.querySelector("#notice > div.message-component.message-row.button-container > button.message-component.message-button.no-children.focusable.primary.sp_choice_type_11").click()')

and

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div/div[2]/div[4]/button[1]'))).click()

Any suggestions on how to bypass this pop-up?

Solution

If you scroll up on devtools you'll notice your modal window is in an iframe. I queried the 'planespotters' header image (because it's the top item) and you can see it quite clearly:

For selenium switch to your iframe first - this finds the iframe wher the url contains 'privacy':

WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(@src,'privacy')]")))

Then you can complete your action. Be sure to keep that sync in there too.

Then you can switch back to your main window and carry in with your test

driver.switch_to_default_content()

Answered By - RichEdwards