[FIXED] How to scrollIntoView() inside a specific dropdown(div) in Python

Issue

I am trying to scrape a website that requires me to first fill out certain dropdowns.
However, most of the dropdown selections are hidden and only appear in the DOM tree when I scroll down WITHIN the dropdown. Is there a solution I can use to somehow mimic a scroll wheel, or are there other libraries that could complement Selenium?

Solution

There are several ways to scroll an element into view but the most reliable one in Selenium is evaluating javascripts scrollIntoview() function.

For example I use this example for scraping twitch.tv in my blog:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.twitch.tv/directory/game/Art")
# find last item and scroll to it
driver.execute_script("""
let items=document.querySelectorAll('.tw-tower>div');
items[items.length-1].scrollIntoView();
""")

The javascript finds all "items" in the pagination page and scrolls the last one into view. In your case you should use:

driver.execute_script("""
let item = document.querySelector('DROPDOWN CSS SELECTOR');
item.scrollIntoView();
""")

You can read more about it here: https://scrapfly.io/blog/web-scraping-with-selenium-and-python/#advanced-selenium-functions

Answered By - Granitosaurus