[FIXED] Python Selenium Finding multiple elements at once

Issue

I am trying to find an element on website and based on which element I find I want to run certain code. I was trying to do something like this:

                elements = [
                    (By.XPATH, "//div[@class='element_1'][contains(text(),'No data')]"),
                    (By.XPATH,
                     '//div[@ng-repeat="element_2"]/adns-data-module/div[@class="cls_1"]'),
                    (By.CSS_SELECTOR, "table.table_1"),
                    (By.CLASS_NAME, "Border_1")]

                for locator in elements:
                    try:
                        print("Trying to verify locator" + str(locator))
                        WebDriverWait(driver, 10).until(EC.presence_of_element_located(locator))
                        print("Locator is present")
                    except:
                        print("Locator was not found")

which is working fine to find an element but it is finding one element by one. What I need is to find which element occured on the website all at once (I want the code to be faster). So I created these functions in order to find which element occured on the website faster (running multiple proccesses at once):

import asyncio
# Define a coroutine to check the presence of an element
async def check_element(driver, locator):
    try:
        element = await asyncio.wait_for(
            WebDriverWait(driver, 10).until(
                EC.presence_of_element_located(locator)), timeout=10)
        return locator, element
    except:
        return None

# Define a function to run the coroutines in parallel and return the first result
async def find_first_element(driver, elements):
    tasks = []
    for locator in elements:
        print(locator)
        tasks.append(asyncio.ensure_future(check_element(driver, locator)))
    done, _ = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    for task in done:
        result = task.result()
        if result is not None:
            return result
    return None


    # Start the event loop and run the function to find the first element
    async def main_2(driver, elements):
    
        result = await find_first_element(driver, elements)
        if result is not None:
            print(f"The first element found is {result[0]}")
            print(f"The element's text is: {result[1].text}")
        else:
            print("No element was found")
    
        def run_main_2(driver, elements):
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            loop.run_until_complete(main_2(driver, elements))

and call them like this from a function inside a class:

    def start_button3_callback(self):      
        # create a new thread to run the button3_callback function on a different thread in order to make it not close the webdriver after the code is finished
        t = threading.Thread(target=self.button3_callback)
        t.start()
    
    def button3_callback(self):
                            options = webdriver.ChromeOptions()
                            options.add_argument("--disable-exit-on-background")
                            driver = webdriver.Chrome(options=options)

                        # check multiple elements at once
                        if __name__ == "__main__":
                            run_main_2(driver, elements)

Which always result in "No element was found" and I do not know why. Because I know that at least one of those elements is present on the website and the previous code is working.

Solution

I think what you want is EC.any_of(), e.g.

wait.until(EC.any_of(
    EC.visibility_of_element_located((By.XPATH, "//div[@class='element_1'][contains(text(),'No data')]")),
    EC.visibility_of_element_located((By.XPATH, "//div[@ng-repeat='element_2']/adns-data-module/div[@class='cls_1']")),
    EC.visibility_of_element_located((By.CSS_SELECTOR, "table.table_1")),
    EC.visibility_of_element_located((By.CLASS_NAME, "Border_1"))
))

That will look for all of the elements at once.

NOTE: Regarding EC.presence_of_element_located(), presence means that the element is in the DOM but not necessarily visible or ready to be interacted with. If you want to click an element, use EC.element_to_be_clickable(). If you want to interact with an element, e.g. get .text, then use EC.visibility_of_element_located(). Presence may work most of the time but is risky because if the page loads slowly, you're more likely to run into an ElementNotInteractableException causing your script to fail.

---

Having said that, I think this is not the right approach. Clearly these elements are very different, e.g. one is a TABLE, others seem to be messages/errors, etc. I'm assuming you're planning to do something different based on whichever of these elements exists. The problem is that once you've gone down this path, how do you tunnel yourself back out and know which element was found? I think the better approach is to wait for an element on the page that you know will always be there. We use that to make sure that the page is completely loaded so we don't need to use WebDriverWait for all 4 potential elements because 3 of them won't be there, wasting time. Then you walk through your list of elements, testing each using .find_elements() to see if they are present and then execute code based on which element was there.

locator = (By.CSS_SELECTOR, "some element that's always there") # use this to make sure the page is loaded
wait.until(EC.visibility_of_element_located(locator))

if driver.find_elements(By.XPATH, "//div[@class='element_1'][contains(text(),'No data')]"):
    no_data_scenario()
elif driver.find_elements(By.XPATH, "//div[@ng-repeat='element_2']/adns-data-module/div[@class='cls_1']"):
    do_something()
elif driver.find_elements(By.CSS_SELECTOR, "table.table_1"):
    table_scenario()
elif driver.find_elements(By.CLASS_NAME, "Border_1"):
    do_something_else()

You'll need to define each of the methods called inside the if statements but this should be very fast and take care of your scenario.

Answered By - JeffC