[FIXED] Selenium/BeautifulSoup - WebScrape this field

Issue

My code runs fine and prints the title for all rows but the rows with dropdowns.

For example, row 4 has a dropdown if clicked. I implemented a 'try' which would in theory click the dropdown, to then pull the titles.

But when i execute click() and try to print, for the rows with these drop downs, they are not printing.

Expected output- Print all titles including the ones in dropdown.

A user has submitted an answer on this link StackOverFlowAnswer but the format of his answer was different and I do not know how to add fields such as date, time, chairs, or the field on the top which says "On demand" with his approach

Any approach would be appreciated, would like to put into a dataframe. Thanks

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.action_chains import ActionChains
import time
driver = webdriver.Chrome()
actions = ActionChains(driver)

driver.get('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')
time.sleep(4)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')

new_titles = set()

productlist=driver.find_elements_by_xpath("//div[@class='card item-container session']")
for property in productlist:
    actions.move_to_element_with_offset(property,0,0).perform()
    time.sleep(4.5)
    sessiontitle=property.find_element_by_xpath(".//h4[@class='session-title card-title']").text
    #print(sessiontitle)
    ifDropdown=property.find_elements_by_xpath(".//*[@class='item-expand-action expand']")
    if(ifDropdown):
        ifDropdown[0].click()
        time.sleep(4)
        open_titles = driver.find_elements_by_class_name('card-title')
        for open_title in open_titles:
            title = open_title.text
            if(title not in new_titles):
                print(title)
                new_titles.add(title)

Solution

Your problem is with driver.find_elements_by_class_name('item-expand-action expand') command. The find_elements_by_class_name('item-expand-action expand') locator is wrong. Those web elements have multiple class names. To locate these elements you can use css_selector or XPath.
Also since there are several elements with dropdowns, to perform clicks on them you should iterate over them. You can not perform .click() on a list of web elements.
So your code should be like this:

ifDropdown=driver.find_elements_by_css_selector('.item-expand-action.expand')
for drop_down in ifDropdown:
    drop_down.click()
    time.sleep(0.5)

Alternatively to the css_selector above you can use XPath as well:

ifDropdown=driver.find_elements_by_xpath('//a[@class="item-expand-action expand"]')

UPD
If you wish to print the added, new titles you can do this:

ifDropdown=driver.find_elements_by_css_selector('.item-expand-action.expand')
for drop_down in ifDropdown:
    drop_down.click()
    time.sleep(0.5)
newTitles=driver.find_elements_by_class_name('card-title')
for new_title in newTitles:
    print(new_title.text)

Here after expanding all the dropdown elements I'm getting all the new titles and then iterate over that list printing each element text.
driver.find_elements_by_class_name returns a list of web elements. You can not apply .text on a list, you have to iterate over list elements getting each single element text each time.
UPD2
The entire code opening dropdowns and printing their inner titles can be like this:
I'm doing this with Selenium, not mixing with bs4.

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.common.action_chains import ActionChains

import time
driver = webdriver.Chrome()
actions = ActionChains(driver)

driver.get('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')
time.sleep(4)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')

new_titles = set()

productlist=driver.find_elements_by_xpath("//div[@class='card item-container session']")
for property in productlist:
    actions.move_to_element(property).perform()
    time.sleep(0.5)
    sessiontitle=property.find_element_by_xpath(".//h4[@class='session-title card-title']").text
    print(sessiontitle)
    ifDropdown=property.find_elements_by_xpath(".//*[@class='item-expand-action expand']")
    if(ifDropdown):
        ifDropdown[0].click()
        time.sleep(4)
        open_titles = driver.find_elements_by_class_name('card-title')
        for open_title in open_titles:
            title = open_title.text
            if(title not in new_titles):
                print(title)
                new_titles.add(title)

Here I am checking if there is a dropdown. If do, I open it. Then getting all the currently existing opened titles. Per each such title I validate if it is new or was opened previously. If the title is new, not existing in the set I print it and add it to the set.

Answered By - Prophet