[FIXED] Following Sibling not working in selenium python

Issue

The part that I want to scrape is something like this:

<dl class="some class">
    <dt> <strong>Text1</strong></dt>
    <dd> Result1</dd>
    <dt> <strong>Text2</strong></dt>
    <dd> Result2</dd>
    <dt> <strong>Text3</strong></dt>
    <dd> Result3</dd>
    <dt> <strong>Text4</strong></dt>
    <dd> Result4</dd>
    .  .  .

What I am doing currently is:

e = driver.find_element_by_xpath("//*[contains(text(), 'Text3')]")

When I do print(e.text) it succesfully prints the Text3. What I want is Result3. When I do this:

driver.find_element_by_xpath("//*[contains(text(), 'Text3')]/following-sibling::dd")

This shows an error with NoSuchElementException. I want the Result which is next to a particular text.

I have also tried this:

for i in driver.find_elements_by_class_name("some class"):
    print(i.find_element_by_xpath("./dt[.='Text3']/following-sibling::dd").text)

still shows NoSuchElementException.

Solution

The text Text3 is not in the <dt> tag but in its child <strong> tag. element.text gives you the correct text because element.text returns all the text in the element and its decedents, but it's not good enough for xpath.

You need to get <dt> which has a child element with text Text3. Note that following-sibling returns all the following siblings, so you need to use index 1 to point to the first sibling

//dt[strong[contains(text(), 'Text3')]]/following-sibling::dd[1]

Answered By - Guy