[FIXED] Beautifulsoup: correct target, wrong output

Issue

I need the data-score value for a bunch of ratings. It is in a span with class index__stars__nfK6S inside a div with class index__block__7hodp

I wrote this for-loop:

         for rating in rating_parents:
            label = rating.find('h4', class_='index__title__Rq0Po')
            test = rating.find_next('div', class_='index__block__7hodp')
            print(label.text)
            if test is not None:
                #data_score = test.get('data-score')
                data_score = test.find('span', class_='index__stars__nfK6S')
                print(data_score.get('data-score'))
            else:
                print('none')

But the output does not match with the DOM. So for example the data-score in the DOM for Work-Life-Balance is 5 (Screenshot) but the output is 4.

Following examplary output:

Es geht immer besser aber auch deutlich schlechter #print(rating_title)
Work-Life-Balance # print(label.text)
4
Karriere/Weiterbildung
4 print(data_score.get('data-score'))

I don't understand the mismatch.

The for-loop is nested into another one, and maybe this causes the error. But I don't see how. This is the whole for-loop

for rating_article in rating_articles_1:
rating_parent_container = rating_article.find('div', class_='index__content__HTaY9')
if rating_parent_container is not None:
    rating_container = rating_parent_container.find('div', class_='index__reviewBlock__I8pdb')
    if rating_container is not None:
        rating_title = rating_container.find('h3')
        print(rating_title)
        rating_parents = rating_container.find_all('div', class_='index__factor__Mo6xW')
        for rating in rating_parents:
            label = rating.find('h4', class_='index__title__Rq0Po')
            test = rating.find_next('div', class_='index__block__7hodp')
            print(label.text)
            if test is not None:
                #data_score = test.get('data-score')
                data_score = test.find('span', class_='index__stars__nfK6S')
                print(data_score.get('data-score'))
            else:
                print('none')

Solution

I think this class-based detection is not the best way as they're not unique for each div. Try XPath navigation instead. Copy element's XPath and check of it's universal (meaning that page structure does not change).

Answered By - dmgzh