[FIXED] Extracting search results data using beautiful soup python

Issue

I am working with scraping data from linkedin. I was able to login with selenium in python and navigate it to the desired page I want to scrape. How ever I am facing challenges in gathering the data.

From the above image, you can see I have 22 results, I need to put those 22 results into csv file and defined by name, roles, company, months in role, etc.,

I am using selenium for this.

below is the html :

<a data-control-id="ù*ì/tH»=Ô¨w" data-control-name="view_lead_panel_via_search_lead_name" href="/sales/lead/ACwAAANJP7UBNIhbRLwsmhj5uyYeaOLU6ktthT4,NAME_SEARCH,Kksu?_ntb=ZnuIJGhESLutEc3rMsX2uQ%3D%3D" id="ember159" class="ember-view" data-lead-search-result="profile-link-st154">
              <span data-anonymize="person-name">Alain Cauwenberghs</span>
            </a>

below is the code:

name_html = soup1.find('a', {'class': 'ember-view'})
name_titles = []

for title in name_html:
    name_titles.append(title.text.strip())
 
print(name_titles)

Please help me, as I am not getting all the names in the list. I need all the names in the list. I am using beautiful soup.

Solution

If you just want to find the names you can use the data-anonymize attribute in order to find the spans with names.

soup1 = ...

titles = soup1.find_all("span", {"data-anonymize": "person-name"})
names = []

for name in titles:
    names.append(name.text)

print(names)

# with provided HTML sample it outputs:
# ['Alain Cauwenberghs']

Answered By - kggn