[FIXED] How to retrieve yahoo search results?

Issue

I am trying to search Yahoo for a query using this code:

import requests
from bs4 import BeautifulSoup

query = "deep"
yahoo = "https://search.yahoo.com/search?q=" + query + "&n=" + str(10)
raw_page = requests.get(yahoo)

soup = BeautifulSoup(raw_page.text)

for link in soup.find_all(attrs={"class": "ac-algo fz-l ac-21th lh-24"}):
    print (link.text, link.get('href'))

But this does not work and the result is empty. How can I get 10 first search results?

Solution

Here are the main problems with your code:

• When using Beautiful soup you should always include a parser (e.g. BeautifulSoup(raw_page.text, "lxml"))

• You were searching for the wrong class, it's " ac-algo fz-l ac-21th lh-24" not "ac-algo fz-l ac-21th lh-24" (notice the space in the begining)

All in all your code should look like this:

import requests
from bs4 import BeautifulSoup

query = "deep"
yahoo = "https://search.yahoo.com/search?q=" + query + "&n=" + str(10)
raw_page = requests.get(yahoo)

soup = BeautifulSoup(raw_page.text, "lxml")
for link in soup.find_all(attrs={"class": " ac-algo fz-l ac-21th lh-24"}):
    print(link.text, link.get('href'))

Hope this helps

Answered By - Nazim Kerimbekov