[FIXED] requests.get not returning anything

Issue

I am trying to retrieve a url using requests.get

import requests
from bs4 import BeautifulSoup

baseurl = "https://www.olx.com.eg/"

headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
}

r = requests.get('https://www.olx.com.eg/jobs/')
soup = BeautifulSoup(r.content, 'lxml')


product_list = soup.findAll('div',class_ = 'ads__item')

print(product_list)

but it returns an empty list because it does not even open the URL.

What is the issue here?

Solution

Add headers= parameter to requests.get:

import requests
from bs4 import BeautifulSoup

baseurl = "https://www.olx.com.eg/"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
}

r = requests.get("https://www.olx.com.eg/jobs/", headers=headers)
soup = BeautifulSoup(r.content, "lxml")


product_list = soup.findAll("div", class_="ads__item")

print(len(product_list))

Prints:

45

Answered By - Andrej Kesely