[FIXED] How to know difference between select and find

Issue

I want to extract the info from each restaurant from this webpage: https://orderappetit.com/restaurants-search

import requests
from bs4 import BeautifulSoup

url = 'https://orderappetit.com/restaurants-search'

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) '\
       'AppleWebKit/537.36 (KHTML, like Gecko) '\
       'Chrome/75.0.3770.80 Safari/537.36'}

response = requests.get(url,headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

restaurants = soup.find(class_='ant-space-item')
print(restaurants)

Output

[]

I would like to know why my statement find is not working and how can I know if I need to use select or find

Solution

The page is loaded dynamically. There is nothing in the static html. You have to fetch the data from the api.

It's up to you to figure out the paramters you need by looking in the Dev Tools:

import requests
import pandas as pd

url = 'https://api.orderappetit.com/api/customers/stores'
payload ={
    'page': '1',
    'limit': '12',
    'location': '609b7a404575ff2493d02d54',
    'isRandom': '1',
    'type': ''
 }


jsonData = requests.get(url, params=payload).json()
stores = jsonData['stores']

df = pd.DataFrame(stores)

Output:

print(df['name'])
0             Orazio's by Zarcone 
1     Carbone's Pizza | South Park
2                       Colter Bay
3                   Saigon Bangkok
4      Marco’s Italian Restaurant 
5                   Lakeshore Cafe
6       Cookies and Cream | Seneca
7         Deep South Taco | Hertel
8            Macy's Place Pizzeria
9                  Wise Guys Pizza
10                     Tappo Pizza
11      Aguacates Mex Bar & Grill 
Name: name, dtype: object

Or to search for specific food and choose dilery/pickup:

search = 'pizza'

url = 'https://api.orderappetit.com/api/customers/stores/search'
payload ={
'generic': search,
'type': 'delivery',
'page': '1',
'limit': '8',
'location': '609b7a404575ff2493d02d54'}

jsonData = requests.get(url, params=payload).json()
stores = jsonData['stores']

df = pd.json_normalize(jsonData, record_path=['stores'])

Output:

print(df['store.name'])
0                Macy's Place Pizzeria
1                     Prohibition 2020
2        Trattoria Aroma Williamsville
3       Sidelines Sports Bar and Grill
4          Marco’s Italian Restaurant 
5                 Orazio's by Zarcone 
6    Butera's Craft Beer & Craft Pizza
7                      Allentown pizza
Name: store.name, dtype: object

Answered By - chitown88