[FIXED] How to scrape data from stacker

Issue

I want scrape the Data from this Domain https://stacker.com/stories/1587/100-best-movies-all-time

Solution

I can get data only if I add header User-Agent

from bs4 import BeautifulSoup as BS
import requests

headers = { 
 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0', 
}

url = 'https://www.the-numbers.com/movie/Avengers-Endgame-(2019)#tab=cast-and-crew'

response = requests.get(url, headers=headers)

# --- response ---

#print(response.status_code)
#print(response.text[:1000])

soup = BS(response.text, 'html.parser')

all_items = soup.find_all('div', id="cast-and-crew") 
for item in all_items:
    print(item.get_text(strip=True, separator='\n'))

Result:

Lead Ensemble Members
Robert Downey, Jr.
Tony Stark/Iron Man
Chris Evans
Steve Rogers/Captain America
Mark Ruffalo
Bruce Banner/Hulk
Chris Hemsworth
Thor
Scarlett Johansson
Natasha Romanoff/Black Widow
Jeremy Renner
Clint Barton/Hawkeye
Don Cheadle
...

Answered By - furas