[FIXED] Beautiful Soup not finding HTML that exists with file

Issue

I'm looking at the Netflix jobs page: https://jobs.netflix.com/search

url = "https://jobs.netflix.com/search"

response=requests.get(url)
response = response.content
soup= BeautifulSoup(response, 'html.parser')

jobList = soup.find_all("div", {"class" : "css-160787d exb5qdx0"})

The last method is returning None and I don't understand why. I am reading the file in properly because I have checked.

I have tried both the find and find all with no success. I looked into using selenium but the website doesn't seem dynamic.

Solution

That page is being hydrated from an API endpoint, via an XHR call. Here is one way of obtaining that information:

import requests
import pandas as pd

headers= {
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}
url = 'https://jobs.netflix.com/api/search'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['records', ['postings']])
print(df)

Result in terminal:

    search_text     text    lever_id    state   updated_at  created_at  team    slug    location    external_id     ...     _index  _type   _score  _version    _explanation    sort    id  subteam     organization    alternate_locations
0   Amsterdam, Netherlands \r\n \r\n Creative Pro...    Senior Program Manager - Creative Production I...   c09326a8-eabe-49bc-844c-28fa3a3d6361    Public  2023-07-10T13:00:10+00:00   2023-07-10T09:38:57Z    [Creative Production]   senior-program-manager-creative-production-inn...   Amsterdam, Netherlands  284698346   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64abd1f9b5811b900a89a18a    NaN     NaN     NaN
1   Manila, Philippines \r\n \r\n Netflix Technol...    Netflix IT Support Specialist   835f66aa-e7fc-46dc-a03d-5b39abf74afb    Public  2023-07-10T13:00:10+00:00   2023-07-10T08:26:19Z    [Netflix Technology Services]   netflix-it-support-specialist   Manila, Philippines     284690884   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64abc190aabfb410a82540a4    [Netflix Technology Services]   NaN     NaN
2   Tokyo, Japan \r\n Corporate Functions \r\n Tal...   HR Associate - Japan    6bdf2520-bbb7-4e40-9f0e-c562fc6391e1    Public  2023-07-10T13:00:10+00:00   2023-07-10T05:43:57Z    [Talent and Recruiting]     hr-associate-japan  Tokyo, Japan    284673328   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64ab9ae37a9df82a8330ae9d    [Human Resources]   [Corporate Functions]   NaN
3   Remote, California \r\n Corporate Functions \r...   People Scientist    1ba707a0-8bb2-436d-921a-1a985dd7fd5b    Public  2023-07-10T13:00:10+00:00   2023-07-07T17:59:45Z    [Talent and Recruiting]     people-scientist    Remote, California  284310527   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64a87cdaaabfb4e0b925418d    [Human Resources]   [Corporate Functions]   NaN
4   Remote, United States \r\n \r\n Core Engineer...    Software Engineer (L5) - Ads Revenue Infrastru...   0e733d7e-3a0a-4dd5-ab5e-51c970b484e3    Public  2023-07-10T13:00:10+00:00   2023-07-07T15:50:51Z    [Core Engineering]  software-engineer-l5-ads-revenue-infrastructure     Remote, United States   284278602   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64a8368db5811b243289a8dc    [Content Engineering Core]  NaN     NaN
5   Remote, United States \r\n Games \r\n Boss Fig...   Director of Game Development    46b233f1-8fff-4052-ac70-94f1e5ff6372    Public  2023-07-10T13:00:10+00:00   2023-07-03T23:29:19Z    [Boss Fight Entertainment]  director-of-game-development    Remote, United States   283904007   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64a950f83d6b6272441b736d    NaN     [Games]     NaN
6   Remote, United States \r\n Corporate Functions...   Executive Coach     c1a8ea67-fb0a-498d-8296-5929f1a2555a    Public  2023-07-10T13:00:10+00:00   2023-06-30T21:04:51Z    [Talent and Recruiting]     executive-coach     Remote, United States   283427947   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649f44af3d6b62a03c7c6f7c    [Human Resources]   [Corporate Functions]   NaN
7   Los Gatos, California \r\n Corporate Functions...   Business Process Analyst, Finance   f1078eb0-bf61-4f03-82f2-2b7c1c786830    Public  2023-07-10T13:00:10+00:00   2023-06-30T14:32:35Z    [Finance]   business-process-analyst-finance    Los Gatos, California   283385129   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649ee81ab5811b9d27624638    [Finance]   [Corporate Functions]   [Los Angeles, California]
8   Seoul, South Korea \r\n Corporate Functions \r...   Senior Manager - Country Lead, Production Fina...   8e35e3d6-f762-4b53-99e8-6cc62f1875db    Public  2023-07-10T13:00:14+00:00   2023-06-30T05:25:07Z    [Finance]   senior-manager-country-lead-production-finance...   Seoul, South Korea  283325929   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649e6863aabfb4704f86b47f    [Finance]   [Corporate Functions]   NaN
9   Singapore, Singapore \r\n Marketing and Public...   Manager, Ads B2B Marketing - APAC   f3275e7e-38af-4bb6-84b6-a99094738986    Public  2023-07-10T13:00:10+00:00   2023-06-29T05:21:12Z    [Partner Marketing]     manager-ads-b2b-marketing-apac  Singapore, Singapore    283167491   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649d15b5aabfb4551aed6809    NaN     [Marketing and Publicity]   NaN
10  Mumbai, India \r\n Content \r\n \r\n \r\n Ma...     Manager, Series - India     9a630609-3041-4698-a95d-543efe80a93c    Public  2023-07-10T13:00:10+00:00   2023-06-29T02:52:34Z    NaN     manager-india-series-and-films  Mumbai, India   283151020   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64a502957a9df822a932aa71    NaN     [Content]   NaN
11  Remote, United States \r\n \r\n Data Science ...    Data Science and Analytics Manager - Streaming...   ecf66443-f3cf-4ded-ab7b-c3ed84a2b348    Public  2023-07-10T13:00:17+00:00   2023-06-29T00:32:39Z    [Data Science and Engineering]  data-science-and-analytics-manager-streaming-p...   Remote, United States   283135676   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649cd1beb5811bc57c7eeed6    [Data Science and Engineering]  NaN     NaN
12  Remote, United States \r\n \r\n Creative Prod...    Regional Planning Lead - UCAN   4e8eb40e-c4fb-4775-b9f1-620c5fcc3f87    Public  2023-07-10T13:00:10+00:00   2023-06-28T22:42:58Z    [Creative Production]   regional-planning-lead-ucan     Remote, United States   283123645   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649cb7f23d6b62cec27a7039    [Product Creative Core]     NaN     NaN
13  Amsterdam, Netherlands \r\n \r\n \r\n \r\n ...  Manager, Finance Strategy & Analysis (FS&A) , ...   4c920cf4-1bd5-4630-9833-c16c7c3cac12    Public  2023-07-10T13:00:17+00:00   2023-06-28T08:44:17Z    NaN     manager-revenue-analytics   Amsterdam, Netherlands  283031908   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    64a587cf7a9df89cbdc9f25d    NaN     NaN     NaN
14  Los Angeles, California \r\n Marketing and Pub...   AV Producer - UCAN  2c76fb58-8c56-442e-98be-028889094b0c    Public  2023-07-10T13:00:10+00:00   2023-06-27T23:13:25Z    [Creative Marketing Production]     av-producer-ucan    Los Angeles, California     282969550   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649b6d7aaabfb4675b853b6d    [Marketing Production]  [Marketing and Publicity]   NaN
15  Manila, Philippines \r\n Corporate Functions \...   AP Invoice Processing Specialist, Manila (Japa...   bd698f4b-aac7-46d7-9087-053f077e05d7    Public  2023-07-10T13:00:10+00:00   2023-06-27T02:45:47Z    [Finance]   ap-invoice-processing-specialist-manila-japane...   Manila, Philippines     282835505   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649a4e5d3d6b62657a2b93c7    [Finance]   [Corporate Functions]   NaN
16  Manila, Philippines \r\n Corporate Functions \...   Accounts Payable Invoice Processing Specialist...   e0048b97-2c8b-4ae3-8fe8-a88e8c17b210    Public  2023-07-10T13:00:10+00:00   2023-06-27T01:41:20Z    [Finance]   ap-invoice-processing-specialist-manila     Manila, Philippines     282828368   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649a3f22b5811b71eb80f9c4    [Finance]   [Corporate Functions]   NaN
17  Dallas, Texas \r\n Games \r\n Boss Fight Enter...   IT Support Specialist - Boss Fight Entertainment    2d3c4f18-4f81-46ea-a8e8-b402865c47ab    Public  2023-07-10T13:00:10+00:00   2023-06-26T23:39:17Z    [Boss Fight Entertainment]  it-support-specialist-boss-fight    Dallas, Texas   282814656   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649b052eaabfb4f2708532c6    NaN     [Games]     NaN
18  Los Gatos, California \r\n Games \r\n Netflix ...   Player Support Lead, Netflix Games Studio   3a3aa925-861a-4567-a367-8c9ce9edf2bc    Public  2023-07-10T13:00:10+00:00   2023-06-26T16:26:37Z    [Netflix Games Studio]  player-support-lead-netflix-games-studio    Los Gatos, California   282767770   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    6499bd167a9df88c1302b79a    NaN     [Games]     NaN
19  Warsaw, Poland \r\n Corporate Functions \r\n T...   HR Business Partner, CEE    4abd34d0-c9a3-4469-8533-f757c2b65063    Public  2023-07-10T13:00:17+00:00   2023-06-23T16:13:14Z    [Talent and Recruiting]     hr-business-partner-cee     Warsaw, Poland  282717815   ...     5e7cf522815a780d9124028c    _doc    None    None    None    None    649951413d6b626dab2b8129    [Human Resources]   [Corporate Functions]   NaN

20 rows × 22 columns

Answered By - Barry the Platipus