[FIXED] How to complete an ADVANCED Web Scrape of NBA player props?

Issue

I would like to scrape the NBA player prop bets from https://www.bovada.lv. I named the question ADVANCED because there are multiple teams, players, and categories.

Here is the HTML code as requested:

https://pastebin.com/UkY071uV

Here is the link that will bring you directly to the basketball section: https://www.bovada.lv/sports/basketball.

To get started...

The NBA player props are located in the basketball section. If you click the arrow next to each game, or ">", it will take you to another betting page. If the player props have been released you can find them located near the bottom of the page in the Player Prop section (note: the props are released before the games start).

Here is a sample of the data of one player I am looking for:

Total Points - LeBron James (LAL)
28.5     -115     -115

I would like to scrape all of the players names, the category and the bet odds. Unfortunately I didn't make it too far. The methods I have learned so far have had no success.

#import modules

from bs4 import BeautifulSoup
import requests, os
from selenium import webdriver

#initiate Selenium

os.chdir('C:\webdrivers')

#enter user agent

header = {'User-agent' : 'ENTER USER_AGENT HERE'}
options = webdriver.ChromeOptions(); options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.bovada.lv/sports/basketball/nba')
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()

#attempt at printing soup 
print(soup)

I can't locate any of the respective players in the code. I figured it couldn't hurt to reach out for some help. Perhaps someone with more experience knows how to do this or can help lead me in the right direction.

I am new to web scraping and very appreciative of any assistance that you may offer. Thanks in advance for your time!

Solution

This sites uses an internal JSON api to get the data. The full JSON data for your example can be found here : https://www.bovada.lv/services/sports/event/v2/events/A/description/basketball/nba/los-angeles-lakers-sacramento-kings-201811102200?lang=en

An example to extract your data with curl & jq :

curl -s "https://www.bovada.lv/services/sports/event/v2/events/A/description/basketball/nba/los-angeles-lakers-sacramento-kings-201811102200?lang=en" | \
     jq '.[0].events[0].displayGroups[] | 
         select(.description=="Player Props") | 
         .markets[] | 
         select(.description=="Total Points - LeBron James (LAL)")' 

With python :

import requests

r = requests.get('https://www.bovada.lv/services/sports/event/v2/events/A/description/basketball/nba/los-angeles-lakers-sacramento-kings-201811102200?lang=en')

player_props = [
    t["markets"]
    for t in r.json()[0]["events"][0]["displayGroups"]
    if t["description"] == "Player Props"
]
specific_player = [
    t
    for t in player_props[0]
    if t["description"] == "Total Points - LeBron James (LAL)"
]
print(specific_player)

Answered By - Bertrand Martel