[FIXED] Webscraping a Javascript Element, preferably without Selenium

Issue

Currently, I have used Selenium to extract text from a table on a website. Following is the code:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager


# Using Chrome to access web
browser = webdriver.Chrome(ChromeDriverManager().install())

# Open the website
browser.get('https://launchstudio.bluetooth.com/Listings/Search')
element = browser.find_element_by_id('searchButton').click()

table_text = browser.find_element_by_class_name('table').text

while len(table_text) < 80:
    table_text = browser.find_element_by_class_name('table').text

print(table_text)

browser.close()

However, I am trying to find a way to do the same with Requests/Beautiful soup or any other library where I can schedule this as a task in windows and store the result in a table at every x interval. Obviously, since I want all this to happen in the background and then trigger a notification etc.

What I Want is- Open this website, click on the search button (or trigger the corresponding javascript), and then export the table as a Dataframe or whatever.

Can you please guide me here?

thanks in advance!!

Solution

If you go to Network Tab you will get the API . You can use this post request to get all the value.Using max result field you can limit the results as well.

https://platformapi.bluetooth.com/api/platform/Listings/Search

import requests
import  pandas as pd
data={
"searchString" : "",
"searchQualificationsAndDesigns": True,
"searchDeclarationOnly": True,
"bqaApprovalStatusId" : -1,
"bqaLockStatusId" : -1,
"layers" : [],
"listingDateEarliest" : "",
"listingDateLatest" : "",
"maxResults": 5000,
"memberId": "",
"productTypeId" : 0,
"searchDeclarationOnly" : True,
"searchEndProductList" : False,
"searchMyCompany" : False,
"searchPRDProductList" : True,
"searchQualificationsAndDesigns" : True,
"searchString" : "",
"specName": 0,
"userId" : 0
}

headers = {'User-Agent':
       'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

url="https://platformapi.bluetooth.com/api/platform/Listings/Search"
response=requests.post(url,headers=headers,data=data).json()
df=pd.DataFrame(response)
print(df)

---

You can import to csv file.

df.to_csv("testresult.csv")

Answered By - KunduK