[FIXED] How to scrape data that appear when click on button?

Issue

I am trying to scrape phone numbers from website, but the numbers will appear only if I click on the first number. In other words, the phone will be hidden in the HTML code, and when I click it will appear. can you help please? I used the following code:

import requests
from bs4 import BeautifulSoup

url = "https://hipages.com.au/connect/makermanservices"

req = requests.get(url).text

soup = BeautifulSoup(req,"html.parser")

phone = soup.find('a', class_='PhoneNumber__MobileOnly-sc-4ewwun-1 izNnbI phone-number__mobile')
print(phone)

Solution

With a little bit of hacking, you can get the phone number with the help of bs4 and pandas.

For example:

import json
import re

import pandas as pd
import requests
from bs4 import BeautifulSoup

url = "https://hipages.com.au/connect/makermanservices"
script_text = "window.__INITIAL_PROPS__="

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.48",
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, "lxml")
script = soup.find("script", string=lambda t: t and script_text in t)
data = json.loads(re.search(script_text + r"(.+)", script.string).group(1))
df = (
    pd.read_json(data)
    ["fetchKey-7-0-0_/connect/makermanservices"]
    ["site"]
    ["primary_location"]
    ["phone"]
)
print(df)

This should print:

1800 801 828

Answered By - baduker