[FIXED] Get, scrape data from page 2 and page 3 by clicking on HTML element selenium python

Issue

I am scraping data(cosmetic name, brand of cosmetic, price of cosmetic, link to cosmetic) from this link:

https://sokoglam.com/collections/face-moisturizer

This link has data in 3 pages. I am able to scrap data from page 1, but I am not able to get data from page 2 and 3.

I have been researching, I here you can click on an element to move to the next page, and get elements of the page to be scraped. If you can write for me the code or you have a resource. It will be highly appreciated

I've tried clicking on the element in button but I do not get the data.

try:
    driver.execute_script('return arguments[0].scrollIntoView(true);',
                          WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//button[@class="mixitup-control mixitup-control-next"]/span'))))
    driver.find_element(By.XPATH, '//button[@class="mixitup-control mixitup-control-next"]/span').click()
    print('Navigating Next Page')
except:
    print('Done done')

Solution

The data you see on the page is stored inside <script> element. To parse it you can use next example:

import re
import requests
import pandas as pd
from ast import literal_eval


url = 'https://sokoglam.com/collections/face-moisturizer'
html_text = requests.get(url).text
data = re.search(r'var product = (\{.*?\});', html_text, flags=re.S)
data = literal_eval(data.group(1))

df = pd.DataFrame(data['products'])
print(df.head())

Prints (first 5 of 108 elements total):

              id sku variantId         productType                             name  price                                                                                                                       imageURL                                                                   productURL      brand comparePrice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  categories     currentCategory              productOptions
0  6879416483909                Facial Moisturizer        Hydrating Facial Emulsion  24.00              https://cdn.shopify.com/s/files/1/0249/1218/products/Sonreve-Hydrating-facial-emulsion-01_grande.png?v=1681823606             https://sokoglam.com/products/son-reve-hydrating-facial-emulsion  Soko Glam         0.00  [Anti-Aging Skin Care, Buy Two, Get One 20% off, Clean Skin Care, Cyber Month Deals!, Facial Moisturizer, Fragrance Free Skin Care, Live SKUs (except for OOS), Master Collection (Employee Discount), Master Collection (excluding TIMY), Master Collection for BF 2020 (11\/27\/2020), Memorial Day Sale 2023, Moisturizer, Moisturizers, Mother's Day Collection, Newly Curated, Newly Curated Slider, Pregnancy & Breastfeeding Safe, Self-Care, Shop All, SON REVE, Spring Curation, Vegan Skin Care]  Facial Moisturizer  {'Title': 'Default Title'}
1  6875368521797                Facial Moisturizer          Water-Bag Barrier Cream  32.00  https://cdn.shopify.com/s/files/1/0249/1218/products/Soko-Glam-PDP-Plodica-Water-Bag-Barrier-Cream-02_grande.png?v=1678387753                https://sokoglam.com/products/plodica-water-bag-barrier-cream  Soko Glam         0.00                                   [Artificial Fragrance Free Shop, Buy Two, Get One 20% off, Clean Skin Care, Cyber Month Deals!, Exclusives, Facial Moisturizer, Fragrance Free Skin Care, Live SKUs (except for OOS), Master Collection (Employee Discount), Master Collection (excluding TIMY), Master Collection for BF 2020 (11\/27\/2020), Memorial Day Sale 2023, Moisturizer, Moisturizers, Newly Curated, Newly Curated Slider, Plodica, Self-Care, Shop All, Sustainable Beauty, Vegan Skin Care]  Facial Moisturizer  {'Title': 'Default Title'}
2  6886078906437                Facial Moisturizer                    Ceramic Cream  33.00            https://cdn.shopify.com/s/files/1/0249/1218/products/Soko-Glam-PDP-Tir-Tir-Ceramic-Cream-01_grande.png?v=1680128889                           https://sokoglam.com/products/tirtir-ceramic-cream  Soko Glam         0.00                                                                                     [Anti-Aging Skin Care, Buy Two, Get One 20% off, Clean Skin Care, Cyber Month Deals!, Dryness, Facial Moisturizer, Fragrance Free Skin Care, Live SKUs (except for OOS), Master Collection (Employee Discount), Master Collection (excluding TIMY), Master Collection for BF 2020 (11\/27\/2020), Moisturizer, Moisturizers, Mother's Day Collection, Newly Curated, Newly Curated Slider, Self-Care, Shop All, TIRTIR]  Facial Moisturizer  {'Title': 'Default Title'}
3  6875364720709                Facial Moisturizer  Bifida Biome Aqua Barrier Cream  25.00  https://cdn.shopify.com/s/files/1/0249/1218/products/Soko-Glam-PDP-Bifida-Biome-Aqua-Barrier-Cream-01_grande.png?v=1679677528  https://sokoglam.com/products/manyo-factory-bifida-biome-aqua-barrier-cream  Soko Glam         0.00                                                         [Anti-Aging Skin Care, Buy Two, Get One 20% off, Clean Skin Care, Cyber Month Deals!, Facial Moisturizer, Live SKUs (except for OOS), MANYO FACTORY, Master Collection (Employee Discount), Master Collection (excluding TIMY), Master Collection for BF 2020 (11\/27\/2020), Memorial Day Sale 2023, Moisturizer, Moisturizers, Newly Curated, Newly Curated Slider, Self-Care, Shop All, Skin Barrier, Skincare Collection Test, Vegan Skin Care]  Facial Moisturizer  {'Title': 'Default Title'}
4  6875364360261                Facial Moisturizer       V Collagen Heart Fit Cream  38.00       https://cdn.shopify.com/s/files/1/0249/1218/products/Soko-Glam-PDP-V-Collagen-Heart-Fit-Cream-01_grande.png?v=1679677524       https://sokoglam.com/products/manyo-factory-v-collagen-heart-fit-cream  Soko Glam         0.00                                                                                   [Anti-Aging Skin Care, Buy Two, Get One 20% off, Clean Skin Care, Cyber Month Deals!, Facial Moisturizer, Live SKUs (except for OOS), MANYO FACTORY, Master Collection (Employee Discount), Master Collection (excluding TIMY), Master Collection for BF 2020 (11\/27\/2020), Memorial Day Sale 2023, Moisturizer, Moisturizers, Newly Curated, Newly Curated Slider, Self-Care, Shop All, Skin Barrier, Vegan Skin Care]  Facial Moisturizer  {'Title': 'Default Title'}

Answered By - Andrej Kesely