[FIXED] How can I scrape from websites that have infinite scrolling?

Issue

I have managed to create a web scraper that can collect the item descriptions however the page loads more items as it scrolls.

from selenium import webdriver 
import time
import requests
from bs4 import BeautifulSoup
from numpy import mean

namelist=[]
driver=webdriver.Chrome()
driver.get("https://waxpeer.com/")
time.sleep(15)



links = driver.find_elements_by_xpath("//div[@class='lpd_div']/a")

I also need the item description to format as:

★ Karambit| Gamma Doppler (Factory new)

rather than:

★ Karambit

Gamma Doppler

Factory new

desc = driver.find_elements_by_xpath("//div[@class='lpd_div']/div[2]/p")
for item in desc:
    print(item.text)

Solution

There's no need to use Selenium. The data is available via sending a GET request to the websites API in the following format:

https://waxpeer.com/api/data/index/?skip={offset}&sort=best_deals&game=csgo&all=0

with the offset of + 50 for every page.

---

For example, to print the names:

import requests

URL = (
    "https://waxpeer.com/api/data/index/?skip={offset}&sort=best_deals&game=csgo&all=0"
)

offset = 0

while True:
    try:
        response = requests.get(URL.format(offset=offset)).json()
        for data in response["items"]:
            print(data["name"])
        print("-" * 80)
        offset += 50
    except KeyError:
        break

Output:

★ Karambit | Gamma Doppler (Factory New)
★ Karambit | Gamma Doppler (Factory New)
★ Karambit | Gamma Doppler (Factory New)
★ Butterfly Knife | Doppler (Factory New)
★ Butterfly Knife | Doppler (Factory New)
★ Karambit | Gamma Doppler (Factory New)
★ Karambit | Gamma Doppler (Factory New)
...
...

Answered By - MendelG