[FIXED] Anyone know why it scrapes only 1 page

Issue

I'm trying to scrape multiple pages, but my current code only manages to scrape 1 page. How can I fix this to scrape multiple pages?

import requests 
from bs4 import BeautifulSoup
import pandas as pd

pages = list(range(1, 20))
for page in pages:
    req = requests.get("https://katmoviehd.sk/page/{}".format(page))
    
    soup=BeautifulSoup(req.text,"html.parser")
    soup
    page1 = soup.find_all('h2')[1:]
        
    Category = soup.find_all('span', class_ = 'meta-category')
    Category
        
    Category_list = []
        
    for i in Category:
        Category2 = i.text
        Category_list.append(Category2)
        
    link_list = []
        
    for i in page1:
        link = (i.find("a")['href'])
        link_list.append(link)       
    title_list = []    
    for i in page1:
        title = (i.find("a")['title'])
        title_list.append(title)
            
            
    Table = pd.DataFrame({'Links':link_list, 'Title':title_list, 'Category':Category_list})

Solution

Here is the working output with pagination:

    import requests 
    from bs4 import BeautifulSoup
    import pandas as pd
    page = 1
    title_list = []
    link_list = []
    Category_list = []
    for page in range(0, 10):
        url = f"https://katmoviehd.sk/page/{page}".format(page=page)
        response = requests.get(url)
        html = response.content
        soup = BeautifulSoup(html, "lxml")
        page1 = soup.find_all('h2')[1:]
            
        Category = soup.find_all('span', class_ = 'meta-category')
        Category
                          
        for i in Category:
            Category2 = i.text
            Category_list.append(Category2)
                  
        for i in page1:
            link = (i.find("a")['href'])
            link_list.append(link)       
              
        for i in page1:
            title = (i.find("a")['title'])
            title_list.append(title)
                    
                    
   Table = pd.DataFrame({'Links':link_list, 'Title':title_list, 'Category':Category_list})
   print(Table)

OUTPUT: Each page contains 16 items and I'm going to scrape only 10 pages and total output 160. You can increase or decrease range of page numbers whatever you wish.

Links  ...
           Category
0           https://katmoviehd.sk/stillwater-s01-2020/  ...  Animated / APPLE TV+ / Dual Audio / Family / T...
1            https://katmoviehd.sk/what-if-2021-hindi/  ...  Action / Adventure / Animated 
/ Disney + / Dua...
2           https://katmoviehd.sk/dcs-titans-season-3/  ...  Action / Adventure / Fantasy / In English (Onl...
3              https://katmoviehd.sk/the-passing-2011/  ...                 Dual Audio / Dubbed Movie / Horror
4    https://katmoviehd.sk/annette-2021-bengali-dub...  ...                        Bengali 
Dubbed (Unofficial)
..                                                 ...  ...
                ...
155          https://katmoviehd.sk/rick-and-morty-s05/  ...  Adventure / Animated / Comedy 
/ sci-fi / TV shows
156                   https://katmoviehd.sk/backtrace/  ...  Action / crime / Drama / Dual 
Audio / Dubbed M...
157       https://katmoviehd.sk/rattle-can-full-movie/  ...                Hindi Subtitles 
/ In English (Only)
158               https://katmoviehd.sk/streetdance-2/  ...  Drama / Dual Audio / Dubbed Movie / Music / ro...
159  https://katmoviehd.sk/midnight-in-the-switchgr...  ...
  Unofficial Dubbed

[160 rows x 3 columns]

Answered By - Fazlul