[FIXED] requests.exceptions.InvalidURL: Failed to parse: <Response [200]> in python

Issue

So i wrote this code for now, to get news from a specific topic from cnn right now im getting an error here is the code:

from bs4 import BeautifulSoup
import requests
import csv
import re

serch_term = input('What News are you looking for today? ')

url = f'https://edition.cnn.com/search?q={serch_term}'
page = requests.get(url)
doc = BeautifulSoup(page, "html.parser")

page_text = doc.find_all(class_="cnn-search__result-headline")
print(page_text)

But i am getting this error, i already tried a bunch of things but none of them worked for me

What News are you looking for today? coronavirus
Traceback (most recent call last):
  File "c:\Users\user\Desktop\Informatik\Praktik\Projekte\Python\news_automation\main.py", line 10, in <module>
    doc = BeautifulSoup(page, "html.parser")
  File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\bs4\__init__.py", line 312, in __init__
    elif len(markup) <= 256 and (
TypeError: object of type 'Response' has no len()

i googled already and tried a bunch of things but none of them worked Does someone know what is wrong? So,

Solution

I tested it myself and you should change this line of code as follows:

from: source = requests.get(url) to: page = source.text

Extra informations:

I found that u can use this search.api.cnn.io as follows and make directly into json as i wrote the code and what you need to do is extract information which you need.

url = f"search.api.cnn.io/content?q={serch_term}"

extra_parameters_sample_url"https://search.api.cnn.io/content?q=coronavirus&sort=newest&category=business,us,politics,world,opinion,health&size=100&from=0"

source = requests.get(url).text 
json_reponse = json.loads(source)

Answered By - Lentian Latifi