[FIXED] Update website after changing attribute value using bs4

Issue

I am currently using bs4 to scrape the following website. I want to change the view from 'Top Reactions' to 'Latest Reactions' but the only way to do this is by adding "selected" as one of the classes of the tag around 'Latest Reactions'. After doing this though, the html code doesn't change and 'Top Reactions' is still displayed. Is there a way to fix this using bs4?

url = "https://finance.yahoo.com/quote/{0}/community?p={0}".format(ticker)
page = urlopen(url)
html = page.read().decode("utf-8")
soup = BeautifulSoup(html, "html.parser")

latest_button = soup.find('li', {'class' : 'H(44px)'})
latest_button['class'] = latest_button['class'] + 'selected'

comments = soup.findAll("li", {"class": "comment"})


for comment in comments: # Goes through the latest n messages
    post = {
        
    }
    
    post["content"] = comment.find("div", {"class": "C($c-fuji-grey-l)"}).text

    print(post)

Solution

No need for Selenium. You can get the data by api. Simply changing a parameter in the url from sort by popular to createdAt will do that for you.

import requests
import time
import re
import statistics


ticker = 'TSLA'

# Get prid
url = 'https://uk.finance.yahoo.com/quote/{ticker}/community?p={ticker}'.format(ticker=ticker)
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36'}
response = requests.get(url, headers=headers).text
prid_search = re.search('{\"prid\":\"([a-zA-Z0-9]*)', response)
prid = prid_search.group(1)

finmb_search = re.findall('finmb_([\d]*)', response)
finmb = statistics.mode(finmb_search)

api_url = 'https://uk.finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;apiVersion=v1;context=finmb_{finmb};count=20;index=null;lang=en-GB;namespace=yahoo_finance;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace%20%3D%20%22yahoo_finance%22%20and%20(contextId%3D%22finmb_{finmb}%22%20or%20tag%3D%22{ticker}%22);rankingProfile=canvassPCRDProfile;region=GB;sortBy=createdAt;spaceId=1186050005;type=null;userActivity=true?bkt=finance-GB-en-GB-def&device=desktop&ecma=modern&feature=canvassOffnet%2CccOnMute%2CdisableCommentsMessage%2Cdebouncesearch100%2CdeferDarla%2CecmaModern%2CemptyServiceWorker%2Cenable3pConsent%2CenableCCPAFooter%2CenableCMP%2CenableConsentData%2CenableGuceJs%2CenableGuceJsOverlay%2CenableNavFeatureCue%2CenablePrivacyUpdate%2CenableStreamDebounce%2CenableTheming%2CenableUpgradeLeafPage%2CenableVideoDocking%2CenableVideoURL%2CenableXrayNcp%2CenableXrayNcpInModal%2CenableXrayTickerEntities%2CenableYahooSans%2CenableYodleeErrorMsgCriOS%2CncpListStream%2CncpPortfolioStream%2CncpQspStream%2CncpStream%2CncpStreamIntl%2CncpTopicStream%2CnewContentAttribution%2CnewLogo%2CoathPlayer%2CrelatedVideoFeature%2CuseNextGenHistory%2CvideoNativePlaylist%2CsunsetMotif2%2CenableUserPrefAPI%2CenableSingleRail%2CenhanceAddToWL%2Carticle2_csn%2CenableStageAds%2CsponsoredAds&intl=uk&lang=en-GB&partner=none&prid={prid}&region=GB&site=finance&tz=Europe%2FLondon&ver=0.102.4664&returnMeta=true'.format(ticker=ticker,finmb=finmb,prid=prid)
jsonData = requests.get(api_url, headers=headers).json()

messages = jsonData['data']['canvassMessages']
for message in messages:
    author = message['meta']['author']['nickname']
    createdAt = message['meta']['createdAt']
    createdAt = time.strftime('%Y-%m-%d %I:%M:%S %p', time.localtime(createdAt))
    text = message['details']['userText']
    print('%s posted at: %s \n%s\n\n' %(author, createdAt, text)) 
        

Output:

trippett posted at: 2021-04-19 02:51:05 PM 
I sat on the fence for 3 weeks about joining (http://Stocks-tracker.com) for their weekly picks. Big mistake. AAPL exploded and I completely missed out. Don’t make the same mistake I made. Join them today.


Maqrsqrt posted at: 2021-04-19 02:51:05 PM 
Gonna makes me lotsa money


festus posted at: 2021-04-19 02:50:27 PM 
165 into earnings


Maqrsqrt posted at: 2021-04-19 02:49:58 PM 
Flush those daytraders!!


matthew posted at: 2021-04-19 02:48:54 PM 
Bought 1 apple share! Woo


steve posted at: 2021-04-19 02:48:36 PM 
go baby, go 🍏🤩🤩🤩🤩🚀🚀🚀🚀🚀🚀 enjoy the ride to 150+ this week 🤩🚀🚀🚀🚀


Aleem posted at: 2021-04-19 02:48:03 PM 
If only this hits 145 this week 😻


felix posted at: 2021-04-19 02:47:55 PM 
I think by July it should be in the $200


Akshay posted at: 2021-04-19 02:47:41 PM 
What's the news?


Maqrsqrt posted at: 2021-04-19 02:46:32 PM 
look out below!!


michael posted at: 2021-04-19 02:45:17 PM 
Where’s Giuseppe ? Lol


Kafil posted at: 2021-04-19 02:43:38 PM 
Lets stay above $ 135 for today


Tina posted at: 2021-04-19 02:42:58 PM 
yesss! $135 crossed!


Kafil posted at: 2021-04-19 02:42:57 PM 
BINGO


Jack posted at: 2021-04-19 02:37:59 PM 
And the climb into earnings continues


Trader posted at: 2021-04-19 02:37:43 PM 
Wants to run to 141??


ray posted at: 2021-04-19 02:36:35 PM 
Turning green


Tina posted at: 2021-04-19 02:36:34 PM 
Way to go! This will be $138 tomorrow at 3.30 pm


Kafil posted at: 2021-04-19 02:36:16 PM 
Watch out if we brake $ 135 next stop will be $ 140


Richie Rich posted at: 2021-04-19 02:34:57 PM 
134.98 then down

Answered By - chitown88