[FIXED] Why is Python requests returning a different text value to what I get when I navigate to the webpage by hand?

Issue

I am trying to build a simple 'stock-checker' for a T-shirt I want to buy. Here is the link: https://yesfriends.co/products/mens-t-shirt-black?variant=40840532689069

As you can see, I am present with 'Coming Soon' text, whereas usually if an item is in stock, it will show 'Add To Cart'.

I thought the simplest way would be to use requests and beautifulsoup to isolate this <button> tag, and read the value of text. If it eventually says 'Add To Cart', then I will write the code to email/message myself it's back in stock.

However, here's the code I have so far, and you'll see that the response says the text contains 'Add To Cart', which is not what the website actually shows?

import requests
import bs4

URL = 'https://yesfriends.co/products/mens-t-shirt-black?variant=40840532689069'


def check_stock(url):

    page = requests.get(url)
    soup = bs4.BeautifulSoup(page.content, "html.parser")
    buttons = soup.find_all('button', {'name': 'add'})

    return buttons


if __name__ == '__main__':

    buttons = check_stock(URL)
    print(buttons[0].text)

Solution

All data available in <script> tag in JSON. So we need to get this, and extract the information we need. Let's use a simple slice by indexes to get clean JSON

import requests
import json

url = 'https://yesfriends.co/products/mens-t-shirt-black'
response = requests.get(url)
index_start = response.text.index('product:', 0) + len('product:')
index_finish = response.text.index(', }', index_start)
json_obj = json.loads(response.text[index_start:index_finish])
for variant in json_obj['variants']:
    available = 'IN STOCK' if variant['available'] else 'OUT OF STOCK'
    print(variant['id'], variant['option1'], available)

OUTPUT:

40840532623533 XXS OUT OF STOCK
40840532656301 XS OUT OF STOCK
40840532689069 S OUT OF STOCK
40840532721837 M OUT OF STOCK
40840532754605 L OUT OF STOCK
40840532787373 XL OUT OF STOCK
40840532820141 XXL OUT OF STOCK
40840532852909 3XL IN STOCK
40840532885677 4XL OUT OF STOCK

Answered By - Sergey K