[FIXED] Why doesn't this code work? *Complete Noob*

Issue

I recently started learning Python, and I wanted to create a program that will show me all the new releases from AllMusic, but it doesn't work. I'm sorry, but I'm a complete noob. At first I want to just see the artist:

import requests
from bs4 import BeautifulSoup


def new_releases():
    url = "http://allmusic.com/newreleases"
    source_code = requests.get(url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    for div in soup.findAll('div', {'class': 'artist'}):
        for a in div.findAll('a'):
            artist = a.string
            print(artist)

new_releases()

What am I doing wrong? I don't get any errors, it jsut doesn't work for whatever reason

Solution

Actually, your code is fine. But the site you are trying to grab prevents you from doing that. You'd find that out if you printed your soup this way: print(soup).

To avoid that you may specify a User Agent (see wiki) header in your requests.get:

source_code = requests.get(url, headers=headers)

where headers is a dictionary like this:

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0'
}

Now it'll be working.

Answered By - vrs