[FIXED] Giving Syntax error in Scrapy (Python) - XPath

Issue

I'm using Scrapy Crawler to extract some details like username, upvotes, join date etc.

I'm using XPath for extracting the contents from each user's webpage.

Code:

import scrapy
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
from scrapy.spiders import BaseSpider
from scrapy.http import FormRequest
from loginform import fill_login_form
from scrapy.selector import Selector
from scrapy.http import HtmlResponse

class UserSpider(scrapy.Spider):
    name = 'userspider'
    start_urls = ['<website-to-scrape>']
    #Getting the list of usernames
    user_names = ['Bob', 'Tom']  #List of Usernames

    def __init__(self, *args, **kwargs):
        super(UserSpider, self).__init__(*args, **kwargs)

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'registerUserName': 'user', 'registerPass': 'password'},
                    callback=self.after_main_login)]

    def after_main_login(self, response):
        for user in self.user_names:
            user_url = 'profile/' + user
            yield response.follow(user_url, callback=self.parse_user_pages)

    def parse_user_pages(self, response):
        yield{
            "USERNAME": response.xpath('//div[contains(@class, "main") and contains(@class, "no-sky-main")]/h1[contains(@class, "thread-title")]/text()').extract_first()
            "UPVOTES": response.xpath('//div[contains(@class, "proUserInfoLabelLeft") and @id="proVotesCap"]/text()').extract()[0]
        }

if __name__ == "__main__":
    spider = UserSpider()

Error looks like this

P.S. I have manually checked the syntax of my XPath on the Scrapy Shell and it was working fine

Is there anything that I'm not noticing in the code?

Solution

You're missing a , after your first dict element:

{"USERNAME": response.xpath(...).extract_first(),
 "UPVOTES": response.xpath(...).extract()[0]}

Answered By - Michael Kohl