[FIXED] To define fields in items class in scrapy using a list

Issue

#i want to create Field for my item class in scrapy im reading the list from a csv and then #converting it to a list but when i want to define fields in items class using init it gives me this error #raise AttributeError(name) #AttributeError: _values. Did you mean: 'values'?

#Anyways here is the code kindly help me fix this

fields = pd.read_csv('E:/pythonProject/webscrapping/postscrape/fields.csv')
fields = list(fields['0'])
fields.insert(0, 'Company')


class PostscrapeItem(scrapy.Item):
    # define the fields for your item here like:
    def __init__(self):
        for f in fields:
            self.__dict__[f] = scrapy.Field()


i = PostscrapeItem()
print(i.Keys())

Solution

the scrapy.Item class uses a non-standard meta class which makes overwriting __init__ both impossible and unnecessary.

If your goal is to dynamically add fields to the item class it should work by setting the values of the Item.fields dictionary.

This example should show all that is needed:

fields = pd.read_csv('E:/pythonProject/webscrapping/postscrape/fields.csv')
fields = list(fields['0'])
fields.insert(0, 'Company')

class PostscrapeItem(scrapy.Item):
    
    @classmethod
    def load_fields(cls, fields):
        for f in fields:
            cls.fields[f] = scrapy.Item()

PostscrapeItem.load_fields(fields)

There is no need to create an instance.

Answered By - alexpdev