[FIXED] How to apply a user-defined function to a column in pandas dataframe?

Issue

I'm working on an NLP task. I defined a function to conduct aspect extraction and sentiment based on the aspect and dependency parsing. The function looks like:

import spacy
nlp = spacy.load('en_core_web_sm')

def aspect_sentiment(text):
    aspects = []
    for sentence in text:
        doc = nlp(sentence)
        descriptive_term = ''
        target = ''
        for token in doc:
            if token.dep_ == 'nsubj' and token.pos_ == 'NOUN':
                target = token.text
            if token.pos_ == 'ADJ':
                prepend = ''
                for child in token.children:
                    if child.pos_ != 'ADV':
                        continue
                    prepend += child.text + ' '
                descriptive_term = prepend + token.text
        aspects.append({'aspect': target, 'description': descriptive_term})
    
    for aspect in aspects:
        aspect['polarity'] = TextBlob(aspect['description']).sentiment.polarity
        aspect['subjectivity'] = TextBlob(aspect['description']).sentiment.subjectivity
    
    return(aspects)

First I made a list of sentences for testing:

sentences = [
  'The food we had yesterday was delicious',
  'My time in Italy was very enjoyable',
  'I found the meal to be tasty',
  'The internet was slow.',
  'Our experience was suboptimal'
]

And when I use aspect_sentiment(sentences) it returns the desired outcome:

[{'aspect': 'food',
  'description': 'delicious',
  'polarity': 1.0,
  'subjectivity': 1.0},
 {'aspect': 'time',
  'description': 'very enjoyable',
  'polarity': 0.65,
  'subjectivity': 0.78},
 {'aspect': 'meal',
  'description': 'tasty',
  'polarity': 0.0,
  'subjectivity': 0.0},
 {'aspect': 'internet',
  'description': 'slow',
  'polarity': -0.30000000000000004,
  'subjectivity': 0.39999999999999997},
 {'aspect': 'experience',
  'description': 'suboptimal',
  'polarity': 0.0,
  'subjectivity': 0.0}]

But when I make the sentence list into a pandas dataframe and apply the function, the result column doesn't return the correct values:

text = pd.DataFrame(sentences, columns=['sentences'])
text['result'] = text['sentences'].map(aspect_sentiment)

it returns:

How can I fix it? The output seems very wrong...

Solution

You can use df[col].apply(fn) instead, which will run the function once on each element in a pandas Series. Just need to adjust aspect_sentiment a bit to expect individual sentences instead of a list of sentences.

Answered By - tnwei