[FIXED] Pandas - column with nested None value, convert to empty string

Issue

I have the following example list that I convert to a df:

new_list = [{'video_id': {'platform': 'facebook', 'id':'123'}, 'title': 'Scam Rapper', 'description': 'truthful', 'keywords':['x', 'B'], 'publish_date': '2022-06-15', 'publish_timestamp':'2022-06-15 23:30:02', 'publisher': {'creator_id': '2pal', 'creator_name': 'X'}, 'thumbnail_url': 'https://sc4', 'video_url':'https:/', 'duration': 278, 'views': 836977, 'engagements': {'total':18463, 'breakdown': [{'platform': 'facebook', 'total': 18463, 'likes':9436, 'shares': 5581, 'comments': 608, 'tweets': None, 'favorites':None, 'hahas': 1049, 'wows': 170, 'loves': 1554, 'sads': 44, 'angrys':21}]}, 'categories': [{'category_name': 'News, Government & Politics'}, {'category_name': 'News, Government & Politics/News'},{'category_name': 'News, Government & Politics/News/Crime'}],'language': 'en', 'tvr': {'v1': 456443, 'v2': 673473, 'v3': 739055,'v7': 832210, 'v30': None, 'er1': 0.638300001621246, 'er2':0.689400017261505, 'er3': 0.7175999879837031, 'er7': 0.7592999935150141, 'er30': None}, 'video_was_live': False}]

video_items_df = pd.DataFrame(new_list)

video_items_df = pd.concat([video_items_df.drop(['engagements'], axis=1), video_items_df['engagements'].apply(pd.Series)], axis=1)

video_items_df = pd.concat(
            [video_items_df.drop(['video_id'], axis=1), video_items_df['video_id'].apply(pd.Series)], axis=1)

But you'll see that in the breakdown column now created, there is a None value, I tried to convert it to string by adding this but it didn't work I think because its a nested None

video_items_df.fillna("",inplace=True)

Solution

Use:

In [1664]: video_items_df['breakdown'] = video_items_df.breakdown.apply(lambda x: {k: v or '' for (k,v) in x[0].items()})

In [1665]: video_items_df.breakdown[0]
Out[1665]: 
{'platform': 'facebook',
 'total': 18463,
 'likes': 9436,
 'shares': 5581,
 'comments': 608,
 'tweets': '',
 'favorites': '',
 'hahas': 1049,
 'wows': 170,
 'loves': 1554,
 'sads': 44,
 'angrys': 21}

Answered By - Mayank Porwal