[FIXED] Using DictWriter to write a CSV when the fields are not known beforehand

Issue

I am parsing a large piece of text into dictionaries, with the end objective of creating a CSV file with the keys as column headers.

csv.DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds)

The problem arises as the dict for any 'n'th row can include a new, never before used key. I then want the CSV to contain a column for this new key as well. In short, all my fields are not known beforehand so I cannot compile a complete fieldnames at the beginning.

Is there a recommended way to have csv.DictWriter not ignore missing fields but add them to fieldnames instead? Merely changing fieldnames at this point would leave the prior lines with an incorrectly lower number of fields.

Solution

Instead of using DictWriter which can be confusing in your case as dictionaries are not ordered I tried using writerow method of csv. Here is what i did :

"""
a) First took all the keys of dictionary and sorted it, which is not necessary.
b) Created a result list which appends value related the headers which is key of our input dict and if key is not available then .get() will return None. 
   So result list will contain lists for rows data.
c) Wrote header and each row from result list in csv file
"""

data_dict = [{ "Header_1":"data_1", "Header_2":"data_2", "Header_3":"data_3"},
             { "Header_1":"data_4", "Header_2":"data_5", "Header_3":"data_6"},
             { "Header_1":"data_7", "Header_2":"data_8", "Header_3":"data_9", "Header_4":"data_10"},
             { "Header_1":"data_11", "Header_3":"data_12"},
             { "Header_1":"data_13", "Header_2":"data_14", "Header_3":"data_15"}]

"""
   In the third dict we have extra key, value.
   In forth we dont have have header_2 were we aspect blank value in our csv file.
"""
process_data = [ [k,v] for _dict in data_dict for k,v in _dict.iteritems() ]           

headers = [ i[0] for i in process_data ]
headers = sorted(list(set(headers)))

result = []
for _dict in data_dict:
    row = []
    for header in headers:
        row.append(_dict.get(header, None))
    result.append(row)


import csv
with open('demo.csv', 'wb') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=';', dialect='excel', 
                            quotechar='|', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(headers)    
    for r in result:
        spamwriter.writerow(r)

Answered By - Tanveer Alam