[FIXED] Graphing a log file with Python

Issue

To start with I don't usually use any scripting language other than bash. However I have a requirement to graph the data from a large number of environmental monitoring log files from our server room monitor and feel that Python will work best. I am using Python 3.7 for this and have installed it and all the, so far, required libraries via macports and pip.

I want to end up with at least 7 graphs, each with multiple lines. Four of the graphs are the temperature and humidity data for each physical measurement point. Two of them are for air flow, hot and cold, and the last is for line voltage.

I have attempted to start this on my own and have gotten decently far. I open the log files and extract the required data. However getting the data into a graph seems to be beyond me. The data to be graphed is a date and time stamp, as X, and a number which is a dotted decimal, which should always be positive, as Y.

When extracting the date I am using time.strptime and time.mktime to turn it into a Unix epoch, which works just fine. When extracting the data I am using re.findall to remove the non-numerical portions. I plan to move the date from an epoch to date and time but that can come later.

When I get to the graphing portion is where I am having the issue.

I first tried graphing the data directly which gave me the error: TypeError: unhashable type: 'numpy.ndarray'

I have also tried using a Pandas dataframe. This gave me the error: TypeError: unhashable type: 'list'

I have even tried to convert the lists to a tuple both with and without the dataframe, the same errors were give.

Based on the output of my lists I think the issue is with using append for the values that will be the Y axis. However I cannot seem to Google well enough to find a solution.

The code, outputs seen, and input data is below. The comments are there from the last run, I use them for testing various portions.

Code so far:

# Import needed libraries
import re
import time

import matplotlib.pyplot as plt
import pandas as pd
#import matplotlib.dates as mpd

# Need to initialize these or append doesn't work
hvacepoch = []
hvacnum = []
endepoch = []
endnum = []

# Known static variables
datepattern = '%m-%d-%Y %H:%M:%S'

# Open the files
coldairfile = open("air-cold.log","r")

# Grab the data and do some initial conversions
for coldairline in coldairfile:

        fields = coldairline.split()

        colddate = fields[0] + " " + fields[1]
#       coldepoch = mpd.epoch2num(int(time.mktime(time.strptime(colddate, datepattern))))
        coldepoch = int(time.mktime(time.strptime(colddate, datepattern)))
        coldnum = re.findall('\d*\.?\d+',fields[4])
        coldloc = fields[9]

        if coldloc == "HVAC":
                hvacepoch.append(coldepoch)
                hvacnum.append(coldnum)

        if coldloc == "Cold":
                endepoch.append(coldepoch)
                endnum.append(coldnum)


# Convert the lists to a tuple. Do I need this?
hvacepocht = tuple(hvacepoch)
hvacnumt = tuple(hvacnum)
endepocht = tuple(endepoch)
endnumt = tuple(endnum)

# Testing output
print(f'HVAC air flow date and time: {hvacepoch}')
print(f'HVAC air flow date and time tuple: {hvacepocht}')
print(f'HVAC air flow numbers: {hvacnum}')
print(f'HVAC air flow numbers tuple: {hvacnumt}')
print(f'Cold end air flow date and time: {endepoch}')
print(f'Cold end air flow date and time tuple: {endepocht}')
print(f'Cold end air flow numbers: {endnum}')
print(f'Cold end air flow numbers tuple: {endnumt}')

# Graph it. How to do for multiple graphs?

# With a Pandas dataframe as a list.
#colddata=pd.DataFrame({'x': endepoch, 'y1': endnum, 'y2': hvacnum })
#plt.plot( 'x', 'y1', data=colddata, marker='', color='blue', linewidth=2, label="Cold Aisle End")
#plt.plot( 'x', 'y2', data=colddata, marker='', color='skyblue', linewidth=2, label="HVAC")

# With a Pandas dataframe as a tuple.
#colddata=pd.DataFrame({'x': endepocht, 'y1': endnumt, 'y2': hvacnumt })
#plt.plot( 'x', 'y1', data=colddata, marker='', color='blue', linewidth=2, label="Cold Aisle End")
#plt.plot( 'x', 'y2', data=colddata, marker='', color='skyblue', linewidth=2, label="HVAC")

# Without a Pandas dataframe as a list.
#plt.plot(hvacepoch, hvacnum, label = "HVAC")
#plt.plot(endepoch, endnum, label = "Cold End")

# Without a Pandas dataframe as a tuple.
#plt.plot(hvacepocht, hvacnumt, label = "HVAC")
#plt.plot(endepocht, endnumt, label = "Cold End")

# Needed regardless
#plt.title('Airflow\nUnder Floor')
#plt.legend()
#plt.show()


# Close the files
coldairfile.close()

The output from the print lines(truncated):

HVAC air flow date and time: [1588531379, 1588531389, 1588531399]
HVAC air flow date and time tuple: (1588531379, 1588531389, 1588531399)
HVAC air flow numbers: [['0.14'], ['0.15'], ['0.15']]
HVAC air flow numbers tuple: (['0.14'], ['0.15'], ['0.15'])
Cold end air flow date and time: [1588531379, 1588531389, 1588531399]
Cold end air flow date and time tuple: (1588531379, 1588531389, 1588531399)
Cold end air flow numbers: [['0.10'], ['0.09'], ['0.07']]
Cold end air flow numbers tuple: (['0.10'], ['0.09'], ['0.07'])

The input(truncated):

05-03-2020  14:42:59   Air Velocit 0.14m/ Under Floor Air Flow HVAC                                   
05-03-2020  14:42:59   Air Velocit 0.10m/ Under Floor Air Flow Cold End                               
05-03-2020  14:43:09   Air Velocit 0.15m/ Under Floor Air Flow HVAC                                   
05-03-2020  14:43:09   Air Velocit 0.09m/ Under Floor Air Flow Cold End                               
05-03-2020  14:43:19   Air Velocit 0.15m/ Under Floor Air Flow HVAC                                   
05-03-2020  14:43:19   Air Velocit 0.07m/ Under Floor Air Flow Cold End                   

Solution

I just checked your data and it looks like the issue is that endnum and hvacnum are not lists of values. They're lists of lists, as you can see below:

In [1]: colddata.head()
Out[1]:
         x        y1      y2
0   1588531379  [0.10]  [0.14]
1   1588531389  [0.09]  [0.15]
2   1588531399  [0.07]  [0.15]

So, when you go to plot the data, matplotlib doesn't know how to plot those rows. What you can do is use a list comprehension to grab unpack the list.

In [2]:
    print(endnum)
    print(hvacnum)
Out[2]:
    [['0.10'], ['0.09'], ['0.07']]
    [['0.14'], ['0.15'], ['0.15']]

In [3]:
    endnum = [i[0] for i in endnum]
    hvacnum = [i[0] for i in hvacnum]
    print(endnum)
    print(hvacnum)
Out[3]:
    ['0.10', '0.09', '0.07']
    ['0.14', '0.15', '0.15']

Answered By - LTheriault