[FIXED] Real time data plotting from a high throughput source

Issue

I want to plot Real time in a way that updates fast.

The data I have:

• arrives via serial port at 62.5 Hz
• data corresponds to 32 sensors (so plot 32 lines vs time).
• 32points *62.5Hz = 2000 points/sec

The problem with my current plotting loop is that it runs slower than 62.5[Hz], meaning I miss some data coming in from serial port.

I am looking for any solution to this problem that allows for:

• All data from serial port to be saved.
• Plots the data (even skipping a few points/using averages/eliminating old points and only keeping the most recent)

Here is my code, I am using random data to simulate the serial port data.

import numpy as np 
import time 
import matplotlib.pyplot as plt

#extra plot debugging
hz_ = [] #list of speed
time_=[] #list for time vs Hz plot


 #store all data generated
store_data = np.zeros((1, 33))
 #only data to plot 
to_plot = np.zeros((1, 33)) 
 #color each line 
colours = [f"C{i}" for i in range (1,33)]

fig,ax = plt.subplots(1,1, figsize=(10,8))
ax.set_xlabel('time(s)')
ax.set_ylabel('y')
ax.set_ylim([0, 300])
ax.set_xlim([0, 200])

start_time = time.time()
for i in range (100):
    loop_time = time.time()
     #make data with col0=time and col[1:11] = y values
    data = np.random.randint(1,255,(1,32)).astype(float) #simulated data, usually comes in at 62.5 [Hz]
    data =  np.insert(data, 0, time.time()-start_time).reshape(1,33) #adding time for first column
    store_data = np.append(store_data, data , axis=0)
    to_plot = store_data[-100:,]
    
    for i in range(1, to_plot.shape[1]):
        ax.plot(to_plot[:,0], to_plot[:,i],c = colours[i-1], marker=(5, 2), linewidth=0, label=i)
        #ax.lines = ax.lines[-33:] #This soluition speeds it up, to clear old code. 

    fig.canvas.draw()  
    fig.canvas.flush_events()
    Hz = 1/(time.time()-loop_time)
     #for time vs Hz plot
    hz_.append(Hz)
    time_.append( time.time()-start_time)
    print(1/(time.time()-loop_time), "Hz - frequncy program loops at")
   
 #extra fig showing how speed drops off vs time
fig,ax = plt.subplots(1,1, figsize=(10,8))
fig.suptitle('Decreasingn Speed vs Time', fontsize=20)
ax.set_xlabel('time(s)')
ax.set_ylabel('Hz')

ax.plot(time_, hz_)
fig.show()

I also tried while using

ax.lines = ax.lines[-33:]

to remove older points, and this speed up the plotting, but still slower than the speed i aquire data.

Any library/solution to make sure I collect all data and plot the general trendlines (so even not all points) is ok. Maybe something that runs acquiring data and plotting in parallel?

Solution

You could try to have two separate processes:

• one for acquiring and storing the data
• one for plotting the data

---

Below there are two basic scripts to get the idea. You first run gen.py which starts to generate numbers and save them in a file. Then, in the same directory, you can run plot.py which will read the last part of the file and will update the a Matplotlib plot.

Here is the gen.py script to generate data:

#!/usr/bin/env python3

import time
import random

LIMIT_TIME = 100  # s
DATA_FILENAME = "data.txt"


def gen_data(filename, limit_time):
    start_time = time.time()
    elapsed_time = time.time() - start_time
    with open(filename, "w") as f:
        while elapsed_time < limit_time:
            f.write(f"{time.time():30.12f} {random.random():30.12f}\n")  # produces 64 bytes
            f.flush()
            elapsed = time.time() - start_time
            

gen_data(DATA_FILENAME, LIMIT_TIME)

and here is the plot.py script to plot the data (reworked from this one):

#!/usr/bin/env python3


import io
import time
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.animation


BUFFER_LEN = 64
DATA_FILENAME = "data.txt"
PLOT_LIMIT = 20
ANIM_FILENAME = "video.gif"


fig, ax = plt.subplots(1, 1, figsize=(10,8))
ax.set_title("Plot of random numbers from `gen.py`")
ax.set_xlabel("time / s")
ax.set_ylabel("random number / #")
ax.set_ylim([0, 1])


def get_data(filename, buffer_len, delay=0.0):
    with open(filename, "r") as f:
        f.seek(0, io.SEEK_END)
        data = f.read(buffer_len)
        if delay:
            time.sleep(delay)
    return data


def animate(i, xs, ys, limit=PLOT_LIMIT, verbose=False):
    # grab the data
    try:
        data = get_data(DATA_FILENAME, BUFFER_LEN)
        if verbose:
            print(data)
        x, y = map(float, data.split())
        if x > xs[-1]:
            # Add x and y to lists
            xs.append(x)
            ys.append(y)
            # Limit x and y lists to 10 items
            xs = xs[-limit:]
            ys = ys[-limit:]
        else:
            print(f"W: {time.time()} :: STALE!")
    except ValueError:
        print(f"W: {time.time()} :: EXCEPTION!")
    else:
        # Draw x and y lists
        ax.clear()
        ax.set_ylim([0, 1])
        ax.plot(xs, ys)


# save video (only to attach here) 
#anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1, frames=3 * PLOT_LIMIT, repeat=False)
#anim.save(ANIM_FILENAME, writer='imagemagick', fps=10)
#print(f"I: Saved to `{ANIM_FILENAME}`")

# show interactively
anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1)
plt.show()
plt.close()

Note that I have also included and commented out the portion of code that I used to generate the animated GIF above.

I believe this should be enough to get you going.

Answered By - norok2