[FIXED] how to detect anomalies for multiple time series?

Issue

I have about 1000 time series each time series is a record of an api latency i want to detect anoamlies for all the time series. Should I create a for loop to train one model(LSTM-Autoencoder) per api(per one time series) ? Or is there a way to fit one model that can detect anomalies for all the time series ?

Solution

The answer to your question is "it depends".

What's important is that you understand what each of these approaches means.

Should I create a for loop to train a model (LSTM-Autoencoder) for each time series ?

In this case it means your model is going to learn sequentially from each time series for each API. This can work well IF the anomaly you're trying to detect and the latency pattern are consistent across the different API. It's just a different way of having your model iterate over a dataset containing all your time series.

IF there's no such consistency and each API display different latency pattern and anomaly, learning sequentially would not make sense. Imagine trying to predict the weather by training your model on temperature, then retraining on wind speed... most likely your model is not going to get any good as it will not learn the interaction between this two factor in such a setting.

Or is there a way to fit one model that can detect anomalies for all the time series ?

There is, you could have your problem structured as dataset with 1000 features (1 for each API). The difficulty might arise if your Time Series are not sampled at the same frequency, you might have to resample them. But in such a scenario you could develop a model that monitor all thousand API.

Of course this is theoretical, as with a 1000 features you might need a lot of sample to avoid the "curse of dimensionality".

Answered By - Yoan B. M.Sc