A Comparative Analysis of Prediction-Based and Reconstruction-Based Deep Learning Models for Time Series Anomaly Detection in Spacecraft Telemetry

Abstract: Anomaly detection in multivariate spacecraft telemetry is usually judged with the point-adjusted (PA) F1 score, a protocol known to reward any detector that fires at least once inside a labelled segment. This paper reports a controlled comparison of three deep models, an LSTM predictor, an LSTM autoencoder and a GRU autoencoder with additive attention, on twelve channels of the NASA SMAP and MSL benchmark. All models share one preprocessing, windowing, smoothing and thresholding pipeline and are scored with both PA and strict point-wise F1; each is also set against two trivial baselines, a random detector and the norm of the raw input. Under PA the random detector reaches a micro- averaged F1 of 0.96, above every trained model (0.79–0.81), while its point-wise F1 is only 0.02 against 0.44–0.45 for the networks. The raw-signal baseline matches the networks on both protocols (0.81 and 0.46). With a fixed threshold multiplier the autoencoders score zero PA F1 on 5 channels; once the multiplier is tuned per channel the three architectures become indistinguishable within run-to-run variation, although on three SMAP channels the PA scores of every model rest on only 2–14 correctly flagged points (point-wise F1 ≤ 0.03), and two contextual-anomaly channels remain undetected by all of them. The results indicate that evaluation protocol and threshold calibration, rather than architecture, account for most of the differences reported on this benchmark.
Keyword: Spacecraft telemetry, multivariate time series, LSTM, Anomaly detection, Autoencoder, attention, SMAP, MSL, point adjustment.