Python Forecasting Techniques Comparison

Introduction

Performance comparison of various time series forecasting techniques, including deep learning methods, presented as Python implemenations in Jupyter notebooks. These notebooks are written so that they can be read and understood in isolation, and as such, there may be repetition in certain parts between them. The models presented in these notebooks have not been fully-optimised, and are intended to serve as a starting framework for anyone interested in implementing these forecasting techniques in their own projects.

This repository is actively being updated with more forecasting methods and datasets.

Forecasting Techniques

The forecasting techniques demonstrated in this repository are as follows:

• Average Method - A simple benchmark model¹
• (Seasonal) Naïve Method - A simple benchmark model¹
• Linear Regression
• ARIMA²
• ARIMAX²
• SARIMA²
• SARIMAX²
• Prophet - An open source time series forecasting method from Facebook³
• SageMaker DeepAR - A deep learning method on the AWS SageMaker platform⁴
• Ludwig (WIP) - A toolkit created by Uber on top of TensorFlow for training and testing deep learning models⁵

Datasets

Datasets that exhibit seasonality, and those that don't, are treated separately.

Seasonal Datasets:

• California Solar Power (CSP) - Power output of 405 photovoltaic power stations at five minute intervals⁶
• Household Electricity Consumption (HEC) - Electricity consumption of 370 households in Portugal at 15 minute intervals⁷
• Hourly Weather Temperature (HWT) - Hourly temperatures for 36 US, Canadian and Israeli cities⁸
• Hourly Weather Wind Speed (HWS) - Hourly wind speeds for 36 US, Canadian and Israeli cities⁸
• Hourly Weather Wind Directions (HWS) - Hourly wind directions for 36 US, Canadian and Israeli cities⁸
• San Francisco Traffic (SFT) - Hourly road occupancy rates for various San Francisco Bay Area freeways⁹

Non-Seasonal Datasets:

• Daily Exchange Rates (DER) - Daily exchange rates between 1990 and 2016 of eight countries (Australia, Britain, Canada, Switzerland, China, Japan, New Zealand, Singapore)¹⁰

Evaluation

For non-seasonal datasets, the mean absolute scaled error (MASE)¹¹ is used to evaluate and compare forecasting accuracies for the different techniques. For seasonal datasets, the seasonal variant (sMASE) is used.

Results

The results are shown below and are also viewable in this notebook.

Seasonal Datasets:

	CSP	HEC	HWT	HWS	HWD	SFT
SARIMA	0.734733	1.60976	1.25492	0.806742	0.713394	1.06255
SARIMAX	0.75265	1.61433	1.3084	0.865172	0.731366	1.1555
Naive	1	1	1	1	1	1
DeepAR	1.0884	1.69218	1.92888	0.795072	0.754109	0.717766
Prophet	1.10443	1.63378	1.32722	0.849303	0.999691	1.4419
LinearRegression	1.46422	2.03795	1.15451	0.821911	0.830871	1.51436
ARIMA	3.14786	2.04262	1.18837	0.794829	0.838663	1.26169
ARIMAX	3.62378	2.01298	1.4158	0.889121	0.875279	1.40512
Average	5.20921	2.39973	2.11568	0.813868	0.878027	1.95147

Non-Seasonal Datasets:

	DER
Naive	5.42765
DeepAR	5.7467
ARIMA	6.1503
Average	16.7777

---

References:

1: Forecasting: Principles and Practice
2: Autoregressive integrated moving average (Wikipedia)
3: facebook prophet
4: Amazon SageMaker DeepAR Forecasting
5: Uber Ludwig
6: National Renewable Energy Laboratory
7: UCI Machine Learning Repository
8: Kaggle
9: Caltrans PeMS
10: GitHub/laiguokun
11: Forecasting: Principles and Practice - Chapter 3.4