Sagemaker Debugger and forecasting

Answer #1: Based on the same technology used for time-series forecasting at Amazon.com, Forecast provides state-of-the-art algorithms to predict future time-series data based on historical data, and requires no machine learning experience. Amazon SageMaker Debugger profiles and debugs your training jobs to improve the performance of machine learning models on compute resource utilization and model predictions. So Debugger is available for your SageMaker training jobs but not for Amazon Forecast which is a fully managed service of its own.

Answer #2 - The way to detect overfitting with forecasting is using "backtests", which are the "train/valid" equivalent of traditional machine learning. The reason Backtesting is used in forecasting is to keep the time order of the train/valid data. With careful analysis of backtest windows, you can have more confidence the model will generalize well with unseen data.

Amazon Forecast now supports export of backtest forecasts as .csv files. Customer can take the backtest forecasts and calculate whatever metric they want and/or visualize.

• Link to documentation: https://docs.aws.amazon.com/forecast/latest/dg/metrics.html
• Link to blog and notebook about using Backtests in Amazon Forecast: https://aws.amazon.com/blogs/machine-learning/amazon-forecast-now-supports-accuracy-measurements-for-individual-items/

One common fear is backtesting will increase overfitting - it does not since each validation dataset is different from the previous backtest validation dataset. To have more certainty about robustness of the model, use more than 1 backtest window. If the Forecast Horizon is long, you may get by with just 1-2 backtest windows; otherwise use the maximum which is 5 backtest windows.

Another concern is the increased training time since each backtest window is in fact another train/forecast iteration. It's a trade-off, if you want more model validation or not.

Ideally, the wQL's of each backtest window are similar. If you see all backtest forecasts with similar metrics (except maybe a few windows where underlying data had anomalies), that is good sign your model is not overfit.

Both under-fitting and over-fitting for now are left to human judgement to examine backtest forecasts and decide whether model is acceptable or not.