How do you know whether a model is broken? Or, how do you conduct a model review? There are many specific steps for any review but there are four major questions that have to be addressed that are separate from risk management. The key question of model efficacy should focus on forecasting skill and action. Does the model forecast correctly and is the model employed properly to make good decisions.
1. Is the data used by the model correct or timely? There is a wealth of data that can be used for building a model, yet this key input can be subject to delays and revisions. Hence, the first place to look to see if a model is doing its job is with the data. Many managed futures managers focus on price for the simple reason it is easy to get the data. The data can be smoothed to obtain a signal and the process of obtaining price data is cheap and efficient. This provides tremendous benefits for the investor. The same cannot be said for fundamental data. Any model with fundamental data is likely to have a delay issue, so data review is important.
2. Is the model using the data correct? It is clear that many financial models can only explain a small portion of the variation of the dependent variable. Additionally, models are subject to structural change when there are changes in the economy and there can exist alternative models that seem to fit the data but can still be wrong. Simply put, is the model theoretically sound, structurally stable, and consistent? This answer can change through time.
3. Is the interpretation of the model output correct? Given the low explanatory power of many models, there is still a lot of room for interpretation for what a model is telling us. The strength of the signal generated from the model can be weak.
4. Is the mapping from model to a decision action correct? What separate modeling research from investing is the mapping from the model to a decision. If the model is given too much weight positions may be too large versus the strength of the signal. Too little weight and the model will not generate any excess return value.
Every investor who uses a model should have a regular model assessment or due diligence. We offer a simple four-step process for discussion purposes. The investor may want to have a deeper analysis with standards for quality control and forecasting skill. What is important is having a review process for model assessment.