Hidden Markov Models for Regime Detection in Trading

Every trader has had the experience of a strategy that works beautifully for six months and then quietly stops working. The instinct is to blame execution, slippage, or transaction costs, but the more common culprit is that the market entered a regime the strategy was not designed for. Momentum works in trending markets and bleeds in mean-reverting ones. Volatility selling works in calm regimes and blows up in crisis regimes. A strategy that does not condition on regime is implicitly making the wrong assumption: that the market is stationary. Hidden Markov Models are the cleanest mathematical tool for detecting regimes.

This article explains the HMM approach to regime detection: the math, the fitting procedure, the parameters that matter, and the common ways the model goes wrong in practice. The audience is someone who has used HMMs in other contexts (speech recognition, biological sequences) and wants to apply them to financial time series, or a quantitative trader who has heard about regime models and wants to understand the mechanics.

The Markov chain intuition

A Markov chain models a system that occupies one of a finite set of states and transitions probabilistically between them, with the transition probabilities depending only on the current state (the "Markov property"). For financial markets, the states are unobserved, we call this a "hidden" Markov model. We do not directly see whether the market is in a "low-volatility bullish" regime or a "high-volatility bearish" regime; we only see returns. The HMM infers the hidden state from the observed returns.

The model has three components. (1) A set of K hidden states. (2) A K×K transition matrix A where A[i,j] is the probability of moving from state i to state j on the next time step. (3) An emission distribution for each state, typically a Gaussian with mean μₖ and variance σₖ², that describes what observations look like when the chain is in state k. With these three components and a starting distribution π, the HMM is fully specified.

Fitting an HMM: the Baum-Welch algorithm

Given a return series and a chosen number of states K, fitting an HMM means estimating the transition matrix A and the emission parameters (μₖ, σₖ²) for each state. The Baum-Welch algorithm is the standard approach. It is a specialization of the expectation-maximization (EM) framework to HMMs.

The algorithm alternates two steps. The E-step computes, for each time t, the posterior probability that the chain was in each state at time t given all observations (forward-backward algorithm). The M-step uses these posteriors as soft assignments to re-estimate the transition matrix and emission parameters. The two steps repeat until the likelihood converges. The result is a maximum-likelihood estimate of the HMM parameters.

Convergence is usually fast (10-50 iterations for a small-K model on monthly data), but the algorithm finds only local optima. The standard remedy is to run Baum-Welch from many random initializations and keep the best (highest-likelihood) result. ARIA Analyst runs 20 random starts per HMM fit and picks the maximum-likelihood solution.

How many states?

The single most important hyperparameter is K, the number of hidden states. There is a vast literature debating K = 2, 3, 4, or 5 for equity returns. The empirical answer for most use cases is K = 2 or K = 3. The K = 2 model captures the basic "calm vs. turbulent" split that explains most of the volatility-of-volatility in equities. K = 3 adds a "transitional" state that catches the early-warning regime before a crisis. K ≥ 4 typically overfits, the additional states either collapse onto each other or pick up idiosyncratic events that do not generalize.

The principled way to choose K is by held-out log-likelihood or AIC/BIC. Run the model with K = 2, 3, 4, 5 on a training period, evaluate likelihood on a held-out test period, and pick the K that maximizes test-period likelihood. In our experience, K = 2 wins for daily S&P 500 returns; K = 3 wins for monthly multi-asset data including bonds and commodities.

What the states look like in practice

A typical K = 2 fit on daily S&P 500 returns from 1990-2024 produces two states with the following characteristics. The "calm" state has annualized mean return ~10%, volatility ~12%, transition probability to itself ~98% (highly persistent). The "turbulent" state has annualized mean return ~-15%, volatility ~35%, self-transition ~92%. The unconditional probabilities are roughly 80% calm, 20% turbulent.

These numbers are stable across reasonable training windows. The transitions occur predictably around crises, 1998 LTCM, 2000-02 tech crash, 2008 financial crisis, 2020 COVID, and the model can detect regime changes one to three trading days after they begin, well in advance of most technical indicators.

Using HMM regime probabilities in trading

The output of a fitted HMM is, for each time t, a probability distribution over hidden states. The trading question is: what do you do with these probabilities? The naive approach, "go long when calm-state probability > 0.7, go short or to cash when turbulent-state probability > 0.5", works, but it discards information by binarizing a probability.

A more sophisticated approach is to use the posterior state probabilities to weight strategy outputs. ARIA Analyst maintains separate strategy parameters per regime, different momentum windows, different mean-reversion thresholds, different position-sizing schedules, and blends the output as a probability-weighted combination of the per-regime strategies. The result is a smooth strategy that transitions gracefully across regimes rather than flipping abruptly.

The state probabilities also feed downstream models. The ML ensemble in ARIA Analyst conditions its forecast on the HMM regime (via the regime × horizon bundle structure) so that the same feature combination produces different forecasts in calm vs. turbulent markets. See our blog post on 5 ways to detect market regimes for a comparison of HMM with alternative methods.

Common pitfalls

• Choosing K by eye. The fitted states look "interpretable" with any K, so it is tempting to pick the K that produces the cleanest narrative. Use held-out likelihood instead.
• Fitting on the full sample. Standard HMM fitting uses all data through time T. For a trading model, the fit at time t should use only data up to time t, otherwise you have look-ahead bias. Re-fit the model in a walk-forward manner.
• Ignoring the persistence parameter. The self-transition probability matters more than the mean returns. A state with 99% persistence is qualitatively different from one with 70%.
• Treating state probabilities as state labels. The output is a probability distribution. Acting only on the argmax discards the uncertainty in the inference and produces brittle decisions.
• Over-engineering with K > 3. For most equity applications, K = 2 or K = 3 is optimal. Higher K overfits.

Alternatives to HMM

HMM is one approach to regime detection. The alternatives include: rolling-window volatility cuts (mechanical, robust, but discards information about return mean), structural break tests (Bai-Perron, useful for one-shot regime change detection), Markov-switching GARCH (combines HMM with time-varying volatility), and supervised classifiers on macro features (more flexible but harder to interpret). For most applications, HMM is the best starting point because it requires only return data and produces interpretable states with calibrated uncertainty.

Conclusion

Hidden Markov Models are the cleanest mathematical approach to detecting unobserved regimes in financial time series. With K = 2 or K = 3 states, fitted via Baum-Welch on rolling windows and used as a soft conditioning signal for downstream strategies, HMMs add real value to almost any trading model. They are particularly useful as the routing layer for ensemble systems that maintain separate sub-models per regime.

ARIA Analyst uses an HMM-based regime detector as the routing layer for its 9-bundle ML ensemble. Create a free account to see live regime probabilities for any ticker, or read our regime detection methods post for the broader picture. The scoring methodology piece explains how regime conditioning feeds the final score.