Kalman Filter Python: Tutorial and Strategies – Part I

The Kalman filter, developed by Rudolf Kalman in the 1960s, is a powerful mathematical tool used for estimating the state of a dynamic system from a series of noisy measurements. Originally designed for aerospace applications, the Kalman filter has found widespread use in various fields, including finance and trading.

At its core, the Kalman filter combines information from a series of measurements with predictions from a dynamic model to produce optimal estimates of the system’s state. It does so by recursively updating its estimate based on new measurements, while also taking into account the uncertainty associated with both the measurements and the model predictions.

This blog covers:

• What is a Kalman filter?
• Applications of Kalman filter in trading
• Real-world examples of Kalman filter usage in trading
• Key statistical terms and concepts of Kalman filter
  • Kalman filter equations
  • Status update equation
  • State extrapolation equation
  • Kalman gain equation
  • Estimate uncertainty update
  • Estimate uncertainty extrapolation
• Kalman filter and other filtering techniques
• Steps for implementing Kalman filter in Python
• Pairs trading using Kalman filter in Python
• Future trends and developments in Kalman filter technology

What is a Kalman filter?

Imagine the Kalman filter as a useful conductor leading an orchestra of data. What is the use of it? ⁽¹⁾

The use would be to seamlessly merge noisy measurements with predictive models and craft an estimation of a system’s state. This blend of past observations and dynamic forecasts is the secret, empowering traders to sail through the uncertainty in the markets with confidence.

Next, we will talk about the applications of the Kalman filter in the trading domain.

Applications of Kalman filter in trading

Below are some useful applications of the Kalman filter in trading.

1. Pairs Trading: One common application of the Kalman filter in trading is pairs trading, where traders identify pairs of assets with a historically stable relationship and exploit deviations from this relationship. The Kalman filter can be used to dynamically estimate the hedge ratio between the two assets and adjust trading positions as the relationship evolves over time.
2. Volatility Estimation: In options trading, accurately estimating volatility is crucial for pricing and risk management. The Kalman filter can be employed to estimate volatility from noisy market data, providing traders with more reliable inputs for option pricing models and hedging strategies.
3. Market Impact Modelling: When executing large trades, traders need to consider the potential impact on market prices. The Kalman filter can help in modelling market impact by estimating the relationship between trade size and price movements, allowing traders to optimise trade execution strategies to minimise costs.
4. Portfolio Optimisation: The Kalman filter can aid in portfolio optimisation by estimating the covariance matrix of asset returns. By incorporating these estimates into mean-variance optimisation models, traders can construct more efficient portfolios that maximise returns for a given level of risk.

Now, let us move ahead to find out the real world examples of usage of the Kalman filter.

Real-world examples of Kalman filter usage in trading

One quite interesting real world example of Kalman filter usage is depicted in an article by Bayes Business School in the United Kingdom. An event was held in the school in 2020 led by Dr Veronika Lunina, Quantitative Vice President at NatWest Markets.

In this event, Dr Veronika spoke about the use of the Kalman filter and was positive about her own experiences using the extended Kalman filter for automated marking of FX implied volatility surface.

According to a research paper, Nkomo et al. (2013) introduced the Kalman filter to process stock price data and proposed the K-AC-M algorithm based on the Kalman filter, leveraging momentum effects to expand the AC algorithm and obtained superior excess returns in strategy simulation compared to the AC algorithm.

Jin et al. (2013) initially combined the traditional autoregressive (AR) model with the Kalman filter to obtain improved predictive performance over a single AR model and a single Kalman filter. They further combined the support vector regression (SVR) with the UKF into a new model, with SVR used to address parameter selection issues in the UKF. ⁽²⁾

As such, the Kalman filter can be considered a heavy topic when it comes to the use of maths and statistics. Thus, we will go through a few terms before we dig into the equations. Feel free to skip this section and head directly to the equations if you wish.

Key statistical terms and concepts of Kalman filter

Kalman Filter uses the concept of a normal distribution in its equation to give us an idea about the accuracy of the estimate. Let us step back a little and understand how we get a normal distribution of a variable.

Let us suppose we have a football team of ten people who are playing the nationals. As part of a standard health check-up, we measure their weights. The weights of the players are given below.

Now if we calculate the average weight, ie the mean, we get the value as (Total of all player weights) / (Total no. of players)

= 720/10 = 72

The mean is usually denoted by the Greek alphabet μ. If we consider the weights as w1, w2 respectively and the total number of players as N, we can write it as: μ = (w1 + w2+ w3+ w4+…..+ wn)/N

Or

Now, on a hunch, we decide on seeing how much each player’s weight varies from the mean. This can be easily calculated by subtracting the individual’s weight from the mean value.

Now, the first team player’s weight varies in the following manner,

(Individual player’s weight) – (Mean value) = 72 – 72 = 0.

Similarly, the second player’s weight varies by the following: 75 – 72 = 3.

Let’s update the table now.

Now, we want to see how much the entire team’s weights vary from the mean. A simple addition of the entire team’s weight difference from the mean would be 0 as shown below.

Thus, we square each individual’s weight difference and find the average. Squaring is done to eliminate the negative sign of a score + penalise greater divergence from the mean.

The updated table is as follows:

Now if we take the average, we get the equation as,

The variance tells us how much the weights have been spread. Since the variance is the average of the squares, we will take the square root of the variance to give us a better idea of the distribution of weights. We call this term the standard deviation and denote it by σ.

Thus,

Since standard deviation is denoted by σ, the variance is denoted by σ2.

But why do we need standard deviation?

While we calculated the variance and standard deviation of one football team, maybe we could find for all the football teams in the tournament, or if we are more ambitious, we can do the same for all the football teams in the world. That would be a large dataset.

One thing to understand is that for a small dataset we used all the values, i.e. the entire population to compute the values. However, if it is a large dataset, we usually take a sample at random from the entire population and find the estimated values.

In this case, we replace N by (N-1) to get the most accurate answer as per Bessel’s correction. Of course, this introduces some errors, but we will ignore it for now.

Thus, the updated equation is,

Now, looking at different research conducted in the past, it was found that given a large dataset, most of the data was concentrated around the mean, with 68% of the entire data variables coming within one standard deviation from the mean.

This means that if we had data about millions of football players, and we got the same standard deviation and variance which we received now, we would say that the probability that the player’s weight is +-3.46 from 72 kg is 68.26%. This means that 68.26% of the players’ weights would be from 68.53 kg to 75.46.

Of course, for this to be right, the data should be random.

Let’s draw a graph to understand this further. This is just a reference of how the distribution will look if we had the weights of 100 people with mean as 72 and standard deviation as 3.46.

This shows how the weights are concentrated around the mean and taper off towards the extremes. If we create a curve, you will find that it is shaped like a bell and thus we call it a bell curve. The normal distribution of the weights with mean as 72 and standard deviation as 3.46 will look similar to the following diagram.

Normal distribution is also called a probability density function. While the derivation is quite lengthy, we have certain observations regarding the probability density function.

• One standard deviation contains 68.26% of the population.
• Two standard deviations contain 95.44% of the population while three contain 99.74%.

The probability density function is given as follows,

The reason we talked about normal distribution is that it forms an important part in Kalman filters.

Let’s now move on to the Kalman filter equations.

Author: Chainika Thakar (Originally written by Rekhit Pachanekar)

Stay tuned to learn about Kalman filter equations.

Originally posted on QuantInsti blog.

Go Source