minimizing portfolio tracking error yield unlogical results

Question

I have a pd.DataFrame with daily stock returns of shape (250,10) and a pd.Series with my benchmarks daily returns of shape (250,). The goal is to minimize the Tracking Error, between a portfolio of stocks and the benchmark. The tracking error is the Standard Deviation of (Portfolio - Benchmark) But somehow scipy.minimize can't correctly minimize the tracking error function, the results just don't make any sense. For other functions like maximizing the return it works flawless.

In the end the two lines should be very similar, but they aren't. Scipy doesn't complain but the results are just not what one would expect, do you know why this objective function troubles scipy?

MWE:

import numpy as np
import pandas as pd
from scipy.optimize import minimize

def portfolio_te(weights, rets, bm_rets):
    port_returns = np.dot(rets, weights)
    te = np.sqrt(np.mean((port_returns - bm_rets)**2))
    return te

stocks_returns = pd.DataFrame(np.random.normal(0, 0.02, (250, 10)))
benchmark_returns = pd.Series(np.random.normal(0, 0.02, 250))

result = minimize(portfolio_te, x0=[0.1 for i in range(10)],
        bounds=[(0,0.3) for i in range(10)], method='SLSQP',
        args=(stocks_returns, benchmark_returns))
        
port_returns = pd.Series(np.dot(stocks_returns, result.x))

ts = pd.concat([(1+port_returns).cumprod(), (1+benchmark_returns).cumprod()], axis=1)
ts.plot()

Answer 1

It's not clear to me that there is a solution to this problem.

You have 10 stocks that you can pick, which all have random returns. You have an index which is also randomly created. You would expect the correlation between any two sets of random numbers to be nearly zero, and therefore that the match between any portfolio and the benchmark would be fairly poor.

If I change the problem so that it does have a solution, by randomly creating some weights, computing the stock returns given those weights, and then solving for the original weights given those given the stock returns, it works fairly reliably.

true_weight = np.random.rand(10)
true_weight /= true_weight.sum()  # Ensure weights sum to 1
stocks_returns = pd.DataFrame(np.random.normal(0, 0.02, (250, 10)))
benchmark_returns = pd.Series((stocks_returns.values @ true_weight))

Now that you know the true weights, you can check if the minimization worked to find them.

average_pct_error = np.abs((((result.x - true_weight) / true_weight) * 100)).mean()
print(f"Average portfolio component was wrong from true weight by {average_pct_error:.2f}%")

The cumulative return graph also agrees.

I get on the order of 0.1% error when doing this, which you can lower to near zero through the ftol option:

result = minimize(portfolio_te, x0=[0.1 for i in range(10)],
        bounds=[(0,0.3) for i in range(10)], method='SLSQP',
        options=dict(ftol=1e-10),
        args=(stocks_returns, benchmark_returns))

An alternative way to specify the loss function would be to minimize cumulative returns RMSE. This improves the graphs you're making, but I would worry that it over-values matching returns early in the time series vs. matching them later.