Reinforcement Learning in Portfolio Management
- Degree
- Bachelor in Data Science and Engineering
- Institution
- Universidad Carlos III de Madrid
- Supervisor
- Francisco Javier Nogales Martín
Most machine learning in finance ends up as a supplementary tool for a human expert: it forecasts a return, scores a signal, flags a risk, and a person still makes the call. This thesis asks what it takes to close that loop — to design a Deep Reinforcement Learning system that manages a portfolio as an independent decision-maker, optimising allocation rather than merely informing it.
The agent is built on a Temporal Convolutional Network pre-trained on a supervised return-forecasting task and then transferred as a frozen state encoder, so the representation of market history is learned separately from the allocation policy. On top of it sits a Soft Actor-Critic (SACv2) agent, with actor variants that place either a Normal or a Dirichlet distribution over the portfolio simplex, and critic variants using bilinear and single-head attention mechanisms.
Two design choices matter most. A confidence score lets the agent decide whether rebalancing is warranted at all, not only how to reallocate — which makes it agnostic to the granularity of the decision interval. And randomly permuting assets during training pushes the policy to be company-agnostic, learning from market structure instead of memorising tickers. Mixup augmentation across price channels rounds out the training recipe.
what's in it
- TCN backbone pre-trained on supervised return forecasting, then frozen and transferred as the state encoder — separating representation learning from policy learning.
- Soft Actor-Critic (SACv2) with Normal and Dirichlet actor variants over the portfolio simplex, so allocations are valid weights by construction.
- Critic variants built on bilinear and single-head attention mechanisms.
- A confidence score that lets the agent skip rebalancing, making it granularity-agnostic rather than tied to a fixed trading cadence.
- Random asset permutation during training to encourage company-agnostic policies, plus mixup augmentation across price channels.
- Benchmarked against equal-weight (with and without rebalancing), minimum-volatility and maximum-Sharpe efficient-frontier portfolios, and a random agent.
evaluation
- Universe
- S&P 500, chronological splits
- Training
- 1986 – 2008
- Validation
- 2009 – 2016
- Test
- 2017 – 2023, unseen companies
- Baselines
- Equal-weight, min-volatility, max-Sharpe, random
- Stated limitations
- Simulated environment; survivorship bias in the data
Licensed CC BY-NC-ND 4.0. Cite as: Christopher Manzano Vimos. (2023). Reinforcement Learning in Portfolio Management [Bachelor Thesis]. Universidad Carlos III de Madrid. Zenodo. https://doi.org/10.5281/zenodo.21541556