91 lines
4.7 KiB
TeX
91 lines
4.7 KiB
TeX
\documentclass[a4paper, 14pt]{scrartcl}
|
|
\usepackage[utf8]{inputenc}
|
|
\usepackage[english]{babel}
|
|
\usepackage{parskip}
|
|
\usepackage{microtype}
|
|
\usepackage[margin=1in]{geometry}
|
|
\usepackage{hyperref}
|
|
|
|
\title{Stock Trading with Reinforcement Learning}
|
|
\author{Marcel Zinkel}
|
|
|
|
\begin{document}
|
|
\maketitle
|
|
\tableofcontents
|
|
|
|
\section{Introduction}
|
|
I want to build a reinforcement learning project about single asset stock trading. First I
|
|
want to start a simple environment with just the actions buy and sell. For the reward function I
|
|
also want to keep it simple at first by just using the profit as reward.
|
|
|
|
In contrast to the algorithms we already heard in the lecture, I have to try out deep
|
|
reinforcement learning algorithms because the price is a continuous variable. In theory, you could
|
|
model the price with a specific resolution with many states. However, this can very quickly become
|
|
impractical for classic reinforcement learning methods. Also, deep reinforcement learning can
|
|
recognize pattern to act good in previously unseen states.
|
|
|
|
I want to try out different reinforcement learning algorithms to see with works best for the trading
|
|
environment. First I want to try out the Deep-Q-Network algorithm. It predicts the Q-function and
|
|
uses Epsilon-Greedy Exploration. I plan to try out different formulas for the epsilon decay.
|
|
|
|
Because DQN often overestimates the Q-values, I want to try out a variation of DQN, called Double
|
|
DQN. It uses two networks for updating the policy. The
|
|
online network selects the action with the highest Q-value and the target network evaluates the
|
|
action. This causes more stable and better learning. I will try, if Double DQN will improve the
|
|
results. At last, I want to try out the Proximal Policy Optimization algorithm.
|
|
|
|
After implementing these different algorithms, I need to train these and compare the
|
|
results.
|
|
|
|
I find it also very interesting, if providing the RL agent with additional information then just the
|
|
price, positively impacts the results. For example, I can add technical indicators, market volume or
|
|
an online news score about the company. The last one is probably a bit difficult because you need a
|
|
LLM which gives web scrapped articles a score how good the news is for a company. After adding this
|
|
information, I need to reevaluate which algorithm is the best.
|
|
|
|
\section{Libraries and Tools}
|
|
The project will be implemented in Python using \texttt{gym-anytrading} to build the trading
|
|
environment. For initial experiments, I will use the built-in datasets from
|
|
\texttt{gym\_anytrading.datasets} such as \texttt{STOCKS\_GOOGL}, and later switch to real
|
|
historical stock data via \texttt{yfinance}.
|
|
|
|
The reinforcement learning algorithms will be implemented using the \texttt{stable-baselines3}
|
|
library. I will start with the standard DQN algorithm and experiment with different epsilon decay
|
|
strategies. Since \texttt{stable-baselines3} does not directly support Double DQN, I plan to modify
|
|
the DQN implementation myself. Specifically, I will adjust the target calculation so that the action
|
|
is selected using the online network but evaluated using the target network, as required in Double
|
|
DQN. This will allow me to better understand the internal workings of the algorithm and directly
|
|
control its behavior.
|
|
|
|
In addition to DQN and Double DQN, I will also train PPO using the standard implementation in
|
|
\texttt{stable-baselines3}.
|
|
|
|
After training, I will evaluate all models using backtesting and performance metrics like total
|
|
profit, Sharpe ratio, and maximum drawdown. Later, I plan to extend the observation space with
|
|
technical indicators, volume data, or sentiment features. For technical indicators, I will use the
|
|
\texttt{pandas-ta} library since it is easy to install, well integrated with \texttt{pandas}, and
|
|
provides a wide range of indicators sufficient for prototyping and research.
|
|
|
|
After adding these features, I will retrain the models and compare their performance again.
|
|
|
|
|
|
\section{Development plan}
|
|
Depending on the exact time my presentation will be scheduled, I have about 9-10 weeks of time.
|
|
|
|
\subsection{Week 1--3}
|
|
I want to integrate the DQN algorithm as an example and train it already with historical data.
|
|
|
|
\subsection{Week 4--6}
|
|
I plan to implement the other RL algorithms and the variations and evaluate which works best. Also
|
|
change the reward function.
|
|
|
|
\subsection{Week 7 to the presentation}
|
|
Add the technical indicators and market volume to the environment. If I have too much time left, I can
|
|
try news analysis.
|
|
|
|
\section{Availability}
|
|
I am on vacation from 04.08 to 13.08. On the 15. I am on an event, but I have time on the 14.
|
|
From the 18. onwards I am available for the next couple of weeks. I look forward to the
|
|
presentation, and thank you for giving me the additional time.
|
|
\end{document}
|