GPT-Powered Stock Trading Research

For my senior research project I asked a simple question. Can a large language model read a news article about a company and predict whether its stock price will go up or down?

Background

This was before ChatGPT took over the internet. Back then, transformer-based text models lived mostly with AI researchers and the nerdiest programmers. OpenAI released GPT-2 in early 2019, and that is when I jumped on the hype train. Like everyone else, I wanted to use transformers for fun and profit. OpenAI limited access to big research labs and deep-pocketed companies, so that plan stalled. In the meantime, Ben Wang and Aran Komatsuzaki built an open-source team to compete. They shipped GPT-J by early 2021. It was free, open, and close enough to GPT-2 and GPT-3 for my needs. That timing mattered. I used their work in my senior research project for my college degree.

Timeline

My Research Project

Concept

In mid-2021, people were excited about how flexible large transformer models looked. I wanted to test GPT-J as an all-in-one news-based trading bot. The bet was simple. If a model has read enough of the internet, it should grasp a news article and return a trading signal for a company.

Input: news article about a company → Output: buy, sell, or neutral signal

Data gathering

I fine-tuned GPT-J-6B on data that matched the prompts I planned to test. I scraped and labeled 140,000 news articles from the sources below:

Rate limits were a problem, so I built a custom scraper. It managed a pool of rotating proxies. I also added a custom rate limiter so I did not hammer any one domain. I ran the scraper on an OracleVM. It pulled about 40 articles per minute and finished all 140,000 articles in about 2.5 days.

class Scraper:
    """
    Scrapes a list of seed urls and calls the parser function to process each result.
    The parser function should accept two arguments: the response object and the url.
    The parser may pass a list of new urls to scrape with the addUrls(urls) method.
    The scraper attempts to never scrape the same url twice.
    """
    def __init__(self, seedURLs = [], parser = None, options = None):
        emptyLogFile()
        self.options = {
            'cookieDirectory': './cookies/',
            'scrapeThreads': 10,
            'rateLimits': {},
            'maxAttempts': 3,
            'proxyUpdateInterval': 10
        }
        if options:
            self.options.update(options)

        self._cookies = loadCookies(self.options['cookieDirectory'])
        self._parser = parser

        self._urlQueue = queue.PriorityQueue()
        self._finishedUrls = {}
        self._finishedUrlsLock = threading.Lock()
        self.addUrls(seedURLs)

        self._threadNumber = self.options['scrapeThreads']

        self._proxySessions = queue.Queue()
        for proxy in self.verifyProxies(self.getProxies()):
            self._proxySessions.put(proxy)

        self._rateLimits = {}
        self._rateLimitsLock = threading.Lock()
        for domain in self.options['rateLimits']:
            self._rateLimits[domain] = {
                'limit': self.options['rateLimits'][domain],
                'lastRequest': {}
            }

    def addUrls(self, urls):
        """ Adds a list of urls to the list of urls to scrape. """
        count = 0
        with self._finishedUrlsLock:
            for url in urls:
                if url not in self._finishedUrls:
                    self._urlQueue.put((0, url))
                    count += 1

    def run(self):
        """ Starts the scraping threads and the proxy maintenance thread. """
        scrapingThreads = []
        for i in range(self._threadNumber):
            t = threading.Thread(target = self._scrape)
            t.start()
            scrapingThreads.append(t)

        proxyMaintThread = threading.Thread(target = self._maintainProxies)
        proxyMaintThread.start()

        for t in scrapingThreads:
            t.join()
        proxyMaintThread.join()
        successfulScrapes = len([v for v in self._finishedUrls.values() if v])
        print(f"Finished scraping. {len(self._finishedUrls)} urls scraped. {successfulScrapes} successful scrapes.")

I then split the dataset. Training got 90,000 articles. Validation got 10,000. Testing got 40,000.

Training

The model weights are over 60GB, so I needed specialized compute. I trained on a Google TPU-v3 from Google's TPU Research Cloud. These were the hyperparameters:

{
  "layers": 28,
  "d_model": 4096,
  "n_heads": 16,
  "n_vocab": 50400,

  "warmup_steps": 160,
  "anneal_steps": 1530,
  "lr": 1.2e-4,
  "end_lr": 1.2e-5,
  "weight_decay": 0.1,
  "total_steps": 1700,

  "tpu_size": 8,

  "bucket": "peckham_tpu_europe",
  "model_dir": "mesh_jax_stock_model_slim_f16",

  "train_set": "stocks.train.index",
  "val_set": {
    "stocks": "stocks.val.index"
  },

  "val_batches": 5777,
  "val_every": 500,
  "ckpt_every": 500,
  "keep_every": 10000
}

Training Loss

At first, a clear drop in training loss looked promising. I now think that drop was mostly overfitting.

Results

Directional accuracy on held-out articles (did the model guess up/down correctly?):

Conclusion

In my tests, GPT-J does no better than chance at predicting price direction from a news story. Fine-tuning does not improve accuracy either.

I think a few factors explain that:

• Outside a small fine-tune, the model was not trained on financial data.
• The data still had too much noise.
• Text is a weak stand-in for financial signal. The model cares more about how numbers sound in the article than about the numbers themselves.
• Many articles may not contain enough information to predict price moves. News often lags price. A lot of the articles I scraped came out after the move had already happened.