min park

NLP and Data Analytics

LLM-assisted end-to-end Financial News Analysis pipeline

July 26, 2026Project

Overview

Since I put my foot into the financial domain with NLP, here are several tasks piqued my interests.

Sentiment Analysis

Yes, sentiment analysis might sound boring as it is one of the most common NLP tasks. That's true in applications such as product reviews where the core focus it to get the overall sentiment of the text.

This is why I set out the Multi-Entity Sentiment Analysis fine-tuning task (previous posts: Original, Revised).

Topic clustering and ticker mapping

Previously, I used BERTopic for topic modeling tasks, based on the BERTopic package.

While the package can cluster texts into certain number of topics, the generated topic title (topic representation) was often not intuitive. Additionally, mapping tickers to the topic clusters and understanding why they are included in the clusters often required manual work.

Thanks to embedding chunks and LLM-powered summary generation, topic clustering and the (plausible) explanation for the clusters along with mapping tickers to them can be enabled.

Automated ticker management

Before and now, the pipeline mainly depends on tickers.

Previously, I occasionally added newly listed tickers to the list of watchlist of tickers.

Since this adds manual intervention, I've always wanted to automate this process.

Now, thanks to LLM, tracking newly large listed tickers can be done with ingested articles in hand.

Data pipelines

Using Prefect, multiple-stage data are being processed.

News ingestion

As the main materials are news articles, the first step is to ingest financial news articles, which will be embedeed, clustered and to be used in sentiment analysis, IPO detection, etc.

Embedding chunks

To prepare for topic clustering, embeddings in overlapping chunks are being done as a prerequisite step.

Sentiment Analysis

Using the fine-tuned sentiment model, sentiments of each entity in a text are being extracted.

The extracted properties from the sentiment model are as follows.

  • Entity: (Subject and object)
  • Polarity: + / - / 0 / ~
  • Category: Legal / Business / Performance / Recruitment / NewsRelease / Bankruptcy
  • Entity_type: ORG / PERSON / GPE / OTHER

Topic clustering

Using a modified version of BERTopic, topics are being clustered from the embeddings chunks.

While the basic steps follow that of BERTopic, to allow better clustering, once clustered, they undergo merging and hierarchical grouping process assisted by the LLM. Moreover, as the main focus is in financial articles, post-filtering applies where only-financial topics are being created (since it's being judged by a SLM, it might not be perfect though).

Mapping tickers to the topics and summary generation

With the topic clusters are in hand, chunks are being scanned to extract which entities are related to which clusters.

Since the financial news often includes many text where certain entities might not be directly related (i.e., banks' analysis of economic status), filtering using the sentiment model, which was intentionally trained to handle these cases, along with more targted post-processing, are being done.

After certain chunks are extracted for each entity, the LLM generate summary of that entity in that cluster, which will show how those tickers are related to the topic.

Automated ticker management

To track large IPOs, LLM tracks recent articles and record possible upcoming listing and their proposed ticker symbol if available in the text. Additionally, to verify the actual IPO plans, once tickers are identified, they are being checked against Yahoo Finance, where upcoming plans are available.

Ticker prices

Prices of tickers which have been extracted at least once are being ingested.

Company Summary

Using the data from the previous steps and the 10-K filings, company summary are generated with the following.

  • Company profile: LLM generates from 10-K filings
  • Company positioning: LLM generates the company's current positioning based on the analyzed text
  • Price
  • Sentiment
  • Peer comparison: Based on the analyzed news and industry (from 10-K), extract peer groups

Outputs

Each daily pipeline produces the JSON outputs to strat-data repo.

  • Market Briefings (briefings.json): Daily market overview including economic and market snapshots and market movers (based on price changes and sentiment changes)
  • Topic Clusters (topics_*.json): News are being grouped into topic clusters
  • Ticker Sentiments (sentiment_*.json): Each ticker's sentiment (within and out of clusters) is analyzed
  • Ticker Prices (prices_*.json): Each ticker's price data are being gathered
  • Company Summary (company_meta.json): Summary of company info with its related peers