Working at JPMorgan this summer, I found I spent much of my time working on internal tools to improve the efficiency of the team.

A recurring bottleneck was working with official statements (OS) of municipal bonds. These 100+ page documents were not page-turners, but they outlined important legal and financial details of an issuance. Unfortunately, the OS was often available only as scanned PDFs (essentially images of text), which made the text non-searchable and painful to copy, edit, or summarize.

To try to extract text from the images, my coworkers turned to Large Language Models (LLMs), JPMorgan called its in-house tool LLMSuite (essentially a ChatGPT wrapper in a secure internal environment). It failed.

LLMs struggle at this task for three reasons:

Scale: The way LLMs break up images means that it rapidly uses up its ontext window.
Lack of Determinism: The “G” in GPT refers to the generative nature of LLMs.
Semantics: The “large” in Large Language Models refers to the billions of parameters that govern the model, giving it greater “world knowledge” or context, which it uses to reason.

Just as LLMs can't process text directly (text must first be converted into numbers called tokens), when you upload an image to a multimodal LLM, the model divides the image into a grid of small squares (patches), and converts each patch into a numerical representation. The model then processes these numerical representations using the same mechanisms it uses for language.

Reason 1: Scale

This is where we encounter the first problem: the hundred-plus-page OS quickly consumed the model's context window. Encoding meaning for a single page image might use 1,000-2,000+ tokens depending on resolution. Trying to extract text from a 100+ page OS could mean 100,000-200,000 tokens before accounting for the prompt or output generation. For reference, ChatGPT currently has a context window of 16,000-196,000 tokens, depending on the model.

But context window is only part of the problem.

Reason 2: Lack of Determinism

The generative nature of LLMs means output is always a "best next token" prediction. Even when we want pure extraction, it's still generating. This is called autoregression: the model produces one token, feeds it back as input, produces the next token, and repeats.

This explains why the output is not deterministic (You might not get the same output every time).

Reason 3: Semantics

The generative models, paired with the billions of parameters that make up these models, creates a tendency for the model to output words that make contextual sense, even if they're not what's actually in the image. The parameters are the weights and biases that tune a model based on its vast training data.

If a blurry word appears in a medical form, the LLM doesn't merely analyze pixel geometry; it leverages its massive internal knowledge base of medical terminology to infer "Patient", effectively hallucinating the correct answer based on context.

This contextual reasoning is extremely valuable in complex situations, such as trying to decipher handwriting. But it inherently introduces points of failure and has meaningful inference costs.

If you need to extract text from thousands of pages of reasonably legible documents, purpose-built Optical Character Recognition (OCR) technologies are, and will continue to be the better tool.

While the overarching objective remains to convert pixels into machine-readable text that you or your computer can manipulate, the approach differs.

OCR-type technologies have been used as far back as the 1910s, and have been widely adopted in critical tasks like processing checks and reading license plates. It's even responsible for how your handwritten addresses on postcards actually find their way to the right destinations.

Traditional OCR processes an image through a series of steps, using pattern recognition (Not AI) to identify individual characters or words.

Preprocessing:
1. Straightening an image (’deskewing”)
2. Binarization: Determine the color of each pixel, and convert them to either a 0 (white) or a 1 (black) to separate the background from the areas of the image where there is text.
Segmentation: Grouping of adjoining pixels of the same color (the black pixels that make up the text)
Classification: Matching the segmented objects to a library of known characters (different fonts and languages)
Post Processing: Using dictionaries to spell check or correct possible errors.

This approach is deterministic, but it has zero semantic awareness. If a coffee stain creates a loop on a vertical line, a traditional OCR engine sees "P" instead of "l". It cannot use the surrounding context to correct this error because it operates at the character level. It "sees" without "reading.”

The technology has advanced over the years and incorporated different architectures, including the use of AI and Machine learning. Modern OCR uses neural networks, but with a fundamentally different architecture than LLMs.

Generally, they use three components:

CNN (Convolutional Neural Network): Scans the image and extracts visual features.
LSTM (Long Short-Term Memory): Reads through the visual features left-to-right, outputting a probability distribution over all possible characters at each position.
CTC (Connectionist Temporal Classification): Collapses the probability distribution into clean text. If the model outputs "HHHeeelllloo," CTC turns it into "Hello.”

The model outputs all character probabilities in one forward pass. There's no feedback loop where each token depends on previous generations.

Transformer OCR (TrOCR)

The latest generation of OCR replaces LSTM with transformers.

TrOCR is autoregressive, but unlike LLMs the text decoder is small and trained specifically for OCR, not general reasoning.

TrOCR represents a middle ground.

Most text extraction doesn't require understanding. It requires accuracy and speed. Use OCR for what it's good at (extraction). Use LLMs for what they're good at (reasoning).

The structure that I find works the best is dual-layered:

Scanned Document -> OCR (extract text) -> LLM (summarize, answer questions, analyze)

In the case of the team at JPMorgan, multi modal LLMs weren't the right tool, and while I didn't have time to build them a more comprehensive solution, a simple locally hosted UI that used pdfplumber was a step in the right direction (although I now know it was a suboptimal engine).

For my own use, I built myself an OCR tool specifically to help me summarize books: I upload a PDF, it runs OCR, and then it generates chapter-by-chapter summaries using an LLM.

The key lesson (learned the hard way) was not relying on the LLM to extract the text in the first place.

I'm sharing it in case it helps anyone who's similarly frustrated by forgetting what they read in books.

Broadly, it be might valuable to continue thinking about some of the foundational technologies that most of us know little about, but might merit a surface-level exploration to help us identify the right solution for our problems, even ones as mundane as extracting text from a PDF.

Built-in / No-setup options:

Google Drive: Upload a PDF or image, right-click, and select "Open with Google Docs." It runs OCR automatically and outputs editable text. The catch: file size is limited to 2MB (Which is about 2-15 pages for a scanned image).
Apple Live Text (iPhone/Mac): Open an image in Photos, long-press on the text, and select it directly. Limit here is that single image.

Browser-based tools for larger jobs:

Adobe Acrobat Online: Can handle larger PDFs than Google Drive, but you hit paywalls pretty quick.

There is a variety of OCR tools that appear by doing a google search but their pricing and quality varies.

Cloud APIs

Google Cloud: First 1,000 units per month is free. You need to setup a project, enable billing. You then get a custom API endpoint to call . The browser test only supports 15 pages.
Azure Document Intelligence: The free tier offers 500-2000 pages per month. I spent 30 minutes trying to sign up and I still wasn’t able to use it.
Amazon Textract: First 1,000 pages/month free for 3 months. Requires AWS setup.

Local/Open-source (no limits, requires setup):

Tesseract: The most established open-source OCR engine. Access it in Python. Free, and no page limits.
EasyOCR / PaddleOCR: Newer deep learning-based alternatives that can outperform Tesseract on complex layouts and multilingual documents, but require more computational resources.

As I found out, unless you are an enterprise client, accessing these types of technologies (without the restrictive size and page limitations) is currently very difficult.
I would recommend you take the time to vibe code yourself a tool that uses the open-source OCRs or use my web-based OCR.

SINGULARITY

Get new memos by email