LLMs vs PFNs: Why ChatGPT-style AI Struggles with Spreadsheets
LLMs like ChatGPT, Claude and Gemini write, code and pass hard exams. So can they also look at a spreadsheet and predict who will leave, which loan will be repaid, or what a house will sell for?
This kind of data, rows and columns like an Excel sheet, is called tabular data, and it’s everywhere. People already try to use LLMs on it. But research tells a surprising story: on simple table tasks, LLMs often lose to decades-old statistical methods. Meanwhile, a different kind of AI built only for tables, the Prior-Data Fitted Network (PFN), now leads benchmarks on smaller datasets.
Here’s why, in simple terms.
What is tabular data, and why is it different?
Tabular data is simply data arranged in a table, made of rows and columns, like an Excel sheet. Each row is one example (a customer, a patient, a house). Each column is one piece of information about it.
Here’s a small made-up example:
The goal is to learn from rows where we already know the answer (the last column), and then predict the answer for new rows.
A table might look like just another kind of text. But tables have three special properties that make them very different from the sentences LLMs are trained on:
- The order of rows doesn’t matter. Shuffle the rows and the data means exactly the same thing. Shuffle the words in a sentence and the meaning is gone.
- The order of columns doesn’t matter either. “Age” could be the first column or the last, and nothing changes. Text depends heavily on word order.
- Tables mix different types of values. One column has numbers (income), another has yes/no answers (has loan), another has categories (city). Text is just a stream of words.
Tree models like XGBoost, and PFNs, are designed around these rules. LLMs are not: they read a table as one long line of text.
LLMs and PFNs in brief
An LLM is trained on huge amounts of text to predict the next word (or token). It sees numbers as text too: GPT-4o writes 0.1875 as four separate pieces, “0”, “.”, “187” and “5”.
A PFN uses the same transformer building block but practices on a huge number of made-up (synthetic) tables, each hiding a rule to discover. The idea comes from the 2022 paper Transformers Can Do Bayesian Inference (Müller et al.). Its best-known version is TabPFN, now developed by Prior Labs; the latest is TabPFN-3.5.
Both share one trick, in-context learning: you put example rows in the input and the model predicts new rows on the spot, with no retraining. So both play the same game. The difference iswhat they practiced on first.
How do LLMs actually perform on tables?
In a 2026 study, researchers gave a top LLM (Claude Opus 4.6) whole tables in one prompt and compared it with six classic methods on 11 simple tasks.
The LLM scored 0.42. Even the weakest classic method scored 0.70.
Why do LLMs struggle with tables?
Researchers point to six problems. Some explain poor accuracy; others explain why good-looking results can’t be trusted.
They memorize famous datasets
Popular datasets like Iris, Wine and Titanic have been online for decades, and GPT-4 can write out Iris and Wine from memory. Its stock forecasts show the effect clearly:
When researchers changed famous datasets very slightly, GPT-4’s accuracy fell by about 6 points. On new datasets, it didn’t beat a simple logistic regression on any of the five tested. One table-trained LLM even had a dataset’s 753 test answers in its training data.
2. Too many columns confuse them
This may be the biggest finding. When researchers spread the same information across more columns, every classic method stayed the same or improved. The LLM was the only one of nine methods that got worse, and by about 16 columns it was close to guessing the most common answer on most datasets.
3. Their answers aren’t stable
Changes that shouldn’t matter moved an LLM’s prediction error a lot, while a random forest barely moved:
Telling it “the order doesn’t matter” didn’t help. It’s like a calculator that gives a different total when you add the same numbers in a different order.
4. They rely on column names, not numbers
When column names like “Age” were replaced with X1, X2, X3, LLM accuracy dropped sharply. So much of an LLM’s success comes from knowing what the columns mean, not from learning the numbers.
5. They’re expensive and run out of space
A table with 100,000 rows and 20 columns would need about 16 million tokens, more than a million-token context window holds, and about $80 per attempt in input alone.
6. Their explanations can’t be trusted
On 20 simple tasks, the LLM’s stated rule didn’t match the data in 14, even when its predictions were right.
Myths busted: what is not the main problem
A 2026 study tested five popular explanations. Only one held up.
In short, the LLM reads tables fine; it just can’t think with them as columns grow. This doesn’t clash with problem 3: format isn’t what makes LLMs inaccurate, but it still makes their answers wobble.
Where LLMs can still help
- Very small tables. With around 64 labeled rows or fewer, understanding what the columns mean can beat pure statistics.
- Clear column names. World knowledge helps when columns are called “Age” or “Income”.
- Writing the code. An LLM can write code that trains a proper model. That’s different from, and usually safer than, asking it to predict directly.
How do PFNs perform?
The best evidence comes from TabArena, a regularly updated leaderboard for table models. Its first version tested 16 models on 51 real datasets. On datasets within its size limits, TabPFN v2 beat the other models by a large margin, and it won on more datasets (11) than any other model. In a separate study with 4,000 training rows, TabPFN was far more accurate than GPT-4o-mini and Llama-3.
Where TabPFN fits best:
- Table size: TabPFN v2 handled up to 10,000 rows and 500 columns; newer versions may handle more.
- Speed or extra accuracy: the default version predicts in seconds; a tuned version is even more accurate but slower.
- Largest datasets: on the full benchmark, tuned neural networks and tree models still ranked top. TabPFN shines on small-to-medium tables.
Why there's no fair LLM vs PFN race yet
A 2026 Prior Labs article explains why LLMs are missing from leaderboards like TabArena:
- Format choices: turning a table into CSV, JSON or Markdown can change the results unfairly.
- Cost: running an LLM over hundreds of datasets costs far more than any other method.
- Memorization: a fair test would need hidden datasets, which goes against open research.
- Little reason to pay: every signal so far says LLMs aren’t good at this.
TabArena also leaves out very small datasets (under 500 rows), exactly where LLMs do best. The article expects LLMs to join leaderboards once benchmarks with hidden test data exist.
Honest notes
- Some authors build tabular models. The Prior Labs article comes from TabPFN’s maker, the Why LLMs Fail authors work on their own model (Nexus), and some TabArena authors co-wrote TabPFN v2. Independent university teams reach similar conclusions.
- Not all are peer-reviewed. Why LLMs Fail and Robustness is Important are preprints.
- Models change fast. Several studies used older models (GPT-4, GPT-4o-mini, TabPFN v2), so check the live TabArena leaderboard for current results.
Practical takeaways
- Real tables with hundreds or thousands of rows: use TabPFN or a similar model for smaller datasets, or proven tools like XGBoost, LightGBM or CatBoost.
- A handful of rows with clear column names: an LLM may give a useful first guess, but check it against a simple baseline.
- Testing an LLM on a famous dataset: be suspicious of great results. Free tools like
tabmemcheckcheck for memorization. - Always compare against “predict the most common answer”. In one study, a model with 94.9% accuracy on a stroke dataset matched exactly what that simple guess scored.
Why PFNs are the right tool for tables
Put the evidence together and PFNs offer exactly what LLMs lack on tabular data:
Their main limit is size: for very large tables, tuned tree models remain the go-to. But most business tables, a few hundred to a few thousand rows of customers, loans, machines or patients, are well within reach. There, the combination is hard to beat: top accuracy without hyperparameter tuning, answers in seconds, and results that don’t change because a column was renamed.
Conclusion
LLMs taught us something useful: knowing language isn’t the same as learning from data. They can read a table, but as columns grow they struggle, their wins on famous datasets are often memory, and harmless changes shift their answers.
PFNs were built for exactly the job LLMs find hard. They practice on endless synthetic tables until learning from a table is second nature, and it shows: top accuracy on small-to-medium datasets, answers in seconds, and results that stay put.
The lesson is simple: use LLMs for words and code, and use PFNs like TabPFN when you need predictions from tables.
Ready to transform the way you solve business challenges?
ProfitOps doesn’t just flag what will fail. It finds the root cause, recommends the intervention, and gets smarter with every plant you add so your operations outlast the experts who built them.
Shravan Talupula
Founder, ProfitOps
