One Model to Rule Them All. Well, Most of Them.

Introduction

OOne fine morning, whilst sipping on the coffee I'd brewed on my AeroPress, I found myself doing that thing where you audit your entire career before 9 AM. Specific question: which single piece of technology have I used the most?

Turns out it's XGBoost.

I started using it at Yahoo, building regression models to pre-fill data for advertising campaigns. Even now, when I'm building graph neural networks for bio-chemistry, the first thing I reach for is an XGBoost baseline. In my three and a half years at Egen, almost everything we shipped was built on XGBoost or another boosting algorithm. Sometimes a Random Forest, sometimes a soft-voting ensemble, occasionally something more exotic. We shipped these to Fortune 500 companies and they worked.

Fifteen-year-old technology. Runs on a laptop. Still paying the bills.

This got me thinking about something that's been bothering me. A lot of companies talk and preach about AI adoption, but what most of them are actually doing is outsourcing grunt work to LLMs. It's the 1990s all over again, except instead of shipping the boring parts to an IBM mainframe and a services contract, we're shipping them to a context window and a per-token bill.

LLMs are excellent generalists. But for specific problems you want specific models. You don't want a general practitioner performing your surgery, no matter how well-read he is.

And a lot of teams would benefit enormously from just using boosting algorithms — XGBoost, LightGBM, CatBoost — to transfer tribal knowledge into machines. That's the actual unglamorous opportunity sitting in most organisations right now. Not agents. Not fine-tuning. Just taking the twenty rules that live in one senior person's head and the fourteen exceptions nobody wrote down, and turning them into something that runs every night at 2 AM.

The Research

Where boosting shines is tabular data, and almost every business on earth runs on tabular data. An Excel sheet, a Postgres table, a warehouse, a CSV somebody emails around. That's the substrate.

Even now, working on problems in molecular biology and bio-chemistry, our XGBoost baselines land around F1 ~0.82 and AUC ~0.86. That's quite good. For a lot of problems that is not the baseline — that's the answer, and everything after it is diminishing returns you'll spend two quarters chasing.

This isn't just me being stubborn, incidentally. On the Therapeutics Data Commons ADMET benchmarks — molecular property prediction, exactly the domain where you'd expect graph networks to dominate — a large share of state-of-the-art results still come from gradient-boosted trees on molecular fingerprints. A 2025 benchmark of 25 pretrained molecular embedding models found that nearly all of them showed negligible improvement over a plain ECFP fingerprint baseline. In chemistry. Where the molecules are literally graphs. I found that genuinely humbling.

Then there's everything that has nothing to do with accuracy:

• Trains in seconds to minutes, so nightly retraining is a cron job, not a platform initiative.
• Runs fine on CPU for medium-sized data. No GPU in the serving path, no CUDA version in your postmortems.
• A small server handles many parallel inference requests. For on-device use cases, a decent model runs on a Raspberry Pi or a phone.
• Explainability that's predictable and that a compliance reviewer will actually accept.
• It fails loudly rather than strangely. A tree that's never seen a region says something dumb and obvious. A network says something confident and wrong.

XGBoost is the Toyota Hilux of machine learning. Nobody puts it on a conference poster. It starts.

The literature backs this up more than the discourse suggests. The Grinsztajn et al. benchmark put tree ensembles against a spread of tabular-specific neural architectures across 45 datasets, with a tuning budget for everyone, and found trees still state of the art on medium-sized data — around 10K samples. The diagnosis was the interesting part: networks get dragged down by uninformative features, and their rotation invariance actively works against them when the meaningful structure is the individual columns. Shwartz-Ziv and Armon arrived at the same place from a different direction.

The one I'd actually point you to, though, is McElfresh et al. — 19 algorithms across 176 datasets. Their conclusion is a knife aimed at both camps: the neural-nets-versus-trees debate is overemphasised, and for a large fraction of datasets the difference is negligible, or light hyperparameter tuning on a gradient-boosted tree matters more than which family you picked in the first place.

Sit with that for a second. A benchmark paper's finding is that the argument everyone is having is mostly not the thing that determines the outcome.

The Implementation for Most Problems

Complexity is expensive and only gets harder. Simplicity is cheap, and you can always climb from there.

So we've established that boosting models and tree ensembles are simple and solve most problems, and that we're fine starting simple and letting probability do us a favour. If we fail or outgrow the model, we iterate upwards. That's a much better position than discovering in month four that your transformer was never the problem.

The other thing worth saying: you don't need PhDs or AI researchers for this. Most engineers and analysts with a decent understanding of the problem can build these models. Heck, even curious business people who aren't technical can get there with an LLM and a bit of guidance. That's not a knock on the work — it's the entire point. The scarce input here isn't modelling talent. It's someone who knows what the data means.

Here's roughly how I run it:

1. Data analysis. Understand the shape of the data and the relationships between features. Do a proper EDA — patterns, outliers, missing values, cardinality, distributions. Sit with it longer than feels productive.
2. Data refinement. Clean it. Handle missing values, outliers, categoricals, numericals. Decide what missing actually means — missing-because-not-collected and missing-because-not-applicable are two different features.
3. Baseline immediately. Before any of the clever stuff, train a default-parameter model. This is your floor and your leak detector. If you're sitting at 0.99 AUC on day one, congratulations, you've found a bug, not a model.
4. Feature importance study. Understand which features matter and how they relate. PCA, t-SNE, UMAP as needed — but treat these as exploration, not evidence.
5. Feature engineering. Create new features, drop dead ones. This is where the real gains live, and it's the step that requires knowing the business rather than knowing the library.
6. Feature selection. Either start with everything and trim, or start small and add. Both work. Pick one and be disciplined about it.
7. Split properly. Split by time and by entity, not randomly. Most spectacular validation scores are just a random split quietly leaking the future into the past. Leakage is a documented reproducibility crisis across seventeen scientific fields, and I promise your pipeline is not the exception.
8. Train. XGBoost, LightGBM, CatBoost, Random Forest. Pick one, they're all fine.
9. Evaluate. RMSE, MAE, R², F1, AUC — whatever matches the decision the model feeds. This is your honest baseline.
10. Hyperparameter tuning. Grid search, random search, Bayesian optimisation. I use Optuna. Note that random search beats grid search at equal budget, which Bergstra and Bengio showed in 2012 and half the industry still hasn't internalised. Then walk away and let it run — this is the part of the job you should be delegating to a machine, not savouring.
11. Iterate on the gap. Ensembling, stacking, blending. Push F1 and AUC where it's worth pushing.
12. Calibrate. If a human or a threshold consumes your score, calibrate it. An uncalibrated 0.7 that actually means 0.4 will cost you more than three points of AUC ever will.
13. Explainability. SHAP values, permutation importance. Use it to understand the model and to catch the feature that's secretly a proxy for the label.
14. Deployment. Docker, Kubernetes, whatever your org already runs. The model is a file. Don't overthink it.
15. Monitoring. Simple logging on CloudWatch or Prometheus. Watch for drift and performance decay.

Iterate and iterate and iterate.

Notice how much of that list isn't modelling. Steps 1, 2, 4, 5, 6, 7 and 13 are data work and business logic wearing an ML costume. The actual “machine learning” is step 8, and step 8 is one line.

Real-World Example

At Egen I worked with large Fortune 500 companies — giant retailers, insurance providers, non-banking financial institutions. We were almost always solving data problems, not model problems.

A lot of those systems already had a predefined if-this-then-that chain of logic buried in them, built up over a decade by people who had mostly left. Our job was to capture the complex relationships and patterns that logic was gesturing at. That was a win-win: the client got a solid model, and the team learned an enormous amount about the business and the data — which, it turns out, is the same thing.

We rarely hit hiccups on accuracy or performance. Most of the time an ensemble got us there. Only if we were still unsatisfied would we start exploring deep learning, and honestly, that conversation didn't come up often.

The takeaway I keep returning to: most teams need to solve data problems, not model problems. Model problems are hard, expensive, and require deep expertise. Data problems are also hard — but they're hard in a way your organisation is actually equipped to solve, because the knowledge required already exists inside the building. It's just sitting in someone's head instead of in a column.

There's good research on why teams avoid this. Sambasivan et al. studied it directly in a paper with the best title in the field: “Everyone wants to do the model work, not the data work.” They found 92% of practitioners hit at least one data cascade, and 45% hit two or more in a single project. Cascades are opaque, delayed, and largely avoidable — and they persist because of incentives. Nobody gets promoted for a well-specified label.

Booking.com documented the business end of this across roughly 150 models validated with randomised trials, and their headline finding should be printed on a mug: improving offline model performance does not reliably translate into business value. You can win the leaderboard and lose the quarter.

Where You Might Fail

I'd be selling you something if I stopped here. Boosting has real limits, and some of them aren't fixable with cleverness.

First, let me correct a thing I believed for years: it's not about row count. Gradient boosting handles millions of rows perfectly well. In fact, in recent head-to-head benchmarks, large numeric datasets are exactly where tuned XGBoost still ranks at or near the top. If anything, the regime under threat right now is small data, which we'll get to.

It's about shape.

• When the table is destroying the structure. Molecules are graphs. Fraud rings are graphs. Sessions are sequences with order and timing. Images are images. If you're flattening the object into 200 hand-rolled descriptors, you've already thrown away the thing that determines the answer. My caveat from earlier stands — for many 2D property-prediction endpoints, fingerprints plus a booster are still at the top. But for 3D geometry, conformers and docking, a graph network isn't a flex, it's the correct data structure.
• Extrapolation. A tree is piecewise-constant. Outside the range it trained on, it flatlines, confidently. Anything with a trend in it — prices, load, growth curves — will quietly disappoint you at precisely the moment it matters.
• Anything semantic. Free text, resumes, support tickets, audio, product photos. Boosting can split on a signal; it cannot interpret one. The production answer here isn't “switch to deep learning,” it's the hybrid that nobody writes blog posts about: use a network to turn the unstructured thing into an embedding, then feed that embedding into your gradient booster alongside the tabular features. Most strong systems I've built look exactly like this.
• Learned interactions at scale. Recommenders, high-cardinality behavioural data, anything where the interaction surface is combinatorial. You can make trees work with enough feature engineering. Past a certain point, though, the feature engineering is a neural network — assembled by hand, at greater cost, worse.

If you find yourself at this crossroads, that's fine. Either you started simple and genuinely outgrew it, or you're in the niche that needed the complex thing from day one. Either way you have a baseline, which means you can prove the complex thing is worth it instead of assuming.

The best account of what that transition actually costs is Airbnb's “Applying Deep Learning to Airbnb Search”. Read the failures. Their first production neural network was a single hidden layer with 32 ReLUs, and it came out neutral against the GBDT it replaced. Listing-ID embeddings failed outright, because a listing can be booked at most 365 times a year and there was never enough interaction data to learn them. They quote Karpathy's “don't be a hero,” which is a very expensive lesson to arrive at via a KDD paper. The sequel a year later is the one people skip: they'd lost the ability to reason about how the model used price, and had to structurally remove price from the network to get it back.

Neural networks learn complex relationships and nuanced patterns between features in a way trees genuinely can't. They handle large and diverse data well. They're the surgeon. They also come with compute, training time, inference latency, hosting, and a permanent tax on explainability. That's the price of the complexity, and sometimes it's absolutely worth paying.

The counterargument I can't dismiss

In fairness, there's a version of this post that's wrong, and it's worth stating properly.

Rich Sutton's “The Bitter Lesson” argues that across sixty years of AI, methods that leverage general learning and raw computation have consistently beaten methods that encode human domain knowledge — and that the human-knowledge approaches feel better right up until the moment they lose. My entire argument here is “encode human domain knowledge into features.” That is exactly the pattern Sutton says loses.

My honest read is that his lesson holds hardest where you can generate or collect effectively unlimited data for the task, and holds much more weakly where the binding constraint is that your label definition is wrong. No amount of computation fixes a target column that's measuring the wrong thing. But I'd be lying if I said I found that fully reassuring, and the next section is why.

The Future

The interesting shift isn't that the models got better. It's that the number of people who can build one went up by about two orders of magnitude, and the minimum viable dataset came down by three. Those two things together open up a category of problem that has been sitting untouched for twenty years because it was never worth staffing a data science team for.

Tabular foundation models. Start with the part that moves the ground under everything else.

TabPFN landed in Nature in early 2025 with an odd proposition: pretrain a transformer on enormous quantities of synthetic tables generated from structural causal models, and it learns a prior over how tables tend to behave. Supervised learning becomes in-context learning. No fitting — the forward pass does the reasoning.

That line kept moving through TabPFN-2.5 and TabICLv2, and on 30 June 2026 Google shipped TabFM: hybrid row/column attention, hundreds of millions of synthetic training datasets, zero-shot classification and regression in a single forward pass, reported to outperform heavily tuned tree ensembles on the TabArena suite. It's going into BigQuery behind a SQL AI.PREDICT call, which means your analyst may well ship a model before your platform team finishes the Terraform.

The caveats are real and worth stating plainly. An independent reproduction found genuine zero-shot wins over Optuna-tuned XGBoost on small-to-mid tables — and then demoted a couple of those wins to ties after checking across random seeds. The released weights carry a non-commercial licence. Classification caps at ten classes. Inference economics are nowhere near a tree ensemble on CPU. If you're serving fifty thousand predictions a second, this isn't your architecture yet.

But notice which regime they're winning: small data. Five thousand to fifty thousand rows.

The long tail finally becomes reachable

That regime is the whole story, and I don't think people have absorbed it yet.

For twenty years the unspoken entry fee for machine learning was tens of thousands of clean labelled rows. That fee quietly excluded almost every real business problem, because most of them are small. A 40-branch NBFC has maybe 6,000 loans that went bad. A hospital chain does 900 of a particular surgery a year. A factory has 200 recorded failures of the machine that matters. A specialty chemicals firm has 1,100 batches. A logistics company has 4,000 detention events.

Every one of those is a genuine, expensive, unsolved problem. Every one of them was below the waterline. That waterline just dropped, and nobody has gone looking in the newly exposed territory yet.

The mundane problems nobody has touched

Here's the part that I think is genuinely underrated. Walk into almost any mid-sized company and you'll find a dozen decisions being made by a spreadsheet, a rule from 2014, and a person's gut. Not exotic decisions. Boring, repetitive, expensive ones:

• Accounts payable exception routing. Which invoices need a human, which can straight-through process. Somebody is manually eyeballing 400 a day.
• Collections prioritisation. Which delinquent accounts to call first. Most NBFCs still work this by days-past-due, which is roughly the least informative feature available.
• No-shows. Clinics, salons, diagnostic labs, restaurants, service centres. A no-show rate of 18% is a business-model problem, and the data to predict it is sitting in the booking system.
• Returns likelihood at the point of purchase. E-commerce companies know this per-SKU. Almost none of them know it per-order-per-customer, which is where the money is.
• Warranty and claim fraud triage. Not catching fraud — just ranking what a human should look at first.
• Field service first-time-fix. Which technician, which part on the van. Every failed first visit is a truck roll you paid for twice.
• Dead stock and reorder points. Still done with a moving average and a safety factor somebody chose in a meeting.
• Shift absenteeism in warehouses and plants. Predictable a week out, and it wrecks the roster every time.
• Learner churn in ed-tech. Which student stops showing up in week three, which is worth an intervention. I've watched this one get solved with a spreadsheet and a lot of hope.
• Machine failure from process parameters. Temperature, pressure, vibration, cycle counts. Tabular. Boring. Worth crores.

None of these will get you a conference talk. Every single one has a rupee value that somebody in finance can compute in about four minutes. That asymmetry is the opportunity.

SQL becomes the interface

The AI.PREDICT-style shift matters more than the accuracy numbers. When the model is a SQL function, the person who ships it is the analyst who already writes the quarterly report — someone who understands the business, knows which column is unreliable, and remembers why that flag exists.

That's the correct person to be building these. It's just never been the achievable person before.

The obvious catch: bad models will now ship faster than ever, too. Somebody still needs to know what a leaky feature looks like and why a random split flatters you. The bottleneck moves from modelling to governance, which is a better problem to have but is still a problem, and I'd rather say so now than write the follow-up post in two years.

Tribal knowledge gets a front door

The hardest step in that 15-step list isn't the model. It's getting what the domain expert knows into a column. The guy who knows that orders from the Hosur warehouse on Fridays are always entered wrong. The underwriter who can tell you which self-declared income figures to distrust and why.

LLMs are turning out to be a decent front door for this. CAAFE — context-aware automated feature engineering — takes a dataset plus a plain-language description and proposes semantically meaningful features, verifying each against cross-validation. It improved 11 of 14 datasets in the original paper, lifting mean ROC AUC from 0.798 to 0.822. That's roughly the gain you'd get from swapping logistic regression for a random forest, obtained by reading the schema and thinking about it.

Note the awkward implication, which I flagged earlier: this is the same research group automating both the tuning and the feature engineering. My honest position is that an LLM can propose good features from what's in the schema, and cannot invent the column that was never collected. That gap is real but it's narrower than I'd like.

Specialist models as tools for agents

Imagine building simple, specialist tools that an agent can call. An LLM that reaches for a purpose-built model instead of reasoning its way to a number is a much better system than an LLM alone — the model gives you a calibrated, deterministic, auditable answer, and the LLM handles everything around it.

Concretely: a support agent that calls a churn model before deciding what retention offer to make. An ops copilot that queries a delay model before promising a delivery window. An underwriting assistant that pulls a risk score rather than free-associating about it.

The boosting model becomes a tool in a toolbox rather than a product in itself, which is honestly where most of them belonged anyway. And it fixes the thing that makes agents unusable in serious contexts — that they're confidently unquantified. A model that outputs 0.31 with a calibration curve behind it can be reasoned about. A paragraph cannot.

On-edge and offline

With a Raspberry Pi and a few sensors you can deploy real solutions in agriculture, healthcare, energy and water. This is where boosting's efficiency stops being a convenience and becomes the whole reason the thing is possible. There's no GPU in a field in Karnataka, and there's frequently no connectivity either.

Irrigation scheduling from soil moisture and weather. Cold chain excursion prediction for vaccines and dairy. Transformer load and distribution loss detection for discoms. Water pump failure. Grain moisture and spoilage risk. Every one of these is a small tabular model that has to run offline on cheap hardware, forever, without a maintenance contract — which rules out essentially everything except a tree ensemble.

The regulated corner where boosting wins outright

This one is a growing moat, not a shrinking one.

In lending, insurance, hiring and clinical triage, you need monotonic constraints, stable feature attributions, reproducibility across runs, and an audit trail a regulator will accept. Gradient boosting gives you all four almost for free. You cannot put a model with a non-commercial licence, a ten-class cap, and no stable explanation story into an underwriting pipeline — not because it's worse, but because you can't defend it in a room with a compliance officer in it.

As AI regulation tightens, the explainable, deterministic, locally-run model gets more valuable, not less. That's the opposite of the direction everyone assumes things are moving.

The India-shaped version of all of this

I'll say the obvious local thing, because I think it's the largest version of the opportunity and it's underwritten.

MSME lending on GST and UPI transaction history. Kirana-level demand forecasting on ONDC data. Crop advisory and yield prediction. Discom loss and theft detection. Claims triage for the newly insured. Dropout risk in state education systems. Health worker triage prioritisation at the last mile.

All of it tabular. All of it small-to-medium data. All of it messy, high-variance, and worth an enormous amount. None of it needs a frontier model — it needs someone who will sit with the data for a week and knows what a good split looks like.

Conclusion

Here's the thing I find genuinely funny about the timing.

If a frozen model that has never heard of your company, trained entirely on synthetic tables, can match six weeks of your careful hyperparameter tuning — then the tuning was never your edge. It was always the commodity. The foundation models aren't refuting the argument in this post; they're automating the exact half of the job I've been telling people to stop hoarding.

What they can't automate is the rest of it. The label definition you argued about for two days. The leakage you caught in week one because a column was only populated after the outcome. The feature that encoded the actual physical mechanism causing the delay. The decision to split by time and entity, which is the difference between a model and a story about a model. The conversation with the operations lead who mentions, offhand, the thing that explains your entire residual.

There is no pretrained prior over your business being wrong about what it's measuring.

So: start boring. Get a baseline up in the first hour. Spend the week on the data, because that's where the ceiling is. Delegate the tuning to Optuna and go do something with a higher return. Add a tabular foundation model to your bake-off, because it's cheap and it's an honest check on whether your effort bought anything a stranger's prior couldn't. And when your data genuinely has structure a table is destroying — you'll know, and you'll be able to justify the complicated thing in one sentence instead of a slide deck.

Fifteen years on, the most useful thing in my toolkit is still a gradient-boosted tree and a week spent understanding the data. I don't think that's nostalgia. I think most of us have been solving the wrong problem, very impressively.

References

The benchmark literature

• Grinsztajn, Oyallon & Varoquaux (2022) — Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS D&B. arXiv:2207.08815
• McElfresh et al. (2023) — When Do Neural Nets Outperform Boosted Trees on Tabular Data? NeurIPS. arXiv:2305.02997
• Shwartz-Ziv & Armon (2022) — Tabular Data: Deep Learning is Not All You Need. Information Fusion 81. ScienceDirect
• Borisov et al. (2024) — Deep Neural Networks and Tabular Data: A Survey. IEEE TNNLS. arXiv:2110.01889
• Holzmüller, Grinsztajn & Steinwart (2024) — Better by Default: Strong Pre-tuned MLPs and Boosted Trees on Tabular Data. NeurIPS. arXiv:2407.04491

The algorithms

• Friedman (2001) — Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics. Project Euclid
• Chen & Guestrin (2016) — XGBoost: A Scalable Tree Boosting System. KDD. arXiv:1603.02754
• Ke et al. (2017) — LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS. arXiv:1706.02940
• Prokhorenkova et al. (2018) — CatBoost: unbiased boosting with categorical features. NeurIPS. arXiv:1706.09516

Data quality, leakage, and why this all matters

• Sambasivan et al. (2021) — “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. CHI. Google Research
• Kapoor & Narayanan (2023) — Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4(9). ScienceDirect
• Sculley et al. (2015) — Hidden Technical Debt in Machine Learning Systems. NeurIPS. Paper
• Zinkevich — Rules of Machine Learning: Best Practices for ML Engineering. Google Developers

Industry case studies

• Bernardi, Mavridis & Estevez (2019) — 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com. KDD. ACM
• Haldar et al. (2019) — Applying Deep Learning to Airbnb Search. KDD. arXiv:1810.09591
• Haldar et al. (2020) — Improving Deep Learning for Airbnb Search. KDD. arXiv:2002.05515
• Makridakis, Spiliotis & Assimakopoulos — The M5 Accuracy Competition: Results, findings and conclusions. International Journal of Forecasting. (LightGBM was the method of choice among winners.) DOI

Tuning

• Bergstra & Bengio (2012) — Random Search for Hyper-Parameter Optimization. JMLR 13. JMLR
• Akiba et al. (2019) — Optuna: A Next-generation Hyperparameter Optimization Framework. KDD. arXiv:1907.10902
• Erickson et al. (2020) — AutoGluon-Tabular. arXiv:2003.06505

Tabular foundation models

• Hollmann et al. (2025) — Accurate predictions on small data with a tabular foundation model. Nature 637:319–326
• Qu et al. (2025) — TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. arXiv:2502.05564
• Erickson et al. (2025) — TabArena: A Living Benchmark for Machine Learning on Tabular Data. arXiv:2506.16791
• Google Research (2026) — Introducing TabFM. Blog
• Pandey (2026) — independent TabFM reproduction. Writeup

Molecular ML

• Benchmarking Pretrained Molecular Embedding Models for Molecular Representation Learning (2025). arXiv:2508.06199
• Gilmer et al. (2017) — Neural Message Passing for Quantum Chemistry. ICML. arXiv:1704.01212
• García-Ortegón et al. (2021) — DOCKSTRING. arXiv:2110.15486

The counterargument

• Sutton (2019) — The Bitter Lesson. incompleteideas.net
• Hollmann et al. (2023) — CAAFE: Context-Aware Automated Feature Engineering. NeurIPS. arXiv:2305.03403

On this site

• The Tree, Not the Titan — when the same tree instinct applies to agent routing, and where it breaks.
• The Nutritionist in the Machine — constrained optimisation over a recommender, not an LLM owning the safety path.