LLMs vs Machine Learning and Deep Learning for Structured Data

Large language models (LLMs) have made extraordinary progress in recent years. They write code, reason through complex problems, summarise documents, and handle tasks that would have previously required significant specialist effort. It is no surprise that teams are now considering whether they can be applied to the kinds of structured data problems that have traditionally been the domain of machine learning and deep learning.

It is a legitimate question, and an important one. LLMs are increasingly capable, and the boundary between what they can and cannot do is shifting at pace. But capability in general does not always translate to the right tool for a specific problem. The best way to answer the question is to run the experiments.

That is what we did.

We applied the same data, the same features and the same train/test approach. Only the method changed.

We used GPT-5.4 on the LLM side. Our goal was to understand where each approach performs well, what it costs, and what the practical trade-offs are for teams choosing between them.

The scaffolding problem no one talks about

Both approaches required proper data understanding and preparation. Regardless of the method you choose, you still need to ensure feature engineering, cleaning, handling imbalanced classes, selecting the right split strategy.

LLMs do introduce one additional practical consideration. Context windows have hard limits, so data must be split into chunks, processed in batches, then reassembled. For some tasks like classification, chunking is a reasonable approximation. For others like regression and clustering, it is a fundamental constraint: the model never sees the full data distribution at once, which is precisely what those tasks require.

This scaffolding must be built separately for every new task. It works around the context window limit rather than removing it, and none of it carries over to the next job. ML and DL skip this extra layer entirely.

We designed and built things in a way to give the LLM the best possible chance.

It is worth noting that these costs reflect an R&D setup, not a production-optimised architecture. Fine-tuning the LLM on domain-specific data would reduce inference costs by removing the need to send the full training context each time, but it introduces its own overhead: substantial training data, additional engineering effort, non-trivial fine-tuning costs, and a model that requires careful quality assessment before it can be trusted in production.

What we found

For those of you who like to understand the specifics, we’ve included some detailed numbers. For the majority the cost and speed advantage will give you the key info.

Each section that follows summarises what we observed.

Why this keeps happening, and why it gets worse at scale

Better prompting narrows the gap in places, but it does not close it. The difference is structural, not a matter of tuning the instructions.

Every test here ran on roughly 100,000 records. That was a deliberate choice: context window limits mean an LLM must operate on chunks rather than the full dataset, and we needed conditions where it could function at all. These are controlled experiments, not production datasets. Most real business systems operate at far greater scale.

At 100,000 records, the cost differences are noticeable but not alarming. What changes at scale is the economics. A LLM pays per token on every single inference call. A trained model pays its computational cost once, during training, and then runs at near-zero marginal cost on every subsequent prediction. The traffic classification dataset used in these tests contained 203 million records in full. Running the LLM approach on the complete dataset at the rates observed here would make the cost entirely unworkable.

This is one of the first constraints encountered when moving from PoC to live system; so it’s not something you want to “put off” for the future.

Different datasets would move the exact figures, but the direction holds, because the gaps are structural, not artefacts of this data. An LLM pays per token and infers over chunks, so it stays more expensive and slower, and it never sees the full distribution a purpose-built model trains on. Different data changes the size of the result, not its shape.

A practical guide to choosing

The experiments above point to a clear decision framework for teams working on similar problems.

If your input is unstructured text, if you have no labelled training data, if the task involves language understanding, document reasoning, or zero-shot inference, an LLM is likely the right starting point. These are the conditions where LLMs have a genuine structural advantage.

If your data is structured and tabular, if the task is regression, classification, anomaly detection, clustering, or time-series forecasting, and if you have training data available, ML and DL approaches will deliver lower inference cost and faster predictions, and most likely better output quality. The experiments above demonstrate this consistently across all five task types.

Once again, the expertise is not in knowing how to wire up the latest model. It is in understanding which tool fits the problem you actually have.

Next steps: putting the right model into practice

Choosing the right model is only the start. The real value comes from assessing the use case, selecting the approach that best fits the data and goals, and turning it into a production-ready solution that delivers value at the right cost.

If this is something that you would like to discuss this further, please get in touch.