Choosing the Right Machine Learning Approach for Production Systems

We get the same opening question on nearly every AI engagement: 'should we use an LLM for this?' It's the wrong first question. The right first question is what the label actually costs to get wrong, how much labeled data exists today, and how many milliseconds of latency the product can tolerate. The model family falls out of those answers — it shouldn't drive them.

Three Questions Before Any Model Gets Chosen

First: how expensive is a wrong prediction? Flagging a support ticket wrong costs a few minutes; approving a fraudulent transaction costs real money. Second: how much labeled data do you actually have, not how much you could theoretically collect? Third: what's the latency and cost budget per inference, at your real traffic volume, not a demo's traffic volume. A gradient-boosted tree trained on tabular data will beat a fine-tuned transformer on most structured business problems — it's faster, cheaper, more interpretable, and just as accurate when the signal is genuinely tabular.

When Deep Learning Actually Earns Its Complexity

Deep learning wins decisively on unstructured input — images, audio, free text where the feature engineering problem is harder than the modeling problem. Even there, we start with a pretrained model and fine-tune narrowly rather than training from scratch; the marginal accuracy gain from a from-scratch model rarely justifies the training infrastructure it demands.

“The most expensive model in your stack should be the one solving the problem nothing simpler can solve — not the default.”

Where LLMs Fit Without Becoming a Liability

Large language models earn their place when the task is genuinely open-ended language understanding or generation — summarization, classification with fuzzy category boundaries, drafting. We wrap every production LLM call with a validation layer and a fallback path, because the failure mode isn't a crash, it's a confident, well-formatted wrong answer that's much harder to catch downstream.

def choose_model_family(problem):
    if problem.input_type == "tabular" and problem.has_labels:
        return "gradient_boosted_trees"

    if problem.input_type in ("image", "audio") and problem.has_labels:
        return "fine_tuned_pretrained_model"

    if problem.is_open_ended_language_task:
        return "llm_with_validation_layer"

    # Default to the simplest thing that could possibly work.
    return "rules_engine_or_logistic_regression"