Small Language Models — What They Are and When to Use Them

Every AI project I have seen in the last two years started the same way: someone suggested using ChatGPT or Claude, the team agreed, and nobody asked whether a much smaller, cheaper model would do the job just as well. For a lot of those tasks, it would have.

The default assumption — that bigger is better, that an LLM is the right starting point for any AI problem — is costing organisations real money and real complexity. Not because LLMs are bad. Because they are overkill for a surprising number of tasks.

Small language models are not a compromise. For the right job, they are the better engineering decision. The point is knowing which job is which.

🔗 Foundation posts

This post assumes you know what a language model is. If not, start with What is a Large Language Model (LLM)? — it covers how LLMs are built and trained. AI vs ML vs Deep Learning gives the broader context of where language models sit in the AI landscape.

What a small language model actually is

A language model’s size is measured in parameters — the numerical weights the model learned during training. More parameters means more capacity to learn patterns, hold general knowledge and handle a wider range of tasks. GPT-4 has hundreds of billions of parameters. Claude 3 Opus is in the same range. These are large language models.

Small language models typically sit in the 1 billion to 10 billion parameter range. They are not miniaturised versions of LLMs. They are purpose-built models — trained on focused datasets, optimised for efficiency, and designed to run on hardware that an LLM would never fit on.

The term is relative. A 7 billion parameter model is not small in any absolute sense — it still represents a massive amount of learned knowledge. It is small compared to the frontier models, and that relative smallness is exactly what makes it useful in certain contexts.

Category	Parameter range	Examples (as of 2026)	Runs on
Large Language Model	70B–1T+	GPT-4o, Claude 3 Opus, Gemini 1.5 Pro	GPU clusters / cloud API only
Mid-size model	10B–70B	Llama 3 70B, Mistral Large, Qwen 3.5 27B	Multiple GPUs or high-end cloud
Small Language Model	1B–10B	Phi-4-mini, Gemma 3 4B, Mistral Small 4, Llama 3.2 3B	Single GPU, laptop, edge device
Micro model	Under 1B	Gemma 3 1B, Llama 3.2 1B, Qwen 2.5 0.5B	Mobile, IoT, CPU-only

What SLMs give up — and what they don’t

Most posts about SLMs underplay the trade-offs. That is not useful. There are things SLMs genuinely cannot do as well as LLMs, and pretending otherwise leads to bad architecture decisions.

Where SLMs fall short:

• Deep multi-step reasoning — tasks that require holding complex chains of logic over long contexts. LLMs handle this better because they have more capacity to model long-range dependencies.
• Broad general knowledge — SLMs trained on focused datasets know their domain well and everything else less well. Ask a document-classification SLM to explain geopolitics and it will struggle.
• Open-ended creative tasks — long-form writing, complex narrative generation, sophisticated code generation across many languages. These benefit from the larger parameter space of an LLM.

What SLMs keep — and where they punch above their size:

• Speed — significantly lower inference latency. Responses in milliseconds rather than seconds. This matters enormously for real-time applications.
• Cost — orders of magnitude cheaper to run. At scale, this is the number that ends conversations about whether to use an SLM.
• Privacy — SLMs run on-premise or on-device. Your data never leaves your infrastructure. For regulated industries and sensitive enterprise data, this is not a nice-to-have.
• Fine-tuning potential — a smaller model is far easier and cheaper to fine-tune on domain-specific data. A fine-tuned 7B SLM regularly outperforms a general-purpose LLM on the narrow task it was trained for.

📌 The fine-tuning advantage

A fine-tuned SLM on your specific task and data will almost always beat a general LLM with a prompt. The LLM is generalising from broad training. The SLM has been taught your exact problem. This is the most underused capability in enterprise AI right now.

Where SLMs actually outperform LLMs

This is where the decision gets interesting. On a narrow, well-defined task with domain-specific training data, an SLM does not just match an LLM — it beats it.

The tasks where fine-tuned SLMs consistently outperform general LLMs:

Task	Why SLMs win	Typical SLM approach
Document classification	Narrow vocabulary, known categories — no general knowledge needed. Fine-tuned SLM learns the exact label space.	Fine-tune on labelled document corpus
Named entity recognition	Structured extraction from text. SLM learns entity patterns faster and runs at much lower latency.	Fine-tune on annotated examples
Intent detection	Classifying user intent into a fixed set of categories. SLM handles this at speed and low cost at scale.	Fine-tune on intent-labelled conversations
Structured data extraction	Pulling specific fields from documents — invoices, contracts, forms. Bounded output space suits SLMs.	Fine-tune or few-shot with strict output schema
Summarisation (domain-specific)	Summarising within a known domain — legal, medical, financial. Fine-tuned SLM uses domain terminology correctly.	Fine-tune on domain document pairs
On-device / edge AI	Any scenario where the model must run locally — mobile apps, IoT, air-gapped systems. Only SLMs are viable.	Quantised SLM deployed via Ollama or device SDK

The models worth knowing in 2026

You do not need to track every SLM release. These are the ones that matter for practical deployment decisions as of 2026.

Model	Provider	Size	Best for	Licence
Phi-4-mini	Microsoft	3.8B	Strong reasoning for its size. On-device and CPU-friendly. Good starting point for most tasks.	MIT
Gemma 3 4B	Google	4B	Best quality-to-size ratio in the range. Multimodal (text + image). Runs on 8GB RAM.	Gemma licence (commercial use permitted)
Mistral Small 4	Mistral AI	~6B active	Strong for coding and agentic tasks. Good choice for custom fine-tuning pipelines.	Apache 2.0
Llama 3.2 (1B / 3B)	Meta	1B or 3B	Mobile and edge deployment. Extremely lightweight. 1B runs on most smartphones.	Llama community licence
Qwen 2.5 (0.5B–7B)	Alibaba	0.5B–7B	Best multilingual support in the SLM range. Strong across 29+ languages.	Apache 2.0

💡 Practical Tip — start with Phi-4-mini

If you are evaluating SLMs for the first time and have no strong reason to choose otherwise, start with Phi-4-mini. It runs on modest hardware, has strong reasoning relative to its size, and is straightforward to deploy via Ollama locally or as a managed endpoint. Benchmark it against your specific task first — then decide if you need anything larger.

The decision framework — LLM or SLM?

The question is never ‘which is better?’ The question is ‘what does this specific task actually need?’. Five questions get you to the right answer almost every time.

Question	Points to LLM	Points to SLM
How narrow is the task?	Open-ended, varied, unpredictable inputs	Well-defined, bounded, repeatable
Does it need general knowledge?	Yes — broad world knowledge, reasoning across domains	No — domain-specific data, internal documents, structured inputs
Where does the data need to stay?	Cloud API is acceptable — data can leave your environment	Must stay on-premise, on-device or air-gapped
What is the query volume?	Low to medium — LLM API cost is manageable	High volume — per-query API cost adds up fast
What are the latency requirements?	Seconds of latency are acceptable	Sub-second response required — real-time UI or pipeline

✅ Best Practice

If three or more of those five questions point to SLM, start there. You can always move to a larger model if the SLM cannot handle the task. The reverse — discovering you are paying LLM costs for a problem a fine-tuned 3B model handles perfectly — is a much more expensive mistake.

Routing isn’t only an LLM-vs-SLM decision. How to Cut LLM Cost and Latency in Production covers routing by task complexity as one lever among several, alongside caching and batching, for teams that have already made this call and are managing the mix day to day.

In the SAP context

The same framework applies directly to SAP AI scenarios. The mistake I see on SAP BTP projects is routing everything through SAP Joule or a large frontier model via the Generative AI Hub, when several of the use cases are narrow enough for a fine-tuned SLM.

Where SLMs make practical sense on SAP BTP:

• Document classification and extraction — purchase orders, invoices, GRN documents. Bounded output, domain-specific vocabulary. A fine-tuned SLM deployed on AI Core handles this faster and cheaper than calling a frontier model.
• Intent detection for Fiori apps — categorising user input in conversational UI into a fixed set of actions. Classic SLM territory.
• Structured data extraction from unstructured text — pulling vendor names, amounts, dates from free-text fields. Repeatable, bounded, trainable.
• Private deployments — any scenario where SAP master data or financial data cannot leave your infrastructure. SAP AI Core supports custom model endpoints, and a self-hosted SLM is the only viable path for air-gapped SAP landscapes.

SAP Joule and the frontier models via AI Core remain the right choice when you need broad SAP process knowledge, cross-module reasoning, or complex multi-step guidance. The point is not to replace them — it is to reserve them for tasks that actually need them.

🔗 Related posts

Fine-Tuning vs Prompt Engineering vs RAG - covers when fine-tuning an SLM is the right call versus other customisation approaches.
AI in the Enterprise — A Practical Map - shows where SLMs fit in a broader enterprise AI architecture.

At a glance — small language models

Concept	One-line summary
Small Language Model (SLM)	A language model in the 1B–10B parameter range — purpose-built for efficiency, speed and narrow tasks
Parameter count	The measure of model size — more parameters means more capacity but more compute and cost
SLM trade-off	Gives up deep multi-step reasoning and broad general knowledge; gains speed, cost, privacy and fine-tuning ease
Fine-tuning advantage	A fine-tuned SLM on a narrow domain task regularly outperforms a general LLM — the SLM knows your specific problem
Where SLMs win	Classification, extraction, intent detection, summarisation in a known domain, and any on-device or air-gapped scenario
Key models (2026)	Phi-4-mini (3.8B), Gemma 3 4B, Mistral Small 4, Llama 3.2 1B/3B, Qwen 2.5 family
Decision trigger for SLM	Task is narrow and bounded, data must stay on-premise, query volume is high, or sub-second latency is required
SAP context	SAP AI Core supports custom SLM endpoints — right for document processing, intent detection and air-gapped scenarios

What to take away

The industry defaulted to large language models because they were the first thing that worked well enough to be useful. That made sense at the time. It makes less sense now that a well-chosen, fine-tuned SLM running on a single GPU can outperform a frontier model on the task you actually need it for — at a tenth of the cost and with your data never leaving your own infrastructure.

The real skill is not picking the best model. It is picking the right model. That means being specific about what the task actually requires: the input type, the output type, the data sensitivity, the volume, the latency tolerance. Most tasks, when you are specific about those things, do not need an LLM.

SLMs are not the future of AI. They are part of how mature AI engineering works today — where you deploy the smallest model that solves the problem reliably, and you save the larger models for the problems that genuinely need them.

🔗 Related posts on this site

What is a Large Language Model (LLM)? — how LLMs are trained and what makes them large; the baseline for understanding what SLMs are optimising against.
Fine-Tuning vs Prompt Engineering vs RAG — fine-tuning is where SLMs become most powerful; this post covers when and how to use it.
AI in the Enterprise — A Practical Map — where SLMs, LLMs and hybrid architectures fit in a real enterprise AI deployment.
How Generative AI Works — Tokens, Embeddings and the Transformer — the transformer architecture that underpins both SLMs and LLMs — understanding it explains why size matters the way it does.
How to Cut LLM Cost and Latency in Production — once you’ve routed to a smaller model, this covers the caching and batching levers that apply on top.

Published on rakeshnarayan.com — Articles

https://rakeshnarayan.com/articles/small-language-models-what-they-are-and-when-to-use-them/