Nobody files a ticket asking for a transformer. They ask you to transcribe a pile of support calls, put a chat box on the internal docs, turn product copy into a thumbnail, or caption ten thousand images. Every one of those is a different kind of model, a different library, and a very different GPU bill. The Hub has well over a million models. The skill is not knowing all of them. It is knowing the dozen families that cover almost every request, and picking a size that fits the hardware you already have.
How many kinds of model are there?
Fewer than the million-model count suggests. Almost every request maps to one of eight buckets, split by what goes in and what comes out. Get the bucket right and you have narrowed a million models to a handful of families.
Pick by task, not by model name
The fastest way to the wrong answer is to start from a model everyone is talking about and look for a use. Start from the request instead. What is the input, what is the output, and does it need to run offline or commercially? Those three questions land you on a family before you ever open a model page.
The model map
One table to keep near your desk. Families are examples, not the only option, and sizes are the ranges you will actually meet on the Hub. Treat the hardware column as a starting point, not a guarantee; quantization (see Part 13) moves these numbers down.
Text and retrieval: the two you will use most
Most internal requests are text. For chat, summarization, and the generation half of a RAG system, the open workhorses are Llama, Qwen, and Mistral. Qwen and Mistral ship under Apache-2.0, so commercial use is clean; Llama is gated and carries its own community license you must accept. For a first deployment, an 8B-class model at 4-bit on a single 24GB card covers a surprising amount of ground. Reach for 70B and up only when a smaller model visibly fails your quality bar, because the hardware cost is not linear.
The other half of RAG is retrieval, and that is embeddings, not an LLM. A small embedding model like BGE or all-MiniLM turns text into vectors for search; it runs on CPU or a tiny GPU and costs almost nothing to host. People routinely reach for a giant LLM when the actual job, find the relevant passages, belongs to a 100MB embedding model. For the concept behind this split, see the GenAI series; for running the model itself, Part 4.
Speech: transcription is easy, the voice side has license traps
Speech to text is the friendliest corner of the Hub. Whisper is MIT-licensed, runs from a tiny 39M checkpoint up to large-v3, and the distilled variants cut latency hard. One pipeline call and you are transcribing.
pip install transformers torch
from transformers import pipeline
asr = pipeline('automatic-speech-recognition', model='openai/whisper-large-v3')
print(asr('call.mp3')['text'])
# -> the transcribed text
# Fails with a CUDA out-of-memory error? Drop to model='openai/whisper-small'.The voice side, text to speech, is where licenses bite. The quality leaders for cloning, XTTS-v2 and F5-TTS, are non-commercial (CPML and CC-BY-NC). If you ship them in a product, that is a compliance problem, not a footnote. For commercial English at scale, Kokoro is tiny (around 82M) and Apache-2.0; Parler-TTS, from Hugging Face itself, is also Apache-2.0. Decide the license before anyone falls for a demo voice.
Vision: understanding versus generation
Two very different jobs share the word vision. Understanding an image, captioning, OCR, document question answering, is a vision-language model. Qwen-VL and InternVL lead the open field, run from small 2B sizes up to large MoE variants, and most ship under Apache-2.0, which keeps commercial use simple. A 7B VLM at 4-bit fits a single 24GB card and handles bulk captioning offline.
Generating an image is a diffusion model through the diffusers library, a different stack entirely. SDXL is the broad, well-supported default. SD 3.5 improves quality and is free under a revenue threshold. FLUX.1 splits cleanly: the schnell variant is Apache-2.0 and fine to ship, the dev variant is non-commercial and gated. For an infra reader the takeaway is that an image model is its own deployment, not an add-on to your LLM service.
Video: the most expensive box on the map
Video generation is the fastest-moving and hungriest corner. As of 2026 the open options through diffusers include Wan, HunyuanVideo, LTX, CogVideoX, and Mochi. Wan and Mochi are Apache-2.0; HunyuanVideo is commercial up to 100 million monthly users, then needs a separate license; LTX requires a commercial agreement with Lightricks for commercial use. Whatever you pick, expect high VRAM and often multiple GPUs, and expect the leader to change within a quarter. If video is a real requirement, validate the current model the week you build, not from a list like this one.
Same task, three sizes
Within a modality, size is the lever that sets your bill. For text, the three rungs look like this.
Licenses at a glance
Code, and models built for one job
If the task is writing or completing code, the general LLMs already handle a lot, but code-specialized models live on the Hub and often punch above their size for that one job. Qwen and others ship dedicated code variants, and they run on the same transformers and TGI path as any text model, so nothing changes for you operationally. The rule of thumb: pick a code-tuned model when the workload is mostly code, and a general model when it is mixed, because a code model can be weaker at ordinary prose. The same logic repeats across the map, a model tuned for one narrow job usually beats a generalist at that job and loses everywhere else, which is exactly why the task-first approach pays off.
When one model covers several jobs
Splitting by task is the default, not a law. A capable vision-language model reads an image and answers questions about it, so a single VLM can cover captioning, OCR, and document Q&A at once rather than three services. A modern LLM handles chat, summarization, and extraction from one deployment. The trade is real: a multi-purpose model is bigger and pricier to run than a narrow one, and it ties several workloads to one upgrade cycle and one license. Consolidate when the jobs are close and the volume is modest; keep them separate when one job is heavy enough to deserve its own sizing and its own scaling. That call is yours to make, because you own the capacity.
Trust your data over the leaderboard
Public leaderboards are a starting filter, not a verdict. They rank on tasks that may not match yours, and the top of an open leaderboard moves month to month, especially in image, audio, and video. Shortlist two or three candidates from a table like the one above, then run them against a sample of your real inputs at your real batch size. The model that wins on your data, at your latency, under a license you can ship, is the one that matters, not the one leading a benchmark the week you happened to look. Write the result down so the next person inherits a decision, not a search.
How to choose, in order
Why pick task-first: it collapses a million models to a family in one question and stops you sizing hardware for a model you do not need. When not to: if one team needs several jobs at once, a single capable VLM can sometimes cover image plus text and save you running two services, so do not force a split that the workload does not want. What to validate first, every time: that the license fits your use, that the model fits your VRAM with headroom at your real batch size, and that any gated model has an access token wired into your pull pipeline before launch day.
Worked example
Request: caption 50,000 product images, runs internally, must be commercial-safe. Map it: image in, text out, so a vision-language model. Pick a 7B Qwen-VL at 4-bit; it fits a single 24GB card, runs offline, and is Apache-2.0, so no license review. Provision one GPU, batch the images, and if throughput is short, scale out with a second replica the way Part 12 handles serving. No 70B model, no multi-GPU node, no legal ticket.
Where I would start
Pick the smallest model that clears your quality bar, confirm the license allows your use, and size the GPU to that model rather than the other way round. Keep a one-page approved list per modality so the next request is a lookup, not a research project. Start from the table above, open the two or three model pages it points to, and read the license and the model card (that habit is Part 2) before anything downloads.
References
- State of open video generation models in Diffusers
- Text-to-speech models on the Hub
- Stable Diffusion 3.5 Large model card
- FLUX.1 schnell model card
About The Author
DrJha