TL;DR
The datasets library is two things at once: a download client that pulls files off the Hub, and an Arrow-backed cache that memory-maps them off disk. Treat both as storage and data-movement problems, not data-science ones.
Default download lands in ~/.cache/huggingface/datasets. It does not clean up after itself. On a shared box or a CI runner, that is the line item that fills the volume at 2am.
Use streaming=True when the dataset is bigger than your disk or you only need one pass. Use the on-disk Arrow cache when you need random access and repeat reads. Know which one you picked, because they fail in completely different ways.
load_dataset() writes to your filesystem, how big it gets, and how to keep it bounded. Prerequisites: comfort with a shell, environment variables, and the idea of a local cache. Python helps but the storage story is the point.A junior engineer pings you: the training node is out of disk and the job died. You SSH in, run du -sh ~/.cache/huggingface, and it comes back 340 GB. Nobody downloaded 340 GB on purpose. They ran one line of Python that said load_dataset('...') a few times against a big corpus, each call quietly writing a raw copy plus an Arrow copy, and none of it ever got cleaned up. This is not a data-science incident. It is a storage incident with a Python front end, and it is the most common way the datasets library bites an infrastructure team.
The library is genuinely good, and most of what trips people up is invisible until you look at the filesystem. So let us look at the filesystem. Part 5 covered tokenizers and token budgets, which is what happens to text once it is in memory. This part is about everything that has to happen on disk before that point.
Think of it as a data pipeline, not a function call
One line of code hides four distinct stages, and each one has a different storage and network cost. If you have ever reasoned about a container registry pull (layers come down, get unpacked, get mounted), this will feel familiar. The Hub is the registry, the dataset repo is the artifact, and the Arrow cache is the unpacked, ready-to-mount form.
The stages: the library resolves the repo on the Hub, downloads the raw files into a download cache, converts them into Apache Arrow tables in a separate cache, then hands your code a Dataset object that reads those Arrow files by memory-mapping them. The conversion is the part people forget. For a while you are holding both the raw copy and the Arrow copy, so peak disk is roughly double the dataset size. Plan capacity for the peak, not the final state.
Loading a dataset, and where the bytes go
Here is the minimum that works, with the output you should expect and the way it fails.
pip install datasets
# python
from datasets import load_dataset
ds = load_dataset('rajpurkar/squad', split='train')
print(ds)
print(ds[0]['question'])Expected output:
Dataset({
features: ['id', 'title', 'context', 'question', 'answers'],
num_rows: 87599
})
To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?The failure mode you will hit is not a Python error, it is the disk. The download succeeds, the Arrow conversion starts, and the volume fills mid-conversion. You get a write error deep in a worker, the cache is left half-built, and the next run may refuse to reuse it. The fix is boring and infra-shaped: point the cache at a volume with room, and watch it. By default the library reuses an existing dataset if it is already cached, so a re-run is cheap; pass download_mode='force_redownload' only when you want to pull fresh files.
Three environment variables decide where everything lands. Set them before the first import, not after.
| Variable | Controls | Default |
|---|---|---|
HF_HOME | The root for everything: hub downloads and datasets cache | ~/.cache/huggingface |
HF_DATASETS_CACHE | Only the Arrow files and indices that datasets writes | under HF_HOME |
HF_HUB_CACHE | Files pulled from the Hub (models, and raw dataset downloads) | under HF_HOME |
The distinction matters because the raw downloads and the Arrow cache can live on different volumes. On a node with a small system disk and a large scratch mount, send HF_HOME to the scratch mount and you are done. Set it in the systemd unit, the container env, or the CI job definition, the same place you set every other path that must not default to a home directory.
Streaming: when you cannot, or should not, download it all
Some datasets are larger than any disk you want to provision. Web-scale text corpora run into terabytes. You do not download those. You stream them: pass streaming=True and the call returns an IterableDataset that pulls and decodes records on the fly as you iterate, with effectively no upfront download and a tiny disk footprint.
from datasets import load_dataset
stream = load_dataset(
'HuggingFaceFW/fineweb',
split='train',
streaming=True,
)
for i, row in enumerate(stream):
print(row['text'][:80])
if i >= 2:
breakThat iterates the first few records of a multi-terabyte corpus on a laptop. The trade is real: an IterableDataset has no random access. You cannot ask for row 5 million directly; you iterate from the front. Shuffling is approximate, done with a buffer (stream.shuffle(buffer_size=10000) samples from a rolling window, not the whole set). You skip and take with stream.skip(n) and stream.take(n), and you split work across workers with shard(). For a single training pass over data too big to store, this is exactly right. For anything that needs to revisit examples, it is the wrong tool.
Why a 200 GB dataset opens on a 16 GB box
The thing that surprises people coming from pandas: a downloaded Dataset does not load into RAM. The library stores data in Apache Arrow on disk and memory-maps it. The operating system pages in only the bytes you touch, using virtual memory the same way it does for any mmap-ed file. So you can open a dataset far larger than physical RAM and index into it, and resident memory barely moves. This is the same trick a database uses to query a table bigger than memory. If you have tuned page cache behavior on a busy server, you already understand the mechanism.
Worked example
Provisioning a node for a fine-tune on a 50 GB dataset. Naive math says 50 GB of disk. Real math: ~50 GB raw download, plus ~50 GB Arrow cache during conversion, so plan for roughly 100 GB peak on the cache volume. RAM for the dataset itself is near zero because it is memory-mapped, so a 32 GB box is fine for data loading; size RAM for the model and batch, not the corpus.
If a map() step rewrites the data (it does, see below), add another ~50 GB for the transformed copy. The team asked for a 64 GB volume. The job needed closer to 150 GB. That gap is the whole reason this section exists.
map(), filter(), and the silent cache explosion
This is the part that quietly fills disks. On a regular Dataset, map() and filter() are not lazy. They run the transform over the whole dataset and write a new Arrow cache file with the result, keyed by a fingerprint of the operation. That caching is a feature: re-run the script and the expensive tokenization step is skipped because the result is already on disk. It is also a liability: every distinct transform leaves another full-size copy behind, and they accumulate.
from datasets import load_dataset
ds = load_dataset('rajpurkar/squad', split='train')
def add_len(row):
row['q_len'] = len(row['question'])
return row
# writes a NEW arrow cache file with the result
ds2 = ds.map(add_len)
# re-run with the same code: loads from cache, no recompute
# pass load_from_cache_file=False to force recompute and skip the lookup
ds3 = ds.map(add_len, load_from_cache_file=False)map() cache grows without bound because nobody deletes it. Each tweak to a preprocessing function is a fresh full copy. Call ds.cleanup_cache_files() to drop the Arrow cache files for a dataset when you are done, or set datasets.config.IN_MEMORY_MAX_SIZE (or HF_DATASETS_IN_MEMORY_MAX_SIZE) so small datasets stay in RAM and never hit disk at all. On ephemeral CI runners this is moot; on anything persistent, it is a scheduled cleanup job.One more distinction worth keeping straight, because it changes how every operation behaves:
| Behavior | Dataset (downloaded) | IterableDataset (streamed) |
|---|---|---|
| Disk footprint | Full dataset, plus copies per transform | Near zero |
| Random access | Yes, by index | No, iterate only |
| map / filter | Eager, writes cache | Lazy, applied on iteration |
| Shuffle | Full, exact | Approximate, buffer-based |
| Best when | Fits on disk, reused, needs indexing | Too big to store, single pass |
Cleanup and quotas: the part nobody owns
The cache is built to never delete anything, on the assumption that disk is cheap and recompute is expensive. On a personal laptop that is correct. On shared infrastructure it is a slow leak. Two controls cover most of it:
# 1. send the whole cache to a sized scratch volume, per job
export HF_HOME=/mnt/scratch/hf
# 2. see what is there
du -sh /mnt/scratch/hf/*
# in python, drop a dataset's arrow cache when done
ds.cleanup_cache_files()For a registry-style approach to artifacts and who is allowed to pull them, the parallel on the model side is exactly the container-registry discipline covered in the Harbor for Beginners guide: scoped access, retention policies, and quotas. Datasets do not yet get the same governance attention as models, but the storage problem is identical, and the fix is the same muscle you already use for registry retention. For the conceptual side of why preprocessing and tokenization matter at all, the Generative AI guide covers the data pipeline as an idea; this part is that idea hitting your filesystem.
HF_HOME or clearing cache on a shared node affects every user and job pointed at that path. Confirm no active job is mid-run, and prefer per-job cache paths over a global wipe. Test cache-location changes on a non-production runner first. Recompute after a wipe is real wall-clock time on real GPUs; budget for it before you delete.What I would do in production
Default to downloading, not streaming, when the dataset fits on a volume you control and the team will run against it more than once. Why: random access, exact shuffling, and a cache that skips recompute on re-runs all save more than they cost, and you keep the bytes inside your network. When it is not the right call: a single throwaway pass over something terabyte-scale, or an egress-metered link where re-pulling every epoch hurts more than storing once. What to validate first: point HF_HOME at a sized scratch volume, run one full pass while watching du -sh, and confirm peak disk against the roughly-double rule before you commit a node spec. Streaming earns its place for the genuinely too-big-to-store case and for quick exploration, not as a default to dodge capacity planning.
The mental shift that makes all of this easy: the datasets library is storage infrastructure with a friendly Python API on top. You already know how to manage a cache, size a volume, set retention, and watch egress. Apply those exact instincts and the library stops surprising you. If you own GPU nodes, the next step is serving and on-prem deployment, which the Private AI guide and the NVIDIA AI guide pick up. Next in this series we move from data to customizing the model itself.
Your move: SSH into your busiest training node and run du -sh ~/.cache/huggingface/datasets right now. Whatever number comes back is your unmanaged dataset cache. Decide today whether it belongs on that volume.
References
Hugging Face Datasets: Stream
Hugging Face Datasets: Cache management
Hugging Face Datasets: Datasets and Arrow
About The Author
DrJha