Five recommendations came back for a user who had never clicked anything, and every score was 0. Not an exception, not a warning, just item ids 4, 3, 2, 1 and 0 in that order, which is what you get when a zero vector is multiplied against every item factor and the tie is broken by index. That output shipped to production once in a project I worked on, and for eleven days roughly nine percent of sessions were served the five lowest numbered items in the catalogue. Nobody caught it, because a recommender that returns something always looks healthier than one that raises.

Recommendation as a ranking problem

Every model so far in this series has predicted a value for a row. A churn classifier takes one customer and returns one probability. Recommenders do something structurally different: given one user, they order the entire catalogue and hand back the top few. Nothing about that is captured by accuracy, and only loosely by AUC, because a user never sees position 400 and does not care how well it was scored.

Two metrics carry most of the load. Recall at K asks a simple question: of the items a user actually went on to consume, how many appeared in the top K we showed. Normalised discounted cumulative gain, usually written NDCG, asks the same thing but pays attention to position, discounting a correct item at rank 9 relative to the same item at rank 1. Both need a cutoff, and that cutoff should match the real surface. Ten is conventional. If your carousel shows four tiles, measure at four.

A third number belongs next to them, and it gets left out constantly. Catalogue coverage counts how much of your inventory ever reaches anybody. A recommender that shows the same forty items to three thousand users can post a perfectly respectable recall score while doing nothing a merchandising team would call recommendation. Report coverage alongside accuracy or you will optimise your way into a shop window with four products in it.

One more distinction shapes everything downstream. Explicit feedback means a user told you a rating, a five star review or a thumbs up. Implicit feedback means you inferred preference from behaviour: a play, a click, a purchase. Implicit data is far more common and far more awkward, because absence is ambiguous. A film you never watched might be one you would love or one you would hate, and the data cannot tell you which. Hu, Koren and Volinsky worked that ambiguity into the loss function directly, treating every unobserved pair as a weak negative and every observed one as a positive with confidence proportional to interaction strength. That framing underpins the ALS implementation used below.

Two shapes of collaborative filtering

Collaborative filtering means recommending from behaviour patterns across users rather than from item attributes. It comes in two families, and they fail in different places, which is why I usually run both.

Neighbourhood methods compute similarity directly. Item to item cosine similarity, the older and simpler shape, asks which items get consumed by overlapping sets of users, then recommends neighbours of whatever a user already touched. Amazon built its early engine on exactly this, and it survives because it is cheap to update, trivially explainable, and degrades gracefully. Add one interaction and you can serve a new recommendation immediately, with no retraining.

Matrix factorisation takes a different route. It decomposes the sparse user by item matrix into two dense low rank matrices, one row of latent factors per user and one per item, such that their dot product approximates the observed interactions. Alternating least squares, ALS, fits it by holding item factors fixed and solving for user factors, then reversing, repeating until it settles. Each least squares step has a closed form, which is why ALS parallelises well and why it remains the default for implicit feedback at scale.

Leave one out evaluation for implicit feedback

Where the project stands: through Part 19 the Telco churn model had been trained, tuned, rebuilt as a network and finally backtested across forecast horizons. Churn has no user by item interaction matrix in it, so this part steps sideways to a second dataset and comes back to churn at the end. MovieLens is the standard public choice here, and the small release carries 100,836 ratings from 610 users across 9,742 movies, available from GroupLens. Rather than make every number in this part depend on a download, I generate a synthetic implicit feedback matrix with a fixed seed, so you can reproduce each figure exactly by running the code as printed. Swapping in MovieLens is two lines, shown at the end of this section.

# tested against: Python 3.10.12, implicit 0.7.3, scipy 1.15.3, numpy 2.2.6
import numpy as np, scipy.sparse as sp

rng = np.random.default_rng(7)
N_USERS, N_ITEMS, N_TOPICS = 3000, 800, 6

item_topic = rng.integers(0, N_TOPICS, N_ITEMS)
pop = rng.pareto(1.1, N_ITEMS) + 1.0
pop = pop / pop.sum()

user_topic = rng.integers(0, N_TOPICS, N_USERS)
n_hist = np.clip(rng.geometric(0.06, N_USERS), 1, 200)

rows, cols = [], []
for u in range(N_USERS):
    aff = np.where(item_topic == user_topic[u], 6.0, 1.0) * pop
    aff = aff / aff.sum()
    k = min(n_hist[u], N_ITEMS)
    picks = rng.choice(N_ITEMS, size=k, replace=False, p=aff)
    rows.extend([u] * k); cols.extend(picks.tolist())

R = sp.csr_matrix((np.ones(len(rows), dtype=np.float32), (rows, cols)),
                  shape=(N_USERS, N_ITEMS))
print('interactions %d  density %.4f' % (R.nnz, R.nnz / (N_USERS * N_ITEMS)))
hist = np.diff(R.indptr)
print('median history %d  users with 1 item %d  users with 20 or more %d' %
      (np.median(hist), (hist == 1).sum(), (hist >= 20).sum()))

interactions 49711  density 0.0207
median history 11  users with 1 item 186  users with 20 or more 886

Splitting comes next, and random row splitting is wrong here for the same structural reason it was wrong in Part 19. Interactions are not independent rows; they belong to users. Hold out random cells and a user can appear in both train and test, which measures interpolation. Leave one out per user is the standard fix: for every user with at least two interactions, remove exactly one item, train on everything that remains, then check whether the removed item comes back in the top K. Where you have timestamps, remove the most recent item instead of a random one, which is closer to what deployment asks.

train = R.tolil(copy=True)
held = np.full(N_USERS, -1)
for u in range(N_USERS):
    idx = R.indices[R.indptr[u]:R.indptr[u + 1]]
    if len(idx) >= 2:
        h = int(rng.choice(idx)); held[u] = h; train[u, h] = 0
train = train.tocsr(); train.eliminate_zeros()
evalu = np.where(held >= 0)[0]
print('eval users %d' % len(evalu))          # eval users 2814

K = 10
def score(rank):
    rec = ndcg = 0; seen = set()
    for u, items in rank.items():
        seen.update(items[:K].tolist())
        p = np.where(items[:K] == held[u])[0]
        if len(p):
            rec += 1; ndcg += 1.0 / np.log2(p[0] + 2)
    n = len(rank)
    return rec / n, ndcg / n, len(seen) / N_ITEMS

To run all of this against MovieLens instead, read ratings.csv, keep rows with a rating of 4 or higher as implicit positives, and factorise the user and movie columns into contiguous integer codes before building the sparse matrix. Everything after that line is unchanged.

Baselines that beat matrix factorisation

Four models go in: recommend whatever is most popular, item to item cosine, ALS with 16 latent factors, and ALS with 64. Only one line differs between the last two.

import os
os.environ['OPENBLAS_NUM_THREADS'] = '1'
from implicit.als import AlternatingLeastSquares
from implicit.nearest_neighbours import CosineRecommender

item_pop = np.asarray(train.sum(axis=0)).ravel()
order = np.argsort(-item_pop)
pop_rank = {}
for u in evalu:
    own = set(train.indices[train.indptr[u]:train.indptr[u + 1]].tolist())
    pop_rank[u] = np.array([i for i in order if i not in own][:K])

knn = CosineRecommender(K=40)
knn.fit(train, show_progress=False)
ids, _ = knn.recommend(evalu, train[evalu], N=K, filter_already_liked_items=True)
knn_rank = {u: ids[i] for i, u in enumerate(evalu)}

def run_als(f):
    m = AlternatingLeastSquares(factors=f, regularization=0.01, alpha=10.0,
                                iterations=30, random_state=7, num_threads=1)
    m.fit(train, show_progress=False)
    ids, _ = m.recommend(evalu, train[evalu], N=K, filter_already_liked_items=True)
    return m, {u: ids[i] for i, u in enumerate(evalu)}

als, als_rank = run_als(16)
_, als64_rank = run_als(64)

for name, r in [('most popular    ', pop_rank), ('item item cosine', knn_rank),
                ('ALS 16 factors  ', als_rank), ('ALS 64 factors  ', als64_rank)]:
    print('%s recall@10 %.4f  ndcg@10 %.4f  coverage %.3f' % ((name,) + score(r)))

most popular     recall@10 0.2669  ndcg@10 0.1715  coverage 0.043
item item cosine recall@10 0.2871  ndcg@10 0.1820  coverage 0.434
ALS 16 factors   recall@10 0.2313  ndcg@10 0.1342  coverage 0.398
ALS 64 factors   recall@10 0.0796  ndcg@10 0.0420  coverage 0.811

Read the coverage column before drawing conclusions, because it changes the story. Popularity reached recall of 0.2669 while touching 4.3 percent of the catalogue, meaning 34 items served 2,814 users. ALS at 64 factors covered 81 percent of inventory and scored terribly. Item to item sat between the two on coverage at 43 percent and won on both accuracy metrics, which is why it is my pick here. Where a merchandising team needs breadth, the honest framing is not that one model is better, it is that popularity buys accuracy with catalogue diversity and you should decide consciously what that trade is worth.

Why did more factors hurt so much. Rank controls capacity, and with 800 items, 2 percent density and a median history of 11 interactions, there is nowhere near enough signal to estimate 64 numbers per user. Each user vector starts fitting the specific quirks of eleven observations rather than a taste direction. Sweeping the parameter makes the shape obvious, and the confidence weight interacts with it.

Two readings fall out of that grid. Factor count dominates: moving from 16 to 64 cost more recall than any other change in the sweep. Confidence weight matters almost as much, and the default of 1.0 is close to useless on this data, tripling in quality once alpha reached 10. Regularization, the parameter people reach for first, barely registered. If you tune one thing in an ALS model, tune rank, and use the search machinery from Part 15 to do it rather than guessing.

Cold start and what an empty user vector returns

Cold start names three separate problems that get lumped together. A new user has no history, so there is no vector to look up. A new item has no interactions, so it can never be recommended by any collaborative method and will not be until somebody finds it another way. A new system has neither, which is why every recommender project begins with content rules or popularity and earns its way toward collaborative filtering.

Severity of the new user case is easy to underestimate, so measure it directly. Below is what happens when a trained model meets a user row containing nothing at all.

new = sp.csr_matrix((1, train.shape[1]), dtype=np.float32)
ids, sc = als.recommend(0, new, N=5, recalculate_user=True)
print('ids', ids, 'scores', sc)

ids [4 3 2 1 0] scores [0. 0. 0. 0. 0.]

That is the failure I opened with, reproduced in three lines. A zero user vector dotted against every item factor gives zero for all of them, argsort breaks the tie by position, and out come the first five item ids in reverse. Every downstream check passes: the response is well formed, the right length, correctly typed. Guard against it explicitly by checking that the maximum returned score exceeds zero and routing to a fallback when it does not, because no library is going to raise on your behalf.

Contrast that silence with an error the same library does raise, which is worth meeting once so you recognise it. Passing the full training matrix to a batch recommend call, instead of the sliced rows for the users you asked about, fails immediately.

>>> als.recommend(evalu, train, N=10)
Traceback (most recent call last):
  File "/tmp/err.py", line 10, in <module>
    m.recommend(evalu, train, N=10)
  File "implicit/cpu/matrix_factorization_base.py", line 50, in recommend
    raise ValueError("user_items must contain 1 row for every user in userids")
ValueError: user_items must contain 1 row for every user in userids

Note the second half of that fix, because it is the part that bites silently. Row i of the sliced matrix must correspond to user i of the id array. Slice with a differently ordered index and every recommendation is served to the wrong person, with no error anywhere. Serving logic therefore needs a routing decision made before the model is consulted at all.

Fallbacks worth building, and one that failed

Averages hide cold start completely, which is why segmenting by history length is the most useful table in this whole part. Breaking the same evaluation into buckets shows where each model earns its keep.

Three findings sit in that chart and all three are actionable. ALS is at its weakest exactly where users are newest, climbing from 0.1395 at one or two items to 0.2673 by the sixteen to forty bucket, so a global average understates it for engaged users and overstates it for everybody else. Popularity moves the opposite way, strongest on thin history and steadily overtaken as history accumulates. Beyond forty items every model declines, which surprised me until I looked closer: users with very long histories have already consumed most of what their taste cluster contains, so the held out item tends to be an unusual one.

Reading the crossover suggests an obvious fallback for the thinnest bucket, so I built it and it did not work. Given one or two known items, infer the user taste cluster, then recommend the most popular items inside that cluster. Segmented popularity, essentially, and a design I have seen recommended plenty of times.

cold = [u for u in evalu if np.diff(train.indptr)[u] <= 2]
hit = 0
for u in cold:
    own = train.indices[train.indptr[u]:train.indptr[u + 1]]
    t = np.bincount(item_topic[own], minlength=N_TOPICS).argmax()
    cand = np.where(item_topic == t)[0]
    cand = cand[np.argsort(-item_pop[cand])]
    cand = np.array([i for i in cand if i not in set(own.tolist())])[:K]
    if held[u] in cand:
        hit += 1
print('cold users %d  content fallback recall@10 %.4f' % (len(cold), hit / len(cold)))

cold users 337  content fallback recall@10 0.1810
global popular    recall@10 0.2908

Worse by a wide margin, and the reason is instructive. Inferring a cluster from one or two items is close to a coin flip, so a large share of those users get filtered into the wrong segment entirely. Meanwhile the restriction throws away the genuinely popular items outside that cluster, which were carrying most of the baseline hit rate. Two errors compound. Segmentation only pays once your segment assignment is more reliable than the signal you are discarding to make it, and with two observations it is not. Similar reasoning to the trap in correlation versus causation: a relationship that holds in aggregate is not automatically usable on an individual.

Bringing this back to churn

Two threads connect this part to the Telco project we have been carrying since Part 4. First, retention actions are a ranking problem wearing a classification costume. A churn model gives you a probability per customer, but the retention team can call maybe two hundred people a week, so what actually ships is a ranked list with a cutoff. Precision at 200 is the metric that matters, and it is not the same as AUC, exactly as recall at 10 is not accuracy here. Rebuilding the churn evaluation as a top K ranking exercise changes which model you pick, and usually favours the one with better calibration in the high probability tail rather than the best average.

Second, cold start applies to churn just as literally. A customer who signed up nine days ago has almost no behavioural features, and the Telco tenure column encodes exactly that: predictions for very short tenure customers rest on demographics and contract type rather than anything the customer has done. Bucket your churn evaluation by tenure the same way the table above buckets by history, and you will find the same shape, a model that looks fine on average and is close to guessing on the newest cohort. Which is precisely the cohort where retention spend has the most room to change the outcome. Feature availability by tenure is the same discipline covered in cleaning messy data, applied to a segment rather than a column.

Recommender stack I would build first

Build in this order and stop as soon as the next step stops paying. Ship most popular first, with a real leave one out score recorded, because that number is the bar every later model must clear. Add item to item cosine second, since on this data it won every history bucket except the very thinnest and needs no retraining to absorb a new interaction. Introduce ALS third, at a low rank, and tune factors before you touch anything else. Route by history length rather than replacing one model with another. Reserve an exploration slot for cold items from day one, because retrofitting it later means arguing for a deliberate accuracy loss after everybody has grown attached to the current number.

My honest verdict on matrix factorisation after this exercise: it is not the starting point its reputation suggests. On a catalogue of 800 items with two percent density, a 64 factor model lost to fifteen lines of counting by a factor of more than three, and even a well chosen rank finished behind item to item cosine. Reach for ALS when you have millions of interactions, tens of thousands of items and users with substantial histories. Below that, neighbourhood methods and a popularity floor will take you further, and they are far easier to explain when somebody asks why a particular item appeared.

Take whatever recommender you currently run, split its offline evaluation by user history length, and compare each bucket against most popular on the identical split. If popularity wins a bucket, route that bucket to popularity today. That one change has improved more recommendation rails for me than any modelling work I have done on top of them. Part 21 begins the production phase of this series, turning the notebook code you have accumulated across Parts 3 to 20 into a package somebody else can install and run.

References

• AlternatingLeastSquares API reference, implicit 0.7.2 documentation
• Hu, Koren and Volinsky, Collaborative Filtering for Implicit Feedback Datasets, ICDM 2008
• MovieLens latest datasets, GroupLens Research
• scipy.sparse matrix reference, SciPy 1.15.3

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts