Implicit feedback
Implicit feedback is information about a user's preferences that a system infers from the user's ordinary behavior, such as clicks, page views, reading or watching time, purchases and repeat visits, rather than from ratings or judgments the user gives on purpose. It is the usual training signal for item recommender systems and a long-studied source of relevance information for search engines. Implicit feedback is cheap and plentiful because it is recorded as a side effect of use, but it is noisy, usually carries no direct negative signal, and is shaped by what the system chose to show, so models built on it need specific techniques for weighting, sampling and bias correction.[1][8][15]
In recommendation the setting is often called one-class collaborative filtering, because the data contain only positive actions and a large mass of unobserved user-item pairs that mix true dislikes with items the user never encountered.[2] In web search the classic form is clickthrough data, the record of which results a user clicked for a query, which researchers have used both to train ranking functions and to evaluate them.[5][6][19]
Definition and contrast with explicit feedback
Explicit feedback is input a user provides deliberately about an item, such as a star rating or a thumbs-up button. Hu, Koren and Volinsky, writing in 2008, gave Netflix star ratings and TiVo thumbs-up/down buttons as examples and described explicit feedback as the most convenient, high-quality input, but noted that it is not always available, either because users are reluctant to rate products or because the system cannot collect ratings.[1] Implicit feedback instead reflects opinion indirectly through observed behavior; their list of types includes purchase history, browsing history, search patterns and mouse movements.[1]
Rendle and colleagues summarized the practical appeal in 2009: implicit feedback is tracked automatically, so it is much easier to collect, and it is already available in almost any information system, since web servers log every page access.[7] Writing in 2021, Rendle described item recommendation models as "usually learned from implicit feedback" such as past purchases or watched videos.[8]
Hu, Koren and Volinsky identified four characteristics that separate implicit from explicit data:[1]
| Property | Explicit feedback | Implicit feedback |
|---|---|---|
| Negative signal | Users say what they like and dislike | No reliable negative feedback; an unconsumed item may be disliked or simply unknown |
| Noise | Stated preference | Behavior is a guess at preference (an item bought as a gift, a TV left on while the viewer sleeps) |
| Meaning of the number | A rating indicates preference | A count or duration indicates confidence, not strength of preference |
| Evaluation | Error metrics such as mean squared error | Needs measures that handle item availability, competing items and repeat consumption |
The third point is easy to miss. In their television data a user might watch a favorite film once but a series they only quite like every week, so a larger count does not mean a higher preference; it does mean the observation is more trustworthy, because a recurring event is more likely to reflect the user's opinion than a one-time event.[1]
History
Early information filtering
Interest in implicit feedback grew out of information filtering research in the 1990s. In a 1998 AAAI workshop paper, Douglas Oard and Jinmook Kim of the University of Maryland argued that the effort of assigning explicit ratings acts as a disincentive that limits the size of user populations and contributes to data sparsity, and that inferring something like a rating from observable behavior could greatly widen the range of applications for recommender systems.[3] They reported that Morita and Shinoda (1994) and Konstan and colleagues (1997) had found a positive correlation between reading time and explicit ratings in USENET news applications, and they built on a 1997 survey of implicit rating techniques by David Nichols.[3]
Oard and Kim grouped observable behaviors into three categories:[3]
| Category | Observable behaviors |
|---|---|
| Examination | Selection, duration, edit wear, repetition, purchase of an object or subscription |
| Retention | Saving a reference or object (with or without annotation or organization), printing, deleting |
| Reference | Forwarding, replying or posting a follow-up; hypertext links and citations; cut-and-paste and quotation |
They also distinguished two strategies for using these signals: converting observations into estimated ratings and then predicting ratings, or predicting future behavior first and inferring value from it.[3] In 2003 Diane Kelly and Jaime Teevan published an annotated bibliography in ACM SIGIR Forum on implicit feedback for query expansion and user profiling. They wrote that the behaviors studied most extensively were reading time, saving, printing and selecting, that implicit measures were generally thought to be less accurate than explicit ones, that the large quantities that can be gathered at no extra cost to the user made them attractive alternatives, and that they could be combined with explicit ratings for a more accurate picture of user interests.[4]
Clickthrough data in web search
Thorsten Joachims's 2002 KDD paper "Optimizing Search Engines using Clickthrough Data" proposed training a retrieval function from the query log and the log of clicked links instead of from expert relevance judgments, which he described as difficult and expensive to obtain.[5] The paper treated each search as a triplet of query, presented ranking and set of clicked links, and argued that such data is available in abundance and can be recorded at very low cost through a logging proxy.[5] Its key observation was that a click is not an absolute relevance judgment, since users typically scan only the top of the list and a much more relevant document further down may never be seen. Clicks can instead be read as relative preferences: if a user clicked the third result but not the second, the third is probably more relevant than the second.[5] Joachims trained a support vector machine on these pairwise preferences and reported that the learned function adapted a meta-search engine to a group of users and outperformed Google in retrieval quality after only a couple of hundred training examples.[5]
Recommender systems
Two papers at the 2008 IEEE International Conference on Data Mining addressed implicit data in collaborative filtering. Hu, Koren and Volinsky, working with television viewing data from about 300,000 set-top boxes, proposed a confidence-weighted matrix factorization model trained with alternating least squares,[1] later often called iALS[12] and also known as weighted alternating least squares. Rong Pan and colleagues framed the same problem as one-class collaborative filtering and proposed two frameworks, one based on weighted low-rank approximation and one based on sampling negative examples.[2] In 2009 Steffen Rendle and colleagues at the University of Hildesheim introduced Bayesian Personalized Ranking (BPR), a pairwise ranking criterion derived specifically for implicit feedback.[7]
Common signals
The signals used in practice depend on the product. Papers in this area have used:
- Clicks on search results, links or recommended items.[5][7]
- Views and watches, including whether a video was completed. YouTube's 2016 deep recommendation paper trained its candidate generator on watches, treating a completed video as a positive example, even though explicit thumbs up/down and surveys existed on the site, because orders of magnitude more implicit history was available, which allowed recommendations deep in the tail where explicit feedback is very sparse.[11]
- Watch time or dwell time. The same YouTube paper ranked by expected watch time per impression rather than click-through rate, noting that ranking by clicks often promotes deceptive videos users do not finish ("clickbait").[11] A 2014 RecSys paper by Yi and colleagues proposed item-level dwell time as a proxy for relevance, described how to compute it from client-side and server-side logs, and normalized it across devices and contexts.[18]
- Purchases and other transactions, such as an online shop's buying history.[7]
- Consumption counts, such as how many times a viewer watched a program, which Hu and colleagues used as confidence rather than as preference.[1]
- Session-level behavior in search. A 2005 study by Fox, Dumais and colleagues found that the best models of page-level satisfaction combined clickthrough, time spent on the search result page, and how the user left a result or ended the session.[17]
Core problems
No negative examples
The defining difficulty is that implicit data records what users did, not what they rejected. Pan and colleagues pointed out that a user not bookmarking a page cannot be attributed to either lack of interest or lack of awareness, so negative examples and unlabeled positive examples are mixed together.[2] Rendle and colleagues put it as a mixture of real negative feedback and missing values, where a user who has not bought an item may still want to buy it in the future.[7]
Two naive solutions both fail. Ignoring all unobserved pairs, as rating-prediction models do with missing ratings, leaves only positive examples and misrepresents the user profile.[1] Treating every unobserved pair as a negative means that the items the system should recommend in future are presented to the learner as negatives; Rendle and colleagues noted that a model expressive enough to fit this data exactly would predict only zeros and could not rank at all, so such methods work only because of regularization.[7]
Exposure and missing-not-at-random data
Whether a user interacts with an item depends on whether the user saw it. Liang, Charlin, McInerney and Blei (WWW 2016) argued that treating every unconsumed item as weak negative evidence clashes with the common-sense view that users have limited awareness of items, for example a paper they never heard of or a restaurant too far away. They modeled exposure as a latent variable in a probabilistic factorization model and showed that the weighted matrix factorization of Hu and colleagues is a special case of their model.[9] A related problem affects explicit ratings: Harald Steck showed in 2010 that, because users typically rate only a small fraction of items, the absence of a rating itself carries useful information for improving top-k recommendation, and he proposed training objectives and test measures for data missing not at random.[20]
Presentation and position bias
In search, the order of results strongly influences what gets clicked. Joachims, Granka, Pan, Hembrooke and Gay studied this with eye tracking on Google result pages (SIGIR 2005).[6] Users viewed the first and second abstracts with almost equal frequency but clicked the first substantially more often. When the researchers swapped the top two results, there was still a strong tendency to click the first link even when the second abstract was more relevant, which they called trust bias. When the ranking was reversed, users clicked on less relevant results on average, which they called quality bias: the meaning of a click depends on the quality of the whole result list.[6]
Craswell, Zoeter, Taylor and Ramsey (WSDM 2008) perturbed the ranking of a major search engine to measure how click probability depends on position. They found the data were not well explained by models in which some users click rank 1 indiscriminately or attention simply decays with rank; a cascade model, in which users scan from top to bottom and stop at the first worthwhile result, best explained position bias at the top ranks.[14]
Methods in recommender systems
Confidence-weighted factorization
Hu, Koren and Volinsky split each raw observation $r_{ui}$ (for example, how many times user $u$ watched show $i$) into a binary preference $p_{ui}$, equal to 1 when $r_{ui} > 0$ and 0 otherwise, and a confidence $c_{ui}$.[1] A suggested choice was
which gives every user-item pair some minimal confidence and raises confidence as evidence accumulates; $\alpha = 40$ worked well in their experiments, and they also used a logarithmic variant $c_{ui} = 1 + \alpha \log(1 + r_{ui}/\epsilon)$.[1] User factors $x_u$ and item factors $y_i$ are found by minimizing a weighted squared error over all user-item pairs, not only the observed ones:
where $\lambda$ controls regularization.[1] Because the sum covers every pair, the cost function is dense, but the paper showed that alternating least squares updates can be computed in time linear in the number of non-zero observations by reusing a shared $Y^\top Y$ term.[1] On their television data, expected percentile ranking (lower is better, 50% for a random predictor) fell from 16.46% for popularity ranking and 10.74% for a neighborhood model to 8.35% for the factor model with 200 factors.[1]
Rendle, Krichene, Zhang and Koren revisited this method (iALS) at Google in 2021. They noted that it had become one of the most commonly used baselines in recommender research while being reported as no longer competitive in quality, and showed that with careful manual tuning it was highly competitive on four standard benchmarks and outperformed every other method on at least half of the comparisons.[12] The open-source Python library Implicit provides multi-threaded implementations of this alternating least squares model, BPR, logistic matrix factorization and item-item nearest-neighbor models for implicit datasets, with GPU kernels for ALS and BPR.[13]
Negative sampling
Instead of weighting every unobserved pair, a learner can iterate over observed positives and draw a few unobserved items as negatives. Pan and colleagues proposed negative example sampling as one of their two frameworks for one-class collaborative filtering.[2] In his 2021 overview, Rendle described sampling-based algorithms as the most common way to learn recommender systems from implicit feedback and showed that a given weighting of the full loss can be obtained either by changing the sampling distribution or by changing the per-example weights. He noted that uniform sampling mostly draws items that are trivially negative, which is one reason for popularity-based and other non-uniform schemes.[8] See negative sampling.
Pairwise ranking (BPR)
BPR reframes the task as ranking. For each user $u$, every observed item $i$ is assumed to be preferred over every unobserved item $j$, giving training triples $(u, i, j)$; pairs of two observed items or two unobserved items give no information.[7] The model scores $\hat{x}_{uij}$, the difference between the user's predicted scores for $i$ and $j$, and maximizes
where $\sigma$ is the logistic function and $\Theta$ the model parameters.[7] The authors derived the criterion as a maximum a posteriori estimator, related it to maximizing the area under the ROC curve, and optimized it with stochastic gradient descent using bootstrap sampling of triples. They applied it to matrix factorization and adaptive k-nearest-neighbor models and reported better ranking quality than those models' standard training methods.[7]
Loss families
Rendle's 2021 overview groups training objectives for implicit feedback into three families:[8]
| Loss | What it models | Examples discussed in the overview |
|---|---|---|
| Pointwise | Whether a single user-item pair is positive, usually with lower weight on unobserved pairs | iALS, SLIM, iCD |
| Pairwise | Whether an observed item scores above an unobserved one | BPR, WARP |
| Softmax | A probability distribution over the whole item catalogue | Sampled softmax training |
All three are defined over every user-item combination, which is why efficient training, through sampling or through algebraic shortcuts for dot-product models, is the second core challenge after choosing the objective.[8]
Neural recommenders
Neural approaches kept implicit feedback as the main signal. Neural Collaborative Filtering (He and colleagues, WWW 2017) targeted collaborative filtering "on the basis of implicit feedback" and replaced the inner product of matrix factorization with a learned multi-layer perceptron.[10] The YouTube system described by Covington, Adams and Sargin split recommendation into a candidate generation network trained on watches and a ranking network that predicts expected watch time using weighted logistic regression: clicked impressions are weighted by observed watch time and unclicked impressions receive unit weight.[11] Rendle's overview describes this kind of two-stage pipeline, in which the first stage is trained from implicit feedback for high recall and the second reranks the top candidates using impression data.[8] Later industrial work scaled the same signal further. Jiaqi Zhai and colleagues at Meta (ICML 2024) reformulated recommendation as sequential transduction over streams of user actions, which they called Generative Recommenders, and proposed an architecture named HSTU; they reported that HSTU-based models with 1.5 trillion parameters improved metrics in online A/B tests by 12.4%, had been deployed on multiple surfaces of a large internet platform with billions of users, and showed model quality scaling as a power law of training compute.[21]
Methods in search and learning to rank
Clicks as relative preferences
Joachims and colleagues compared several rules for turning clicks into pairwise preferences against explicit relevance judgments. The rule "Click > Skip Above", which prefers a clicked result over each unclicked result ranked above it, agreed with judges' preferences 80.8% of the time on their first phase of data, against an inter-judge agreement of 89.5%, and its accuracy did not change significantly when the ranking was swapped or reversed.[6] They concluded that clicks are informative but biased, and that relative preferences derived from clicks are reasonably accurate on average.[6]
Unbiased learning to rank
Joachims, Swaminathan and Schnabel (WSDM 2017) argued that treating click/no-click as a positive/negative label is severely biased by presentation order, that skip-based preferences only produce preferences against the presented order, and that click models that infer relevance typically need the same query to be seen multiple times.[15] They instead used inverse propensity scoring from causal inference: each clicked result is weighted by the inverse of the estimated probability that the user examined it, which under their assumptions gives an unbiased estimate of ranking loss. Under the simplest position-based model, the examination probability depends only on rank. They estimated those probabilities with a small randomized experiment that swaps results into the top position, trained a Propensity SVM-Rank, and reported substantial improvements on an operational search engine.[15]
Wang, Golbandi, Bendersky, Metzler and Najork (WSDM 2018) noted that randomizing results to estimate position bias can hurt users' search experience, and proposed a regression-based expectation-maximization algorithm that estimates position bias from regular clicks in personal search. They found that the estimated bias significantly improved learning-to-rank models, and that pairwise models used it more effectively than pointwise ones.[16]
Online evaluation
Implicit feedback is also used to compare systems. Interleaving is a technique for evaluating retrieval systems from implicit user feedback; Chapelle, Joachims, Radlinski and Yue analyzed it with data from two major commercial search engines and a scientific literature search system, comparing its agreement with manual relevance judgments and other implicit measures and estimating its statistical efficiency.[19] Production systems also rely on live A/B testing; the YouTube team used offline metrics during development but made final decisions from live experiments that measure changes in click-through rate, watch time and other engagement metrics.[11]
Evaluation challenges
Because there is no ground truth of dislikes, standard rating metrics do not apply directly. Hu and colleagues evaluated with expected percentile ranking of actually watched programs and noted that, for television, it is unclear how to score a show watched more than once or how to compare two shows broadcast at the same time.[1] The 2021 iALS study by Rendle and colleagues attributed contradictory published results for iALS to the difficulty of evaluating baselines, noting that such benchmark findings influence both the direction of research and the algorithms practitioners adopt.[12] For data missing not at random, Steck proposed performance measures that can be estimated without bias under mild assumptions, together with training objectives that account for all ratings, observed or missing.[20]
Continue reading
- Collaborative filteringCollaborative filtering (CF) is a family of techniques used in recommendation systems that predicts a user's preferences by collecting and analyzing preference signals from many other users.Mentioned 6 times above
- Recommender SystemA recommender system (also called a recommendation system or recommendation engine) is an information filtering system that predicts the preference a user would give to an item and uses those predictions to…Mentioned 5 times above
- Matrix factorizationMatrix factorization is a family of mathematical techniques that decompose a matrix into a product of two or more smaller matrices.Mentioned 5 times above
- Weighted Alternating Least Squares (WALS)Weighted Alternating Least Squares (WALS), also called implicit Alternating Least Squares (iALS) or weighted regularized matrix factorization (WRMF), is a matrix factorization algorithm for collaborative…Listed in See also
- User matrixIn collaborative filtering and matrix factorization recommender systems, the user matrix (commonly written U or P) is the matrix of latent-factor vectors for users: each row is one user's embedding in a…Listed in See also
- Information RetrievalInformation retrieval (IR) is the study and engineering of systems that identify material likely to satisfy an information need.Listed in See also
More from See also
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14Yifan Hu, Yehuda Koren, Chris Volinsky. "Collaborative Filtering for Implicit Feedback Datasets". 2008 Eighth IEEE International Conference on Data Mining, pp. 263-272, December 2008. yifanhu.net/...cf.pdf
- ^1 ^2 ^3 ^4Rong Pan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, Qiang Yang. "One-Class Collaborative Filtering". 2008 Eighth IEEE International Conference on Data Mining, pp. 502-511, December 2008. doi.org/...ICDM.2008.16
- ^1 ^2 ^3 ^4Douglas W. Oard, Jinmook Kim. "Implicit Feedback for Recommender Systems". AAAI Technical Report WS-98-08, 1998. cdn.aaai.org/...WS98-08-021.pdf
- ^Diane Kelly, Jaime Teevan. "Implicit feedback for inferring user preference: a bibliography". ACM SIGIR Forum 37(2), pp. 18-28, September 2003. doi.org/...959258.959260
- ^1 ^2 ^3 ^4 ^5 ^6Thorsten Joachims. "Optimizing Search Engines using Clickthrough Data". Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133-142, July 2002. cs.cornell.edu/...joachims_02c.pdf
- ^1 ^2 ^3 ^4 ^5Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Geri Gay. "Accurately Interpreting Clickthrough Data as Implicit Feedback". Proceedings of the 28th Annual International ACM SIGIR Conference, pp. 154-161, August 2005. cs.cornell.edu/...joachims_etal_05a.pdf
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, Lars Schmidt-Thieme. "BPR: Bayesian Personalized Ranking from Implicit Feedback". Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009). arXiv:1205.2618. arxiv.org/...1205.2618
- ^1 ^2 ^3 ^4 ^5 ^6Steffen Rendle. "Item Recommendation from Implicit Feedback". arXiv:2101.08769, January 2021. arxiv.org/...2101.08769
- ^Dawen Liang, Laurent Charlin, James McInerney, David M. Blei. "Modeling User Exposure in Recommendation". Proceedings of the 25th International Conference on World Wide Web, pp. 951-961, April 2016. arxiv.org/...1510.07025
- ^Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, Tat-Seng Chua. "Neural Collaborative Filtering". Proceedings of the 26th International Conference on World Wide Web, pp. 173-182, April 2017. arxiv.org/...1708.05031
- ^1 ^2 ^3 ^4Paul Covington, Jay Adams, Emre Sargin. "Deep Neural Networks for YouTube Recommendations". Proceedings of the 10th ACM Conference on Recommender Systems, pp. 191-198, September 2016. static.googleusercontent.com/...45530.pdf
- ^1 ^2 ^3Steffen Rendle, Walid Krichene, Li Zhang, Yehuda Koren. "Revisiting the Performance of iALS on Item Recommendation Benchmarks". arXiv:2110.14037, October 2021. arxiv.org/...2110.14037
- ^benfred/implicit. "Implicit: Fast Python Collaborative Filtering for Implicit Datasets". GitHub repository README. github.com/...implicit
- ^Nick Craswell, Onno Zoeter, Michael Taylor, Bill Ramsey. "An experimental comparison of click position-bias models". Proceedings of the 2008 International Conference on Web Search and Data Mining (WSDM '08), p. 87, 2008. microsoft.com/...son-of-click-position-bias-models
- ^1 ^2 ^3Thorsten Joachims, Adith Swaminathan, Tobias Schnabel. "Unbiased Learning-to-Rank with Biased Feedback". Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 781-789, February 2017. arxiv.org/...1608.04468
- ^Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, Marc Najork. "Position Bias Estimation for Unbiased Learning to Rank in Personal Search". Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 610-618, February 2018. doi.org/...3159652.3159732
- ^Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, Thomas White. "Evaluating implicit measures to improve web search". ACM Transactions on Information Systems 23(2), pp. 147-168, April 2005. doi.org/...1059981.1059982
- ^Xing Yi, Liangjie Hong, Erheng Zhong, Nathan Nan Liu, Suju Rajan. "Beyond clicks: dwell time for personalization". Proceedings of the 8th ACM Conference on Recommender Systems, pp. 113-120, October 2014. doi.org/...2645710.2645724
- ^1 ^2Olivier Chapelle, Thorsten Joachims, Filip Radlinski, Yisong Yue. "Large-scale validation and analysis of interleaved search evaluation". ACM Transactions on Information Systems 30(1), Article 6, February 2012. doi.org/...2094072.2094078
- ^1 ^2Harald Steck. "Training and testing of recommender systems on data missing not at random". Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 713-722, July 2010. doi.org/...1835804.1835895
- ^Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, Yu Shi. "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations". Proceedings of the 41st International Conference on Machine Learning (ICML 2024), PMLR 235. arXiv:2402.17152. arxiv.org/...2402.17152
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 3,875 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 22 cited and independent sources; 0 material and 3 minor corrections applied before publication.
Cite this page: AI Wiki. "Implicit feedback." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/implicit_feedback

