------------------------- METAREVIEW ------------------------ This paper suggests methods for efficient candidate generation in recommender systems. As the authors point out, a lot of work has happened on improving accuracy of predictions. However, candidate generation is an important stage for large scale recommender systems and needs more research. The authors provide a nice framework and a solution. The paper is well written and well structured. The experiments are extensive. ----------------------- REVIEW 1 --------------------- SUBMISSION: 519 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper: ----------- 1.The paper has good readability and clear thinking on the whole. 2.It propose a framework(CIGAR) which combines candidate generation and re-ranking. The candidate generation is accelerated by an improved hash-based method and re-ranking candidates with real-valued models. CIGAR exhibits both the efficiency of hashing and the accuracy of real-valued methods. 3.The author(s) conducted comprehensive experiments on several large-scale datasets, where observed orders-of-magnitude improvements in ranking efficiency, while maintaining or improving upon state-of-the-art accuracy. ----------- Three negative aspects of the paper: ----------- 1.Compared with the existing learning-to-hash methods, the part of Section 3.1 and 3.3 is only a small improvement from an innovative point of view. 2.This paper proposes a framework called CIGAR, which includes candidate generation and re-ranking. The final results of the method may rely more on the method used in the re-rank section. 3.The main idea of the model is to "exchange space for time" and speed up the retrieval process by mapping binary codes and building hash tables. The experiments mainly focus on the comparative analysis of the accuracy and time efficiency of the model, It could be interesting to see the discussion of time complexity as well as other settings in the experiments ----------- Overall evaluation ----------- SCORE: 1 (weak accept) ----- TEXT: This paper presents a systematic approach to generate high-quality candidates efficiently.The authors present the method and some experimental results about the algorithm. The algorithm is based on a common industrial strategy of candidate generation and re-ranking. Candidate generation is partially implemented by an improved hash-based method.The authors expose that their algorithm is able to maintain or improve upon state-of-the-art accuracy with orders-of-magnitude improvements in ranking efficiency. Are there any disadvantage associated to this improvement in resources? Section 3.1(used by other learning-to-hash approaches ) is relatively modest in terms of innovation. ----------------------- REVIEW 2 --------------------- SUBMISSION: 519 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper: ----------- 1. Propose a method for candidate generation, a critical (but somewhat overlooked) subroutine in recommender systems that seek to efficiently generate Top-N recommendations. 2. Proposed method bridges the gap between existing state-of-the-art for Top-N recommendation and the methods based on binary codes. 3. Perform a comprehensive study on several large-scale real-world datasets consisting of millions of users/items and hundreds of millions of interactions. Our results show that CIGAR significantly boosts the Top-N accuracy against state-of-the-art recommendation models, while reducing the query time by orders of magnitude ----------- Three negative aspects of the paper: ----------- 1. Most techniques used in the proposed method are from previous works 2. The reason why not using all candidates in the re-ranking stage is missing 3. The setting for the number of sampling data points in D^+ is missing ----------- Overall evaluation ----------- SCORE: 0 (borderline paper) ----- TEXT: This paper conducts many experiments and provide detailed discussion about the proposed method, but several issues need to be addressed. 1. Many techniques used in the proposed method are from previous works. For example, the learning of preference-preserving with binary codes is from [44], the approximation of sign function is from hashNet, the building of hash table is from MIH [30]. The authors should justify the novelty of their proposed method. 2. In the manuscript: a. the authors claimed that "we argue that such an approach is sub-optimal as existing models are typically trained to produce rankings for all items, while our re-ranking models only rank the c generated candidates.", why? The difference is the items in the pool, and ranking for all items considers all items, giving a base to view all the items and that seems to be more likely come to an optimal result. b. the authors claimed that "the retrieved candidates are often ‘difficult’ items (i.e., items that are hard for ranking models to discriminate) or at the very least are not a uniform sample of items.", why? 3. The experimental results show promising results owing to the sampling strategy listed in Eq. (10), but I have two comments regarding Eq. (10). a. The reason why using only portion of the candidate items is a better choice is missing. The candidate items are supposed to be with good quantity, why not using all candidates along with some items from D? b. The element in set C is a tuple (u, i, j), where i is from I^+, and j is from the intersect of I^- and C_u. The candidates, namely the items in C_u, are supposed to be all positive items, but C considers to retrieve negative item from C_u, why? 4. The information for Figure 4 is not clear. The y-axis means improvement of HR@200 or HR@10, what are the base models? 5. In Section 4.5, the manuscript states that "Figure 3 shows the HR@10 of various approaches with different numbers of candidates.", but the y values in Figure 3 are not HR@10, why? ----------------------- REVIEW 3 --------------------- SUBMISSION: 519 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper: ----------- 1. This paper has a clear structure. The shown figures as well as algorithm support understanding well. There is a sufficient background chapter including both representative rankings models for implicit feedback and hash-based models for effecient recommendation. 2. Comprehensive experiments are conducted for a series of research questions which covers a wide range of comparison against major related works introduced. The datasets used in the experiment vary significantly in domains, sparsity, and #user/items. The experimental section provides details and exclusive explanation. 3. The proposed a candidate generation and re-ranking based framework (CIGAR) shows solid experiment results which significantly boosts the Top-N accuracy and recommendation efficiency. ----------- Three negative aspects of the paper: ----------- 1. There is a hyper-parameter beta used for learning preference-preserving binary codes, while there is no detailed explanation about the intuition of beta's formulas in Algorithm 1. 2. In section 4.4, the paper provides a comparison against DPR on MovieLens-1M dataset which seems like a denser dataset. It would be better to add an experiment to compare with DPR on a sparse dataset since the sparsity is one crucial issue. 3. In section 4.7, the recommendation efficiency analysis claims the fast retrieval speed of CIGAR is due to the efficiency of hashtable lookup and the small number of candidates for re-ranking. Well, it also mentions that it adopted pre-compute and store all candidates for constructing triplets in C during candidate reranking in section 3.3. Would this also be one possible reason for better efficiency? ----------- Overall evaluation ----------- SCORE: 2 (accept) ----- TEXT: The paper proposed a candidate generation and re-ranking based framework (CIGAR) which shows solid experiment results and potential to be used widely. It performs a comprehensive study on several large-scale real-world datasets consisting of various domains, sparsity, and #user/items. The shown figures and exclusive explanation supports the solid experiment results. The proposed preference-preserving binary codes learning and multi-index hashing contribute to the small number of high-quality candidates for reranking, which improve the recommendation efficiency significantly. And the introduced candidate-oriented sampling strategies boost the Top-N accuracy against SOTA recommendation models. I would recommend to accept the paper. ============================= KDD (reject) ============================== ------------------------- METAREVIEW ------------------------ RECOMMENDATION: accept TEXT: The reviewers agree that the paper is well written, but have a concern about the novelty. The paper should clearly describe the challenges in integrating previously proposed techniques. ----------------------- REVIEW 1 --------------------- SUBMISSION: 604 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper ----------- 1 Combining a factorization model (e.g., BPR) with binary codes, i.e., HashRec in this paper, and an ordinal factorization model (BPR) in a two-step solution is new. 2 The authors conduct extensive empirical studies to show the effectiveness of the proposed two-step solution. 3 The paper is very well presented. ----------- Three negative aspects of the paper ----------- 1 The novelty of the two-step solution, i.e., candidate items generation and re-ranking, is limited. 2 For the model used in each of the two steps, the technical contribution is limited considering the existing works, e.g., [3], [18], [32] and [44]. 3 Lack of convincing analysis of why the two-step solution performs much better in terms of recommendation accuracy as shown in Table 3. For example, HashRec < BPR-MF < Hash-Rec + BPR-MF, where Hash-Rec is basically BPR with binary codes. ----------- Overall evaluation ----------- The authors study a top-N recommendation problem where users' implicit feedback to items are available. Specifically, the authors design a two-step solution, including (i) candidate items generation via a model with binary codes, and (ii) candidate items re-ranking via a continuous model (instead of that with binary codes). More specifically, one typical configuration is: (i) BPR with binary codes shown in Eqs.(7-8) and ordinal BPR shown in Eq.(11). The authors then conduct extensive empirical studies to show the effectiveness of the proposed solution. Overall, the paper is very well presented. Some major concerns: 1 The novelty of the overall two-step solution, i.e., candidate items generation and re-ranking, is limited, though it can indeed improve efficiency because of the binary codes. 2 For the model used in each of the two steps, the technical contribution is limited considering the existing works, e.g., [3], [18], [32] and [44]. 3 Lack of convincing analysis and discussion of why the two-step solution performs much better in terms of recommendation accuracy as shown in Table 3. For example, HashRec < BPR-MF < Hash-Rec + BPR-MF, where Hash-Rec is basically BPR with binary codes. ----------- Reproducibility ----------- SCORE: 5 (Yes - Excellent. It provides excellent and complete information that will make it easy to reproduce the insights/results.) ----------- Please justify your answer regarding Reproducibility ----------- The two-step solution is straightforward and is easy to reproduce the results with the configurations provided in the supplementary material. ----------------------- REVIEW 2 --------------------- SUBMISSION: 604 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper ----------- This paper is clearly structured and well written. This method achieves better top-N accuracy compared with some baselines that perform exhaustive rankings. There are sufficient tables, figures and formulas to prove that the proposed method is great. ----------- Three negative aspects of the paper ----------- This paper lacks some novelty and only combines some existing works. There is no detailed analysis of why this model performs better than other methods. There are no detailed formulas for the evaluation protocol. ----------- Overall evaluation ----------- The method of this paper aims to accelerates the retrieval time on large-scale datasets. It actually achieves better performance compared with some baselines that perform exhaustive rankings, but there is no detailed analysis of why this model performs better than other methods. In addition, the model lacks innovation and it just integrates some existing works. ----------- Reproducibility ----------- SCORE: 4 (Yes - Good. It provides reasonably complete information that will help reproduce the results.) ----------- Please justify your answer regarding Reproducibility ----------- Description of experimental methodology, empirical evaluation, tuning parameter list and search space is detailed. We can reach the public datasets in this paper. ----------------------- REVIEW 3 --------------------- SUBMISSION: 604 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper ----------- 1. Evaluations can well support the effectiveness of this proposed top-N recommendation framework. 2. Presentation is easy to follow. 3. Most related work are included. ----------- Three negative aspects of the paper ----------- 1. The paper integrates the previous work and there are no new methods proposed. 2. There is no discussion on the challenge of integration and solutions. ----------- Overall evaluation ----------- For large-scale top-N recommendation task, this paper presents a
 two-round framework, i.e., candidate generation and re-ranking.
 The main contribution here is how to generate binary codes for items,
 which enables efficient online candidate generation that only requires
 Hamming distance of two binary codes.

 For optimizing binary code generations, this paper uses implicit feedback proposed in [32].

 To retrieving candidate items online, this paper adopts Multi-Index Hashing
 (MIH) proposed in [30]. Training with implicit feedback is discussed in the following paper, cited as [32]. Training an auxiliary real-valued embeddings, and converting real-valued embeddings binary code are common techniques used in [3]. The use of tanh(\beta * x) to approximate sign(x) can also be found in [3]. Overall this paper is easy to follow and discusses recent advancements for
 recommendation tasks. However, the novelty of proposed techniques is limited and the paper does not discuss the challenge in integrating these techniques, and the novelty in integration. ----------- Reproducibility ----------- SCORE: 5 (Yes - Excellent. It provides excellent and complete information that will make it easy to reproduce the insights/results.) ----------- Please justify your answer regarding Reproducibility ----------- Details about implementations are clearly presented. Parameter settings are also shown. Most datasets are public. ----------------------- REVIEW 4 --------------------- SUBMISSION: 604 TITLE: Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation AUTHORS: Wang-Cheng Kang and Julian McAuley ----------- Three positive aspects of the paper ----------- the paper presents a novel approach to content recommendation that combines several intuitive ideas into a very efficient and effective framework the material is presented in an easy to follow manner and appears to be technically sound experimental protocols and results are convincing and demonstrate the advantages of the approach ----------- Three negative aspects of the paper ----------- this is but nitpicking: some of the figures are too small to be easily legible ----------- Overall evaluation ----------- The authors present a content recommender approach where they first learn to encode users and items in terms of binary representations which then allow for efficient hashing and subsequent ranking in order to generate recommendations of other content. The paper is well written and appears to be technical sound. Extensive and convincing experiments on several large scale standard benchmark data sets show that the proposed approach is efficient and effective (i.e. is fast and produced reasonable results). ----------- Reproducibility ----------- SCORE: 3 (Yes - Fair. The information provided represents a fair effort to make it possible for readers to reproduce the results.) ----------- Please justify your answer regarding Reproducibility ----------- To people working in this area, the paper should definitely present enough details so as to reproduce its results.