----------------------- REVIEW 1 --------------------- PAPER: 2602 TITLE: Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering AUTHORS: Ruining He, Chunbin Lin, Jianguo Wang and Julian McAuley Significance: 1 (low (minimal contribution or weak impact)) Soundness: 3 (correct) Scholarship: 3 (excellent coverage of related work) Clarity: 3 (well written) Breadth of Interest: 2 (limited to specialty area) SUMMARY RATING: 1 (+ (weak accept)) CONFIDENCE: 2 (reasonably confident) ----------- Summarize the Main Contribution of the Paper ----------- The authors use the category tree hierarchy to improve recommendations. They do so by assigning The authors show that by using the tree, their achieve better results than methods that do not use this tree. ----------- Comments for the Authors ----------- Although the paper is well written, this work seems to be a small improvement over: He, Ruining, and Julian McAuley. "VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback."(2016), by taking into account the category tree hierarchy. VBPR-C seems to have a negligible improvement over VBPR, I wonder if the authors have considered different ways (aside of Sherlock and VBPR-C) to use the category tree hierarchy information. I wonder how well a method that will simply add a bias term to each category on the first or second level perform (rather than only to the leaves). ----------------------- REVIEW 2 --------------------- PAPER: 2602 TITLE: Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering AUTHORS: Ruining He, Chunbin Lin, Jianguo Wang and Julian McAuley Significance: 2 (medium (modest contribution or average impact)) Soundness: 3 (correct) Scholarship: 2 (relevant literature cited but could expand) Clarity: 2 (mostly readable with some scope of improvement) Breadth of Interest: 3 (some interest beyond specialty area) SUMMARY RATING: -1 (- (weak reject)) CONFIDENCE: 3 (highly confident) ----------- Summarize the Main Contribution of the Paper ----------- This paper proposes a method that integrates the modelling of item visual appearance with item category hierarchy for better recommendation. Built upon an existing method, which learns a single global embedding matrix for projecting visual features, the proposed method learns different embedding matrices for different leaf categories, thus it is capable to learn the variance and subtlety of item visual properties. By sharing several dimensions in the embedding matrices, different leaf categories also share certain amount of commonalities, thus can learn global item properties ----------- Comments for the Authors ----------- This paper focuses on a relevant and timely topic, i.e. using structured auxiliary features (item category) for recommendation. The paper is well structured and well written, and can be easily followed. ==== MAJOR COMMENTS ==== The main weakness of this paper comes from the limited originality and the power of the proposed method: 1) The main innovating idea, i.e. stacking the embedding matrix using category hierarchy (section 3.3), is based on existing method of visually-aware bayesian ranking (section 3.1, 3.4) and the embedding kernel (section 3.2). 2) The proposed method does not fully exploit the category hierarchy. For example, a same node in different paths of different leaf nodes can have different influences. For instance for K’=7 one leaf node could have a embedding matrix with 4:2:1 split, while another node could have a 3 :3:1 split. The way of stacking the embedding matrices is now, however, all the same for different leaf nodes. 3) The proposed method is not generally applicable to imbalanced category hierarchy. The authors propose to reduce an imbalanced category hierarchy to a balanced one, which, however, could lose information. The problem will become severe if only a small percent of leaf nodes have paths of small height (e.g. 2), while most leaf categories have paths of large height (e.g. 10); the reduced balanced hierarchy will only have a height of 2, losing most of the information in the original imbalance hierarchy. The current method is not extensively validated in the experiment section. First of all, one would expect to see the influence of the main parameter K’ on recommendation performance. More importantly, the split of K’ is now empirically given (section 4.4); while it is highly interesting to show the influences of different split settings on recommendation performance, as well as their corresponding visualisations and interpretations (as in section 4.6). There are some popular and generally applicable feature-based recommendation methods, such as Collective Matrix Factorisation, SVDFeatures, and Factorisation Machine. It would be nice to also compare the proposed method to them (by transforming the feature hierarchy to acceptable structures for these methods). ==== MINOR COMMENTS ==== 1. Following concepts need to be clarified in the abstract and introduction : - What’s the relationship between categories and styles? This is not explicit in the abstract and the example in the introduction section; as this paper focuses on visually aware recommendation, the basic concepts like these (category, style) need to be clearly introduced. - What’s the relationship between embedding matrix and visual dimension (paragrah 'existing works ...' in Introduction). 2. Tn the 2nd paragraph of section 3.3, it is said that with 100 categories the number of parameters will reach millions, if separated embedding matrices are learnt for different root nodes. The calculation needs further explanation. Following this, one would expect to see the comparison of the proposed method with the one learning separated embedding matrices. ----------------------- REVIEW 3 --------------------- PAPER: 2602 TITLE: Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering AUTHORS: Ruining He, Chunbin Lin, Jianguo Wang and Julian McAuley Significance: 2 (medium (modest contribution or average impact)) Soundness: 3 (correct) Scholarship: 2 (relevant literature cited but could expand) Clarity: 3 (well written) Breadth of Interest: 3 (some interest beyond specialty area) SUMMARY RATING: 3 (+++) CONFIDENCE: 2 (reasonably confident) ----------- Summarize the Main Contribution of the Paper ----------- This paper addresses the one-class collaborative filtering problem under the setting of utilizing visual factors. In traditional approaches, the item set is treated homogeneously. However, visual dimensions may have different semantics for different item types, e.g., the features to determine whether a coat or a watch is casual are totally different. Assuming that items are organized in a hierarchy, this paper investigates Sherlock to build hierarchical features for item recommendation. In the hierarchy, the feature matrix at each leaf node is the concatenation of the feature fragments from the root to the leaf. In experiments, such a feature representation helps the proposed method outperform three state-of-the-art baselines. ----------- Comments for the Authors ----------- This paper is generally well-written and easy to understand. The motivation that visual semantics differ from type to type sounds interesting, and is reasonably addressed by incorporating hierarchical feature representations in training the recommendation model. Experiments are carried out on real-world datasets, and the four baselines based on variations of BPR seem reasonably set. Strong points: 1. Interesting problem. Item type hierarchy exists in domains like clothes, books, and locations. Methods exploiting the hierarchy signal for recommendation may potentially generalize to multiple domains; 2. The proposed hierarchical feature representation is intuitive and effective; 3. Experiments are conducted on real-life data and the superiority over each baseline is clearly interpreted. Weak points: More spaces could be saved (e.g., from Table 1 and Figure 2) to provide more parameter study. ----------------------- REVIEW 4 --------------------- PAPER: 2602 TITLE: Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering AUTHORS: Ruining He, Chunbin Lin, Jianguo Wang and Julian McAuley Significance: 2 (medium (modest contribution or average impact)) Soundness: 3 (correct) Scholarship: 2 (relevant literature cited but could expand) Clarity: 2 (mostly readable with some scope of improvement) Breadth of Interest: 3 (some interest beyond specialty area) SUMMARY RATING: 3 (+++) CONFIDENCE: 2 (reasonably confident) ----------- Summarize the Main Contribution of the Paper ----------- This paper improves one class collaborative filtering by building a hierarchical embedding to combat sparsity and variability. ----------- Comments for the Authors ----------- The paper is well written and the litterature is well covered even though it could be extended. The idea of using a hierarchy of embeddings is good. However, its novelty is questionnable. Indeed similar approaches have already been used in large scale classification. For example, see: Label embedding trees for large multiclass tasks. In addition, a tree structure may be too restrictive. Indeed, the authors could explore DAG structure that would allow richer representations via composition of segments from several paths to a given leaf. The illustration about visual dimensions in Figure 3 is interesting. It raises the question of further use of this information for better interpretability of the recommendations. How sensitive is the model to its free parameters ? ------------------------- METAREVIEW ------------------------ PAPER: 2602 TITLE: Sherlock: Sparse Hierarchical Embeddings for Visually-aware One-class Collaborative Filtering The reviewers like the clarity of this paper, the well-motivated use of hierarchical methods and experiments on real data. While the reviewers were not entirely convinced by certain aspects of the methods (use of a fixed hierarchy), evaluation (no exploration of K' values) in the overall the strengths of the paper outweigh the areas for improvement. The reviewers suggest that the authors try to substantiate experimentally in the paper the claim made in the rebuttal, that Sherlock offers better interpretability over previous techniques.