SUBMISSION: 510 TITLE: Attentive Sequential Models of Latent Intent for Next Item Recommendation ------------------------- METAREVIEW ------------------------ This is the meta-review of the paper based on reviews and discussions. Intent identification, when the intent is latent, is an important challenge faced by many recommender systems, and is a challenging task given that intent values are never observed. This makes the present work very timely, and important. While the reviewers make some excellent suggestions, around areas of improvement for the paper (e.g. better motivation, experimental analysis etc), the paper is promising enough to be accepted as a short paper. ----------------------- REVIEW 1 --------------------- SUBMISSION: 510 TITLE: Attentive Sequential Models of Latent Intent for Next Item Recommendation AUTHORS: Md Mehrab Tanjim, Congzhe Su, Ethan Benjamin, Diane Hu, Liangjie Hong and Julian McAuley ----------- Strengths ----------- This submission proposes a model that learns item similarities from users’ interaction histories via a self-attention layer and uses a Temporal Convolutional Network layer to obtain a latent representation of the user’s intent from her actions on a particular category to predict the next item. The motivation is clear. It yields better performance than existing methods. ----------- Weaknesses ----------- The novelty of this paper is a little frail. It just combines the existing self-attention with TCN model. The experiment is not sufficient. An ablation study is necessary. Fig 1 and 2 need to be improved. ----------- Relevance ----------- SCORE: 2 (Yes, but only to a small group of people) ----------- Summary and review comments ----------- This paper proposes a model that learns item similarities from users’ interaction histories via a self-attention layer and uses a Temporal Convolutional Network layer to obtain a latent representation of the user’s intent from her actions on a particular category to predict the next item. My major concerns are: 1. The novelty of this paper is a little frail. It just combines the existing self-attention with TCN model to find similarities among items and capture users’ hidden intents. 2. An ablation study is necessary, and Section 4.6 mentions the ‘Seq-Seq’ attention of SASRec, but it does not show the results in Table 4. 3. Fig 1 and 2 need to be improved. ---------------- Go through the rebuttal, the idea of this work seems acceptable for a short paper. The rebuttal addressed my major concerns, and I suggest the authors should also highlight the idea and its novelty in their paper. Overall, my suggestion is weak accept. ----------- Overall score ----------- SCORE: 1 (weak accept) ----------- Reviewer's confidence ----------- SCORE: 3 ((medium)) ----------------------- REVIEW 2 --------------------- SUBMISSION: 510 TITLE: Attentive Sequential Models of Latent Intent for Next Item Recommendation AUTHORS: Md Mehrab Tanjim, Congzhe Su, Ethan Benjamin, Diane Hu, Liangjie Hong and Julian McAuley ----------- Strengths ----------- -. The paper proposes an attentive sequential model. -. Paper is well written with explanations for latent intent and other components of the model, easy to read -. Well experimented results and analysis including considerations for training scalability ----------- Weaknesses ----------- -. Requires a large number of categories for reasonable improvements in results, as shown in tmall-small dataset -. Latent Intent Embeddings are not explored as to what exactly they embed and why exactly they provide improvements -. Can include the performance metrics in table 3 for ablated ASLI without Latent Intent, especially for tmall-small to check if there is any performance degradation ----------- Relevance ----------- SCORE: 2 (Yes, but only to a small group of people) ----------- Summary and review comments ----------- Problem & Summary: The paper is based on the observation that the previous sequential recommendation models do not consider user intent (such as shopping, browsing or discovering new items) when recommending next item. The paper proceeds with the fact that the user intent is a latent variable and so the embedding for the latent user intent is learnt from various actions of the user on each item category (checkout, adding to cart, clicks etc.). This is different from other models that use embeddings of actions or categories directly in the sense that the model predicts the next interaction from the latent user intent along with the next item. Motivation & Contribution: The major contribution is the motivation for the use of user intent and approach of obtaining latent user intent by jointly predicting next item and next interaction with a good amount of experimentation as to empirically measure the impact of the latent intent on next item recommendation Improvements: The analysis of latent user intent representation can provide interesting findings as to what they embed (perhaps a correlation between other behaviors of user) and why they provide improvements The self-attention mechanism is applied on sequence of items however the its effect is not analyzed. Does the self-attention help recognize which historical items are more important to predict the next item? Authors could visualize attention matrix n x n where n is the length of sequence of items to see if self-attention really works. Clarification: Report performance of ablated ASLI without user latent intent in table 3 Minor error: dataest -> dataset ----------- Overall score ----------- SCORE: 1 (weak accept) ----------- Reviewer's confidence ----------- SCORE: 3 ((medium)) ----------------------- REVIEW 3 --------------------- SUBMISSION: 510 TITLE: Attentive Sequential Models of Latent Intent for Next Item Recommendation AUTHORS: Md Mehrab Tanjim, Congzhe Su, Ethan Benjamin, Diane Hu, Liangjie Hong and Julian McAuley ----------- Strengths ----------- + In general, it is well written and easy to read. + Solid experiments with reasonable experimental results. ----------- Weaknesses ----------- - Less of novelty in technical part, given that the self-attentive networks (or Transformers), Temporal Convolutional Networks (TCN), and Feed-forward Network (FFN) methods are quite common in the context of deep learning. - Lack of insightful analyses / discussions on why the proposed ASLI can be better than other compared methods. ----------- Relevance ----------- SCORE: 3 (Yes, to a large group of people) ----------- Summary and review comments ----------- This paper proposes an Attentive Sequential model of Latent Intent (ASLI) to capture user's latent intents across different services for recommendation. The idea of using self-attention layer and temporal convolutional network layer to capture user's intent from his/her previous actions is interesting. Overall the experimental results look good, outperforming several state-of-the-art methods, such as BPR-MF, NextItRec, and SASRec. To me, the submission feels almost like an applied work of using transformer and TCN on next-item recommendation task. The idea is interesting, but there are few insightful analyses / discussions provide in the paper. However, there's still possibly a substantial contribution in the work, so I'd lean towards a "weak accept", on the basis that the authors have a good idea that deserves deeper exploration and better presentation. ----------- Overall score ----------- SCORE: 1 (weak accept) ----------- Reviewer's confidence ----------- SCORE: 5 ((expert)) ----------------------- REVIEW 4 --------------------- SUBMISSION: 510 TITLE: Attentive Sequential Models of Latent Intent for Next Item Recommendation AUTHORS: Md Mehrab Tanjim, Congzhe Su, Ethan Benjamin, Diane Hu, Liangjie Hong and Julian McAuley ----------- Strengths ----------- 1. The motivation of this work is clear, i.e., using user-category interactions instead of user-item interactions to reduce the data sparsity. 2. The structure of this paper is easy to follow. 3. The details of experimental datasets, setting, metrics and implementation are relatively clear. ----------- Weaknesses ----------- 1. The scale of negative examples is too small. In subsection 3.4.5, for training the model, the authors select a negative example for each positive example. In fact, this scale is not enough. As introduced in [9], the scale is 1 : 4, i.e., 4 negative examples for each positive example. In fact, for the recommendation task in [9] (see experimental results), when the ratio is around 1:7, the model can achieve a best performance. 2. This work ignores some important state-of-the-art approaches. The authors should review or compare with these approaches, such as [1] Wang, Shoujin, et al. "Modeling multi-purpose sessions for next-item recommendations via mixture-channel purpose routing networks." Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, 2019. [2] Zhang, Shuai, et al. "Next Item Recommendation with Self-Attentive Metric Learning." Thirty-Third AAAI Conference on Artificial Intelligence. Vol. 9. 2019. 3. Figure 1 is not clear, the data flow is messy. ----------- Relevance ----------- SCORE: 2 (Yes, but only to a small group of people) ----------- Summary and review comments ----------- This work is closely related to the scope of the conference. The technical details are carefully explained in Section 3. However, the authors do not highlight the contribution of user-category interactions in the proposed model. In subsection 2.1, the authors clearly explained the motivation, but Section 3 only focuses on explaining the technical details and ignore the rationales behind those. Also, my concern is that using user-category interactions may ease the data sparsity and improve the recommendation accuracy of next-category recommendation rather than next-item recommendation. You should explain more about this issue. As for the experiment part, compared with the baseline models, the results of the experiments seem to be good. However, my concern is about fair comparison. As I mentioned in “Weaknesses”, the work ignores some important state-of-the-art approaches, and, authors should compare the proposed model with them to make the experiments more convincing. In addition, I suggest the authors to show the detailed user-item sparsity and user-category sparsity of each dataset in Table 2, which is better for comparison. Overall, the idea of this paper is not too hard to follow, but it lacks in clear rational explanation. Somre more experiments are necessary. ----------- Overall score ----------- SCORE: -1 (weak reject) ----------- Reviewer's confidence ----------- SCORE: 4 ((high))