----------------------- REVIEW 1 --------------------- PAPER: 277 TITLE: Generating and Personalizing Bundle Recommendations on Steam AUTHORS: Apurva Pathak, Kshitiz Gupta and Julian McAuley ----------- Overall evaluation ----------- This article studied the generating and personalizing bundle recommendations on the Steam video game distribution platform. ----------- Comments to the Author(s) ----------- You stated this work was related in terms of methodology are systems that recommend items to groups of users/clustering of users, but it seems you did not aware the state-of-the-art multi-armed bandits based recommendation algorithms, see, e.g., SIGIR 2016: Collaborative Filtering Bandits; ICML 2016: Distributed Clustering of Linear Bandits in Peer to Peer Networks; ICML 2014: Online Clustering of Bandits, which should be discussed at least. The BR model you proposed basically it's a simple extension of original BPR, which leads to fair performance, but not impressive. Study and exploit these advances above I expect you can get a great gain to beat BPR. In the meanwhile, it would be awesome if you can provide the corresponding formal theoretical guarantee and analysis. The experiment conducted only in one real data is not strong, and you might wanna add other metrics rather than just AUC. Overall, this draft's idea is interesting and the corresponding experimental result is reasonable, I would love to fight for an acceptance for this article. Also, this draft has a couple of English grammar errors, please get the proof-reading done by a good English speaker. ----------------------- REVIEW 2 --------------------- PAPER: 277 TITLE: Generating and Personalizing Bundle Recommendations on Steam AUTHORS: Apurva Pathak, Kshitiz Gupta and Julian McAuley ----------- Overall evaluation ----------- We believe this paper to be highly relevant to SIGIR, because it proposes a solution to the problem of bundle recommendation, a problem relevant to the SIGIR community that has not been researched extensively. The authors also plan to release the data used in the study, which, to the best of our knowledge, is the first release of bundle sales data from an online retailer, which can prove valuable to the recommendation systems community and beyond. As this is an under-researched area, the solution proposed is novel, although the learning algorithm used (Bayesian Personalized Ranking) was published in 2009. The paper shows a good understanding of the problem and limitations, with good technical quality. The presentation is also excellent. The potential impact of the work presented in the paper is somewhat limited, due to the specificity of the problem, and the fact that no new learning algorithm is proposed. However the released dataset could be used in many other papers. ----------- Comments to the Author(s) ----------- The paper tackles the problem of bundle recommendation and generation. Bundles are sets of items in online marketplaces that can be purchased together, usually while offering a discount. Bundles can be beneficial to the retailer as a means to drive revenue, and to content creators by giving them a wider audience for their products. The paper deals with two tasks: 1) Recommending existing bundles to individual users 2) Generating new, personalized, bundles to recommend The learning algorithm used is Bayesian Personalized Ranking (BPR), a recommendation algorithm aimed at directly optimizing rankings between items. The proposed solutions to both tasks rely on first running BPR using matrix factorization on the training data, in order to obtain latent representations for users and items. Those latent representations are then used to train another BPR model customized for the bundle recommendation task to provide personalized rankings between bundles for each user. Finally the results from the bundle recommendation task are used in the bundle generation task, to greedily grow an initial set of items according to the preferences of each user. The evaluation is performed on historical data. For the bundle recommendation task, AUC is selected as the metric, to measure how often the model correctly ranks item pairs for each user. The proposed model is shown to outperform the plain BPR baseline, and is shown to be resistant to a variant of the "cold start" problem, when bundles contain at least one game which has not been observed in any other bundle. For the bundle generation task, the average rank of the items in the generated bundles and the diversity of the items within are used as indicators of performance. In our view, the contributions of this paper are two-fold: * A novel dataset to be made available for a recommendation task that, to the best of our knowledge, there are no alternative real-world datasets available. * A new objective customized for the bundle recommendation problem is proposed for BPR that is shown to outperform plain BPR for the bundle recommendation task, the output of which is used to then generate new, personalized bundles. Overall we feel that these contributions, the relevance to SIGIR, and the technical/presentation quality of the paper warrant an acceptance. However there are a few areas that could be improved or clarified: 1) Related work: While the authors include a number of references to works on recommendations to groups of users, almost no work is cited for recommendations of groups of items (packages) to users, which is the task at hand. After a search through related literature we found some works on bundle/package recommendation that could be relevant: * Qi et al., Recommending packages to groups, ICDM 2016, DOI: 10.1109/ICDM.2016.0056 * Yuan M., Pavlidis Y., Jain M., Caster K. (2016) Walmart Online Grocery Personalization: Behavioral Insights and Basket Recommendations. In: Advances in Conceptual Modeling. ER 2016. Lecture Notes in Computer Science, vol 9975. Springer * Moran Beladev, Lior Rokach, Bracha Shapira, Recommender systems for product bundling, Knowledge-Based Systems, Volume 111, November 2016, Pages 193-206, http://dx.doi.org/10.1016/j.knosys.2016.08.013. * Min Xie, Laks V. S. Lakshmanan, and Peter T. Wood. 2014. Generating top-k packages via preference elicitation. Proc. VLDB Endow. 7, 14 (October 2014), 1941-1952. DOI=http://dx.doi.org/10.14778/2733085.2733099 * Tao Zhu, Patrick Harrington, Junjun Li, and Lei Tang. 2014. Bundle recommendation in e-commerce. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval (SIGIR '14). ACM, New York, NY, USA, 657-666. DOI=http://dx.doi.org/10.1145/2600428.2609603 We believe including some more related work from such relevant tasks will help place the paper in context and clarify its contribution compared to existing approaches. The above list is of course only indicative, there might very well be other work we have failed to include here, we only mean it as an example. 2) Some unclear parameter settings: The algorithms used in the paper are quite flexible and include a number of hyper-parameters that can significantly influence their performance. As such we feel the paper could be strengthened by including more parameter settings, and the motivation behind them. Examples include: the number of factors used in the matrix factorization model, the selection of k=10 neighbors for the greedy bundle generation algorithm, and the hyper-parameters used for t-SNE which can adversely affect its output [1]. These could be reported through cross-validated (CV) experiments for example on the left-out validation set. 3) Cross-validated evaluation: On the note of CV, the reported results could be strengthened by performing CV experiments to give the readers a sense of the sensitivity of the algorithm to the underlying train/test set. However, we understand that runtime can be an issue here. Minor notes: * Typos: * Introduction, first sentence: "goal of a Recommender System*s*" * Related Work, last sentence: "association rule *minig*" * Evaluation sections could be merged. Sections 5.2 and 6.2 could be merged and moved under Section 7, to improve the flow of the document. * Section 5.1: it is not mentioned that the the sigma function is the sigmoid. * Math notation can be clarified by using uppercase bold letters for matrices and lowercase bold for vectors. * Section 5.2: Has the $\hat{x}_{(u,p,q)}$ been introduced before? Maybe clarify that the delta corresponds to the Dirac function. * Section 6.1: (Limited reviewer knowledge) Is it guaranteed that the algorithm will always converge to a solution? It could be useful to include such an assertion if the convergence guarantee is not straight-forward. To conclude, we think the work presented is valuable and presented in a clear manner, with attention to detail and technical rigour. The promise to release the dataset is of importance to the community as well. We therefore recommend this paper to be accepted. inclusi References: [1] Wattenberg, et al., "How to Use t-SNE Effectively", Distill, 2016. Few more comments: please look at this paper, that is solving the problem of bundle recommendation in the case of a set of tweets, in section 3 Interests-Spanning TweetRec. https://pdfs.semanticscholar.org/5994/0eb2acea7bfe1b2debf8fec00a383ae04083.pdf ----------------------- REVIEW 3 --------------------- PAPER: 277 TITLE: Generating and Personalizing Bundle Recommendations on Steam AUTHORS: Apurva Pathak, Kshitiz Gupta and Julian McAuley ----------- Overall evaluation ----------- Pros: - Interesting problem - Method is straightforward and clear Cons: - Discount/manufacturer not taken into account when modeling the data ----------- Comments to the Author(s) ----------- This paper looks at the problem of recommending bundles of items (rather than single items) in a recommender system. The authors looked at bundle download data from a games website and found that users would purchase bundles containing items, some of which they already owned, which means that there must have been an economic incentive to buy bundles rather than placing individual items in the shopping cart. The authors ignored this discount issue in their modeling of the data/task, leaving its investigation to future work, which to me is a bit worrying since I think the data may be heavily influenced by it. I'm guessing a lot of the games bundles are put together because the games are made by the same developer, rather than because they are of the same genre (and therefore appeal to the same group of users), although the two aspects will be very correlated (games developers will make a series of games in the same genre). The authors take the standard Bayesian Personalised Ranking formulation of recommender systems and extend it in a plausible way to handle bundle predictions. Further comments: For the most part, the notation was nice and clear, although using the subscript 'q' to denote a negative example was a bit confusing! I'd suggest either 'n' or '-'. I liked the fact that the authors were careful in their method for sampling negative examples, although I was a bit confused what they meant exactly by "and bundles follow the same degree distribution". Degree of what? In the equation in Section 5.2, \hat{x}_{upq} should be \hat{x}_{ui_p} - \hat{x}_{ui_q} Figure 1 looks nice but I'm not sure what the take-home message is, except that the recommended and downloaded bundles are usually extreme / outlier products. Again Figure 2 looks nice but isn't very convincing.