Reviewer #1
Questions
1. {Summary} Please briefly summarize the main claims/contributions of the paper in your own words. (Please do not include your evaluation of the paper here).
This work reviews the metrics, benchmarks, and methods in model compression for NLP, organizing these papers in a new taxonomy. The survey includes weight sharing, low-rank factorization, pruning, quantization, knowledge distillation, and early exit.
2. {Strengths and Weaknesses} Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: novelty, quality, clarity, and significance.
Strengths
1. The paper structure is well and easy to follow.
2. The survey already covered almost techniques in language model compression.
3. The paper points out the future direction of the language model compression.
Weakness
Please see the question for the authors
3. {Questions for the Authors} Please carefully describe questions that you would like the authors to answer during the author feedback period. Think of the things where a response from the author may change your opinion, clarify a confusion or address a limitation. Please number your questions.
There also has another work can be consider to include to this survey: Huang, S., Xu, D., Yen, I. E., Chang, S. E., Li, B., Chen, S., ... & Ding, C. (2021). Sparse progressive distillation (SPD): Resolving overfitting under pretrain-and-finetune paradigm.
4. {Evaluation: Novelty} How novel are the concepts, problems addressed, or methods introduced in the paper?
Good: The paper makes non-trivial advances over the current state-of-the-art.
5. {Evaluation: Quality} Is the paper technically sound?
Good: The paper appears to be technically sound. The proofs, if applicable, appear to be correct, but I have not carefully checked the details. The experimental evaluation, if applicable, is adequate, and the results convincingly support the main claims.
6. {Evaluation: Significance} How do you rate the likely impact of the paper on the AI research community?
Good: The paper is likely to have high impact within a subfield of AI OR modest impact across more than one subfield of AI.
7. {Evaluation: Clarity} Is the paper well-organized and clearly written?
Good: The paper is well organized but the presentation has minor details that could be improved.
8. (Evaluation: Reproducibility) Are the results (e.g., theorems, experimental results) in the paper easily reproducible? (It may help to consult the paper’s reproducibility checklist.)checklist.)
Good: key resources (e.g., proofs, code, data) are available and sufficient details (e.g., proofs, experimental setup) are described such that an expert should be able to reproduce the main results.
9. {Evaluation: Resources} If applicable, how would you rate the new resources (code, data sets) the paper contributes? (It might help to consult the paper’s reproducibility checklist)
Good: The shared resources are likely to be very useful to other AI researchers.
10. {Evaluation: Ethical considerations} Does the paper adequately address the applicable ethical considerations, e.g., responsible data collection and use (e.g., informed consent, privacy), possible societal harm (e.g., exacerbating injustice or discrimination due to algorithmic bias), etc.?
Good: The paper adequately addresses most, but not all, of the applicable ethical considerations.
11. (OVERALL EVALUATION) Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.
Weak Accept: Technically solid, modest-to-high impact paper, with no major concerns with respect to quality, reproducibility, and if applicable, resources, ethical considerations.
13. (CONFIDENCE) How confident are you in your evaluation?
Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
14. (EXPERTISE) How well does this paper align with your expertise?
Very Knowledgeable: This paper significantly overlaps with my current work and I am very knowledgeable about most of the topics covered by the paper.
Reviewer #4
Questions
1. {Summary} Please briefly summarize the main claims/contributions of the paper in your own words. (Please do not include your evaluation of the paper here).
This paper is a survey of model compression technique for pretrained language models, including weight sharing, low-rank factorization, pruning, quantization, knowledge distillation, and early exit. It summarized the recent works in compressing pretrained language models. Overall, this paper has its merit.
2. {Strengths and Weaknesses} Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: novelty, quality, clarity, and significance.
Strength:
1. The paper is easy to follow.
2. Many benchamrks, metrics, and methods are discussed in this paper.

Weakness:
1. In introduction, authors discussed some previous survey works. What is the difference of this survey and the previous surveys? The previous survey works are published in 2020 or 2021. Are there many works in this field in rencent years? It is better to explain the major contribution of this paper compared with the previous publised surveys in details.
3. {Questions for the Authors} Please carefully describe questions that you would like the authors to answer during the author feedback period. Think of the things where a response from the author may change your opinion, clarify a confusion or address a limitation. Please number your questions.
1. The authors states the number of parameters and model size are the indicator of the memory footprint. To the best of my knowledge, memory footprint is decided by the size of activation in the network. Can the authors provide some evidence that the number of parameters is related to the memory footprint?
4. {Evaluation: Novelty} How novel are the concepts, problems addressed, or methods introduced in the paper?
Fair: The paper contributes some new ideas or represents incremental advances.
5. {Evaluation: Quality} Is the paper technically sound?
Good: The paper appears to be technically sound. The proofs, if applicable, appear to be correct, but I have not carefully checked the details. The experimental evaluation, if applicable, is adequate, and the results convincingly support the main claims.
6. {Evaluation: Significance} How do you rate the likely impact of the paper on the AI research community?
Good: The paper is likely to have high impact within a subfield of AI OR modest impact across more than one subfield of AI.
7. {Evaluation: Clarity} Is the paper well-organized and clearly written?
Excellent: The paper is well-organized and clearly written.
8. (Evaluation: Reproducibility) Are the results (e.g., theorems, experimental results) in the paper easily reproducible? (It may help to consult the paper’s reproducibility checklist.)checklist.)
Fair: key resources (e.g., proofs, code, data) are unavailable and/or some key details (e.g., proof sketches, experimental setup) are unavailable which make it difficult to reproduce the main results.
9. {Evaluation: Resources} If applicable, how would you rate the new resources (code, data sets) the paper contributes? (It might help to consult the paper’s reproducibility checklist)
Fair: The shared resources are likely to be of some use to other AI researchers.
10. {Evaluation: Ethical considerations} Does the paper adequately address the applicable ethical considerations, e.g., responsible data collection and use (e.g., informed consent, privacy), possible societal harm (e.g., exacerbating injustice or discrimination due to algorithmic bias), etc.?
Not Applicable: The paper does not have any ethical considerations to address.
11. (OVERALL EVALUATION) Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.
Borderline accept: Technically solid paper where reasons to accept, e.g., good novelty, outweigh reasons to reject, e.g., fair quality. Please use sparingly.
13. (CONFIDENCE) How confident are you in your evaluation?
Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation.
14. (EXPERTISE) How well does this paper align with your expertise?
Expert: This paper is within my current core research focus and I am deeply knowledgeable about all of the topics covered by the paper.
Reviewer #6
Questions
1. {Summary} Please briefly summarize the main claims/contributions of the paper in your own words. (Please do not include your evaluation of the paper here).
The work compiles a summary of the current state of model compression for NLP models. It focuses on the NLP pre-trained models that increase efficiency of the inference stage and deployment.

The paper divides methods into several common groups: weight sharing, Low-rank factorization, Pruning, Quantization, Knowledge Distillation, Early Exit
The paper focuses on transformer model but mentions also some techniques for LSTM
2. {Strengths and Weaknesses} Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: novelty, quality, clarity, and significance.
The topic is relevant since currently NLP much bigger than vision models, however there is relatively less research on NLP model compression. The presentation is neat, introducing the topic and then listing the relevant work. The paper systematizes metrics and benchmarks, which is lacking in many review papers.

The visual aids are very helpful and there could be more of them.

But it also feels to me that the paper is an extended “Related Work” section, that is it provides a dry summary of the current literature. A sentence for each paper. Perhaps it’s be worth to go into more detail for some of the more relevant works.

For pruning, I would not say “Lottery Ticket Hypothesis” is a separate sub-field. It’s a prominent method within unstructured/structured pruning.

I am not sure what is the policy of AAAI on publishing such sort of literature review manuscripts but I think they’re helpful for research community to get a better understanding of the field.
3. {Questions for the Authors} Please carefully describe questions that you would like the authors to answer during the author feedback period. Think of the things where a response from the author may change your opinion, clarify a confusion or address a limitation. Please number your questions.
How did you choose the relevant work to include in this paper?
Have you considered an extended version of this papers?
4. {Evaluation: Novelty} How novel are the concepts, problems addressed, or methods introduced in the paper?
Fair: The paper contributes some new ideas or represents incremental advances.
5. {Evaluation: Quality} Is the paper technically sound?
Good: The paper appears to be technically sound. The proofs, if applicable, appear to be correct, but I have not carefully checked the details. The experimental evaluation, if applicable, is adequate, and the results convincingly support the main claims.
6. {Evaluation: Significance} How do you rate the likely impact of the paper on the AI research community?
Fair: The paper is likely to have modest impact within a subfield of AI.
7. {Evaluation: Clarity} Is the paper well-organized and clearly written?
Excellent: The paper is well-organized and clearly written.
8. (Evaluation: Reproducibility) Are the results (e.g., theorems, experimental results) in the paper easily reproducible? (It may help to consult the paper’s reproducibility checklist.)checklist.)
Good: key resources (e.g., proofs, code, data) are available and sufficient details (e.g., proofs, experimental setup) are described such that an expert should be able to reproduce the main results.
9. {Evaluation: Resources} If applicable, how would you rate the new resources (code, data sets) the paper contributes? (It might help to consult the paper’s reproducibility checklist)
Not applicable: For instance, the primary contributions of the paper are theoretical.
10. {Evaluation: Ethical considerations} Does the paper adequately address the applicable ethical considerations, e.g., responsible data collection and use (e.g., informed consent, privacy), possible societal harm (e.g., exacerbating injustice or discrimination due to algorithmic bias), etc.?
Not Applicable: The paper does not have any ethical considerations to address.
11. (OVERALL EVALUATION) Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.
Weak Accept: Technically solid, modest-to-high impact paper, with no major concerns with respect to quality, reproducibility, and if applicable, resources, ethical considerations.
13. (CONFIDENCE) How confident are you in your evaluation?
Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
14. (EXPERTISE) How well does this paper align with your expertise?
Knowledgeable: This paper has some overlap with my current work. My recent work was focused on closely related topics and I am knowledgeable about most of the topics covered by the paper.
Reviewer #7
Questions
1. {Summary} Please briefly summarize the main claims/contributions of the paper in your own words. (Please do not include your evaluation of the paper here).
This paper is (similar to what the title would suggest) a summary of recent work on compressing (massive) pretrained language models. It covers aspects such as metrics that people might care about, techniques such as pruning and quantization, and very briefly discusses future directions. Unlike what the title would suggest, I would not call this a "survey", as it does not explicitly survey the literature; rather, it summarizes recent work on this topic.
2. {Strengths and Weaknesses} Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: novelty, quality, clarity, and significance.
The strength of this paper is that is appears to be a reasonably comprehensive summary of recent work on this topic. It is also well written, and generally easy to follow, although if the authors had more space, it would be good to preview or map out the organization / structure of the paper slightly more in the introduction. I do not have sufficient expertise to say definitively how complete it is, but I would nevertheless assess it as high quality.

The main weakness is that although the paper calls itself as a survey, it does not follow the standard methodology of a survey, which would be to use a procedure to identify a set of relevant papers. (Or rather, if the authors did follow such a methodology, they did not describe it here). The advantage of this is that the authors are able to focus on whatever they find most relevant. The disadvantage is that the scope can appear somewhat more arbitrary, and there may be questions about completeness. I am also not sure about how well this sort of work fits into the typical scope of AAAI specifically.
3. {Questions for the Authors} Please carefully describe questions that you would like the authors to answer during the author feedback period. Think of the things where a response from the author may change your opinion, clarify a confusion or address a limitation. Please number your questions.
How did you decide what works should be included for discussion in this paper?

4. {Evaluation: Novelty} How novel are the concepts, problems addressed, or methods introduced in the paper?
Fair: The paper contributes some new ideas or represents incremental advances.
5. {Evaluation: Quality} Is the paper technically sound?
Good: The paper appears to be technically sound. The proofs, if applicable, appear to be correct, but I have not carefully checked the details. The experimental evaluation, if applicable, is adequate, and the results convincingly support the main claims.
6. {Evaluation: Significance} How do you rate the likely impact of the paper on the AI research community?
Fair: The paper is likely to have modest impact within a subfield of AI.
7. {Evaluation: Clarity} Is the paper well-organized and clearly written?
Good: The paper is well organized but the presentation has minor details that could be improved.
8. (Evaluation: Reproducibility) Are the results (e.g., theorems, experimental results) in the paper easily reproducible? (It may help to consult the paper’s reproducibility checklist.)checklist.)
Poor: key details (e.g., proof sketches, experimental setup) are incomplete/unclear, or key resources (e.g., proofs, code, data) are unavailable.
9. {Evaluation: Resources} If applicable, how would you rate the new resources (code, data sets) the paper contributes? (It might help to consult the paper’s reproducibility checklist)
Not applicable: For instance, the primary contributions of the paper are theoretical.
10. {Evaluation: Ethical considerations} Does the paper adequately address the applicable ethical considerations, e.g., responsible data collection and use (e.g., informed consent, privacy), possible societal harm (e.g., exacerbating injustice or discrimination due to algorithmic bias), etc.?
Good: The paper adequately addresses most, but not all, of the applicable ethical considerations.
11. (OVERALL EVALUATION) Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.
Weak Accept: Technically solid, modest-to-high impact paper, with no major concerns with respect to quality, reproducibility, and if applicable, resources, ethical considerations.
13. (CONFIDENCE) How confident are you in your evaluation?
Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
14. (EXPERTISE) How well does this paper align with your expertise?
Somewhat Knowledgeable: This paper has little overlap with my current work. I am somewhat knowledgeable about some of the topics covered by the paper.
16. I acknowledge that I have read the author's rebuttal (if applicable) and made changes to my review as needed.
Agreement accepted