Who’s Ahead in AI Research at NeurIPS 2020? Insights and AI Research Rankings at the Leading AI Conference
Please subscribe to our Twitter if you’d like to follow our Deep Tech research. 🤖
Introduction
The Conference on Neural Information Processing Systems (NeurIPS) is one of the most prestigious AI research conferences (the other one is the International Conference on Machine Learning, or ICML). In 2020, the acceptance rate at NeurIPS was 20% — a total of 1,990 papers out of 9,467 submissions got in (source). Compared to 2019, the number of submissions increased by 40%, which is similar to the growth from 2018 to 2019. Using conference proceedings (NeurIPS 2020), we compiled a list of authors and their affiliated organizations and then calculated the Publication Index for each organization (see “Methodology” section below). The most intuitive way to think of the Publication Index is from the point of view of full paper equivalents: Google’s Publication Index of 128 can be interpreted as if Google published 128 full papers at NeurIPS 2020.
We will start this analysis with details on methodology, continue on to AI research rankings at NeurIPS 2020, then show further interesting descriptive statistics, discuss the changes between NeurIPS 2019 and NeurIPS 2020, and finally conclude with a link to the dataset.
Methodology
The methodology of our Publication Index is inspired by the Nature Index:
To glean a country’s, a region’s or an institution’s contribution to an article, and to ensure they are not counted more than once, the Nature Index uses fractional count (FC), which takes into account the share of authorship on each article. The total FC available per article is 1, which is shared among all authors under the assumption that each contributed equally. For instance, an article with 10 authors means that each author receives an FC of 0.1. For authors who are affiliated with more than one institution, the author’s FC is then split equally between each institution. The total FC for an institution is calculated by summing the FC for individual affiliated authors. The process is similar for countries/regions, although complicated by the fact that some institutions have overseas labs that will be counted towards host country/region totals.
The only difference is that our Publication Index counts overseas labs towards the headquarters country/region (instead of the host country/region). This is a contentious point, but we believe that this approach better reflects the assignment of intellectual property and respective accrual of benefit to the headquarters, rather than the local lab.
Here is an example of the Publication Index calculation. If a paper has five authors — three from MIT, one from the University of Oxford, and one from Google — each author will get 1/5th of one point, or 0.2. As a result, from this paper alone, MIT will increase its Publication Index by 3*0.2=0.6 points, the University of Oxford will increase its index by 0.2, and Google will add 0.2. Since MIT is based in the United States, MIT affiliation will increase the Publication Index of the United States by 0.6. Similarly, since the University of Oxford is based in the UK, the EEA + Switzerland category will increase by 0.2. Finally, Google is a multinational corporation headquartered in the United States, therefore the United States will increase its Publication Index by an additional 0.2, for a total increase of 0.8. If an author has multiple affiliations, we split his/her fraction across each of those affiliated institutions. For instance, in the case above, if the last author listed two affiliations, Google and Stanford University (instead of just Google), both Google and Stanford University would get additional 0.2/2=0.1 points.
Who’s Ahead in AI Research at NeurIPS 2020?
Top 100 Global Organizations (Industry & Academia) Leading in AI Research at NeurIPS 2020 (with Publication Indices):
1. Google (USA) — 128.0
2. Stanford University (USA) — 67.0
3. MIT (USA) — 61.1
4. UC Berkeley (USA) — 52.4
5. Carnegie Mellon University (USA) — 47.3
6. Microsoft (USA) — 42.9
7. University of Oxford (UK) — 35.5
8. Tsinghua University (China) — 34.5
9. Facebook (USA) — 31.4
10. Princeton University (USA) — 28.0
11. ETH (Switzerland) — 26.6
12. New York University (USA) — 26.1
13. UT Austin (USA) — 25.7
14. Columbia University (USA) — 25.5
15. KAIST (South Korea) — 23.8
16. University of Illinois at Urbana-Champaign (USA) — 23.6
17. Cornell University (USA) — 23.0
18. EPFL (Switzerland) — 22.6
19. Harvard University (USA) — 22.3
20. University of Cambridge (UK) — 21.6
21. IBM (USA) — 19.0
22. UCLA (USA) — 18.7
23. UC San Diego (USA) — 18.3
24. Peking University (China) — 18.1
25. University College London (UK) — 15.5
26. Georgia Institute of Technology (USA) — 15.1
27. University of Pennsylvania (USA) — 14.3
28. University of Maryland (USA) — 13.8
29. University of Michigan (USA) — 13.6
30. University of Toronto (Canada) — 13.2
31. National University of Singapore (Singapore) — 12.2
32. NVIDIA (USA) — 11.9
33. Purdue University (USA) — 11.8
34. Huawei (China) — 11.7
35. University of Washington (USA) — 11.5
36. INRIA (France) — 11.1
37. Duke University (USA) — 10.7
38. Boston University (USA) — 10.3
39. UMass Amherst (USA) — 9.9
40. University of Southern California (USA) — 9.7
41. Tel Aviv University (Israel) — 9.4
42. Imperial College London (UK) — 9.1
43. Shanghai Jiao Tong University (China) — 9.1
44. University of Tokyo (Japan) — 8.8
45. Seoul National University (South Korea) — 8.8
46. California Institute of Technology (USA) — 8.8
47. University of Science and Technology of China (China) — 8.4
48. Amazon (USA) — 8.3
49. Tencent (China) — 8.3
50. Northeastern University (USA) — 8.1
51. University of Wisconsin-Madison (USA) — 8.0
52. University of Sydney (Australia) — 8.0
53. Chinese University of Hong Kong (China) — 8.0
54. Technion (Israel) — 7.8
55. TU Munich (Germany) — 7.8
56. Texas A&M University (USA) — 7.7
57. Johns Hopkins University (USA) — 7.6
58. Alibaba (China) — 7.6
59. Mila (Canada) — 7.5
60. University of Minnesota (USA) — 7.3
61. MPI Intelligent Systems (Germany) — 7.2
62. Samsung (South Korea) — 6.9
63. McGill University (Canada) — 6.8
64. University of British Columbia (Canada) — 6.8
65. Hong Kong University of Science and Technology (China) — 6.8
66. Nanjing University (China) — 6.6
67. University of Chicago (USA) — 6.5
68. UC Santa Barbara (USA) — 6.4
69. University of Tuebingen (Germany) — 6.4
70. Rutgers University (USA) — 6.3
71. University of Edinburgh (UK) — 6.3
72. ENS Paris (France) — 6.2
73. University of Technology Sydney (Australia) — 6.0
74. KAUST (Saudi Arabia) — 5.8
75. RIKEN (Japan) — 5.7
76. Nanyang Technological University (China) — 5.5
77. Toyota Technological Institute at Chicago (USA) — 5.4
78. UC Davis (USA) — 5.3
79. CNRS (France) — 5.3
80. Ohio State University (USA) — 5.3
81. Rice University (USA) — 5.2
82. Zhejiang University (China) — 5.1
83. Northwestern University (USA) — 4.9
84. Indiana University (USA) — 4.9
85. University of Alberta (Canada) — 4.7
86. University of Amsterdam (Netherlands) — 4.7
87. Intel (USA) — 4.5
88. MPI Informatics (Germany) — 4.5
89. SUNY Stony Brook (USA) — 4.5
90. Hebrew University (Israel) — 4.5
91. UC Irvine (USA) — 4.4
92. Weizmann Institute (Israel) — 4.4
93. University of Montreal (Canada) — 4.4
94. Aalto University (Finland) — 4.4
95. Ecole Polytechnique (France) — 4.2
96. Adobe (USA) — 4.1
97. Xidian University (China) — 4.1
98. Australian National University (Australia) — 4.0
99. Salesforce (USA) — 4.0
100. KTH Royal Institute of Technology (Sweden) — 3.8
Top 40 American Universities Leading in AI Research at NeurIPS 2020 (with Publication Indices):
1. Stanford University — 67.0
2. MIT — 61.1
3. UC Berkeley — 52.4
4. Carnegie Mellon University — 47.3
5. Princeton University — 28.0
6. New York University — 26.1
7. UT Austin — 25.7
8. Columbia University — 25.5
9. University of Illinois at Urbana-Champaign — 23.6
10. Cornell University — 23.0
11. Harvard University — 22.3
12. UCLA — 18.7
13. UC San Diego — 18.3
14. Georgia Institute of Technology — 15.1
15. University of Pennsylvania — 14.3
16. University of Maryland — 13.8
17. University of Michigan — 13.6
18. Purdue University — 11.8
19. University of Washington — 11.5
20. Duke University — 10.7
21. Boston University — 10.3
22. UMass Amherst — 9.9
23. University of Southern California — 9.7
24. California Institute of Technology — 8.8
25. Northeastern University — 8.1
26. University of Wisconsin-Madison — 8.0
27. Texas A&M University — 7.7
28. Johns Hopkins University — 7.6
29. University of Minnesota — 7.3
30. University of Chicago — 6.5
31. UC Santa Barbara — 6.4
32. Rutgers University — 6.3
33. Toyota Technological Institute at Chicago — 5.4
34. UC Davis — 5.3
35. Ohio State University — 5.3
36. Rice University — 5.2
37. Northwestern University — 4.9
38. Indiana University — 4.9
39. SUNY Stony Brook — 4.5
40. UC Irvine — 4.4
Top 40 Global Universities Leading in AI Research at NeurIPS 2020 (with Publication Indices):
1. Stanford University (USA) — 67.0
2. MIT (USA) — 61.1
3. UC Berkeley (USA) — 52.4
4. Carnegie Mellon University (USA) — 47.3
5. University of Oxford (UK) — 35.5
6. Tsinghua University (China) — 34.5
7. Princeton University (USA) — 28.0
8. ETH (Switzerland) — 26.6
9. New York University (USA) — 26.1
10. UT Austin (USA) — 25.7
11. Columbia University (USA) — 25.5
12. KAIST (South Korea) — 23.8
13. University of Illinois at Urbana-Champaign (USA) — 23.6
14. Cornell University (USA) — 23.0
15. EPFL (Switzerland) — 22.6
16. Harvard University (USA) — 22.3
17. University of Cambridge (UK) — 21.6
18. UCLA (USA) — 18.7
19. UC San Diego (USA) — 18.3
20. Peking University (China) — 18.1
21. University College London (UK) — 15.5
22. Georgia Institute of Technology (USA) — 15.1
23. University of Pennsylvania (USA) — 14.3
24. University of Maryland (USA) — 13.8
25. University of Michigan (USA) — 13.6
26. University of Toronto (Canada) — 13.2
27. National University of Singapore (Singapore) — 12.2
28. Purdue University (USA) — 11.8
29. University of Washington (USA) — 11.5
30. INRIA (France) — 11.1
31. Duke University (USA) — 10.7
32. Boston University (USA) — 10.3
33. UMass Amherst (USA) — 9.9
34. University of Southern California (USA) — 9.7
35. Tel Aviv University (Israel) — 9.4
36. Imperial College London (UK) — 9.1
37. Shanghai Jiao Tong University (China) — 9.1
38. University of Tokyo (Japan) — 8.8
39. Seoul National University (South Korea) — 8.8
40. California Institute of Technology (USA) — 8.8
Top 20 Global Companies Leading in AI Research at NeurIPS 2020 (with Publication Indices):
1. Google (USA) — 128.0
2. Microsoft (USA) — 42.9
3. Facebook (USA) — 31.4
4. IBM (USA) — 19.0
5. NVIDIA (USA) — 11.9
6. Huawei (China) — 11.7
7. Amazon (USA) — 8.3
8. Tencent (China) — 8.3
9. Alibaba (China) — 7.6
10. Samsung (South Korea) — 6.9
11. Intel (USA) — 4.5
12. Adobe (USA) — 4.1
13. Salesforce (USA) — 4.0
14. Apple (USA) — 3.7
15. Qualcomm (USA) — 3.5
16. Bosch (Germany) — 3.5
17. Baidu (China) — 3.4
18. SenseTime (China) — 3.0
19. OpenAI (USA) — 2.9
20. Criteo (France) — 2.7
Further Analysis
Word Cloud of Paper Titles at NeurIPS 2020:
Changes in rank at Top 10 Global Organizations at NeurIPS 2020 vs. NeurIPS 2019:
Google: stayed at 1st place
Stanford University: stayed at 2nd place
MIT: improved from 4th to 3rd place
UC Berkeley: improved from 6th to 4th place
Carnegie Mellon University: dropped from 3rd to 5th place
Microsoft: dropped from 5th to 6th place
University of Oxford: stayed at 7th place
Tsinghua University: improved significantly from 13th to 8th place
Facebook: dropped from 8th to 9th place
Princeton University: stayed at 10th place
Changes in Publication Indices at Top 50 Global Organizations at NeurIPS 2020 vs. NeurIPS 2019 (positive change means more publications at NeurIPS 2020 than at NeurIPS 2019):
1. Google (USA): up 33.5 (35%) — from 94.5 to 128.0
2. Stanford University (USA): up 9.1 (16%) — from 57.8 to 67.0
3. MIT (USA): up 14.4 (31%) — from 46.7 to 61.1
4. UC Berkeley (USA): up 22.8 (77%) — from 29.7 to 52.4
5. Carnegie Mellon University (USA): down -1.2 (-3%) — from 48.5 to 47.3
6. Microsoft (USA): up 7.8 (22%) — from 35.1 to 42.9
7. University of Oxford (UK): up 11.9 (51%) — from 23.6 to 35.5
8. Tsinghua University (China): up 15.6 (82%) — from 18.9 to 34.5
9. Facebook (USA): up 7.8 (33%) — from 23.6 to 31.4
10. Princeton University (USA): up 7.2 (35%) — from 20.7 to 28.0
11. ETH (Switzerland): up 11.7 (78%) — from 14.9 to 26.6
12. New York University (USA): up 12.1 (86%) — from 14.0 to 26.1
13. UT Austin (USA): up 6.9 (36%) — from 18.9 to 25.7
14. Columbia University (USA): up 2.2 (10%) — from 23.3 to 25.5
15. KAIST (South Korea): up 20.0 (525%) — from 3.8 to 23.8
16. University of Illinois at Urbana-Champaign (USA): up 3.7 (18%) — from 20.0 to 23.6
17. Cornell University (USA): up 2.7 (13%) — from 20.3 to 23.0
18. EPFL (Switzerland): up 10.0 (79%) — from 12.6 to 22.6
19. Harvard University (USA): up 10.1 (82%) — from 12.2 to 22.3
20. University of Cambridge (UK): up 12.5 (138%) — from 9.1 to 21.6
21. IBM (USA): up 3.6 (23%) — from 15.4 to 19.0
22. UCLA (USA): down -0.1 (-1%) — from 18.8 to 18.7
23. UC San Diego (USA): up 6 (48%) — from 12.3 to 18.3
24. Peking University (China): up 2.4 (15%) — from 15.7 to 18.1
25. University College London (UK): up 6.0 (63%) — from 9.5 to 15.5
26. Georgia Institute of Technology (USA): down -0.5 (-3%) — from 15.6 to 15.1
27. University of Pennsylvania (USA): up 5.1 (56%) — from 9.2 to 14.3
28. University of Maryland (USA): up 7.5 (119%) — from 6.3 to 13.8
29. University of Michigan (USA): up 6.9 (103%) — from 6.7 to 13.6
30. University of Toronto (Canada): down -1.7 (-11%) — from 14.9 to 13.2
31. National University of Singapore (Singapore): up 6.8 (128%) — from 5.4 to 12.2
32. NVIDIA (USA): up 7.4 (163%) — from 4.5 to 11.9
33. Purdue University (USA): up 4.1 (52%) — from 7.7 to 11.8
34. Huawei (China): up 9.2 (381%) — from 2.4 to 11.7
35. University of Washington (USA): down -3.4 (-23%) — from 15.0 to 11.5
36. INRIA (France): down -5.6 (-34%) — from 16.7 to 11.1
37. Duke University (USA): down -0.8 (-7%) — from 11.5 to 10.7
38. Boston University (USA): up 5.6 (119%) — from 4.7 to 10.3
39. UMass Amherst (USA): up 0.3 (3%) — from 9.6 to 9.9
40. University of Southern California (USA): up 0.6 (6%) — from 9.1 to 9.7
41. Tel Aviv University (Israel): up 6.2 (193%) — from 3.2 to 9.4
42. Imperial College London (UK): up 3.1 (53%) — from 5.9 to 9.1
43. Shanghai Jiao Tong University (China): up 7.3 (409%) — from 1.8 to 9.1
44. University of Tokyo (Japan): up 4.7 (114%) — from 4.1 to 8.8
45. Seoul National University (South Korea): up 3.5 (66%) — from 5.3 to 8.8
46. California Institute of Technology (USA): up 0.8 (10%) — from 8.0 to 8.8
47. University of Science and Technology of China (China): up 4.5 (118%) — from 3.8 to 8.4
48. Amazon (USA): down -2.8 (-25%) — from 11.1 to 8.3
49. Tencent (China): up 3 (57%) — from 5.3 to 8.3
50. Northeastern University (USA): up 0.8 (11%) — from 7.3 to 8.1
Discussion
Let’s see what changed in the top 10 rankings between NeurIPS 2019 and NeurIPS 2020 (for 2019 data please see our AI Research Rankings 2019, where we combined insights from NeurIPS 2019 and ICML 2019).
The top two positions, Google and Stanford University, remained the same. MIT moved up from 4th to 3rd place. UC Berkeley moved up from 6th to 4th place. Carnegie Mellon dropped from 3rd to 5th place. Microsoft dropped from 5th to 6th place. The University of Oxford stayed at 7th place. Tsinghua University jumped from 13th to 8th place. Facebook dropped from 8th to 9th place. Princeton stayed at 10th place.
To move up in rankings, each organization had to significantly increase their Publication Index: Google published an equivalent of 33.5 more papers, Stanford University is up by 9.1, MIT is up by 14.4, UC Berkeley is up by 22.8, etc. (see table above). Just like in Lewis Carroll’s Red Queen’s race, top organizations need to publish more papers each year just to maintain the lead.
“My dear, here we must run as fast as we can, just to stay in place. And if you wish to go anywhere you must run twice as fast as that.” (Lewis Carroll)
Dataset
Please note that even data science conferences still don’t release publication data in any sort of Python-friendly form 🤷♂️, so our analysis ended up being quite manual (i.e. first parse HTML, then fix typos in organization names, standardize them, split lines with multiple organizations, summarize with a pivot table, etc.). If you find any bugs, please email us, and we’ll be happy to fix them.
About the author: My name is Gleb Chuvpilo, and I’m the Managing Partner at Thundermark Capital, a Venture Capital firm that invests in Deep Tech startups. I have a Master’s degree from the MIT Computer Science and Artificial Intelligence Lab and an MBA in Finance and Strategic Management from The Wharton School at the University of Pennsylvania.
