Who’s Ahead in AI Research at NeurIPS 2020? Insights and AI Research Rankings at the Leading AI Conference

UPDATE: We have published an update to this analysis for 2020 here. Enjoy!

Introduction

We will start this analysis with details on methodology, continue on to AI research rankings at NeurIPS 2020, then show further interesting descriptive statistics, discuss the changes between NeurIPS 2019 and NeurIPS 2020, and finally conclude with a link to the dataset.

Methodology

To glean a country’s, a region’s or an institution’s contribution to an article, and to ensure they are not counted more than once, the Nature Index uses fractional count (FC), which takes into account the share of authorship on each article. The total FC available per article is 1, which is shared among all authors under the assumption that each contributed equally. For instance, an article with 10 authors means that each author receives an FC of 0.1. For authors who are affiliated with more than one institution, the author’s FC is then split equally between each institution. The total FC for an institution is calculated by summing the FC for individual affiliated authors. The process is similar for countries/regions, although complicated by the fact that some institutions have overseas labs that will be counted towards host country/region totals.

The only difference is that our Publication Index counts overseas labs towards the headquarters country/region (instead of the host country/region). This is a contentious point, but we believe that this approach better reflects the assignment of intellectual property and respective accrual of benefit to the headquarters, rather than the local lab.

Here is an example of the Publication Index calculation. If a paper has five authors — three from MIT, one from the University of Oxford, and one from Google — each author will get 1/5th of one point, or 0.2. As a result, from this paper alone, MIT will increase its Publication Index by 3*0.2=0.6 points, the University of Oxford will increase its index by 0.2, and Google will add 0.2. Since MIT is based in the United States, MIT affiliation will increase the Publication Index of the United States by 0.6. Similarly, since the University of Oxford is based in the UK, the EEA + Switzerland category will increase by 0.2. Finally, Google is a multinational corporation headquartered in the United States, therefore the United States will increase its Publication Index by an additional 0.2, for a total increase of 0.8. If an author has multiple affiliations, we split his/her fraction across each of those affiliated institutions. For instance, in the case above, if the last author listed two affiliations, Google and Stanford University (instead of just Google), both Google and Stanford University would get additional 0.2/2=0.1 points.

Who’s Ahead in AI Research at NeurIPS 2020?

Top 100 Global Organizations (Industry & Academia) Leading in AI Research at NeurIPS 2020

1. Google (USA) — 128.0
2. Stanford University (USA) — 67.0
3. MIT (USA) — 61.1
4. UC Berkeley (USA) — 52.4
5. Carnegie Mellon University (USA) — 47.3
6. Microsoft (USA) — 42.9
7. University of Oxford (UK) — 35.5
8. Tsinghua University (China) — 34.5
9. Facebook (USA) — 31.4
10. Princeton University (USA) — 28.0
11. ETH (Switzerland) — 26.6
12. New York University (USA) — 26.1
13. UT Austin (USA) — 25.7
14. Columbia University (USA) — 25.5
15. KAIST (South Korea) — 23.8
16. University of Illinois at Urbana-Champaign (USA) — 23.6
17. Cornell University (USA) — 23.0
18. EPFL (Switzerland) — 22.6
19. Harvard University (USA) — 22.3
20. University of Cambridge (UK) — 21.6
21. IBM (USA) — 19.0
22. UCLA (USA) — 18.7
23. UC San Diego (USA) — 18.3
24. Peking University (China) — 18.1
25. University College London (UK) — 15.5
26. Georgia Institute of Technology (USA) — 15.1
27. University of Pennsylvania (USA) — 14.3
28. University of Maryland (USA) — 13.8
29. University of Michigan (USA) — 13.6
30. University of Toronto (Canada) — 13.2
31. National University of Singapore (Singapore) — 12.2
32. NVIDIA (USA) — 11.9
33. Purdue University (USA) — 11.8
34. Huawei (China) — 11.7
35. University of Washington (USA) — 11.5
36. INRIA (France) — 11.1
37. Duke University (USA) — 10.7
38. Boston University (USA) — 10.3
39. UMass Amherst (USA) — 9.9
40. University of Southern California (USA) — 9.7
41. Tel Aviv University (Israel) — 9.4
42. Imperial College London (UK) — 9.1
43. Shanghai Jiao Tong University (China) — 9.1
44. University of Tokyo (Japan) — 8.8
45. Seoul National University (South Korea) — 8.8
46. California Institute of Technology (USA) — 8.8
47. University of Science and Technology of China (China) — 8.4
48. Amazon (USA) — 8.3
49. Tencent (China) — 8.3
50. Northeastern University (USA) — 8.1
51. University of Wisconsin-Madison (USA) — 8.0
52. University of Sydney (Australia) — 8.0
53. Chinese University of Hong Kong (China) — 8.0
54. Technion (Israel) — 7.8
55. TU Munich (Germany) — 7.8
56. Texas A&M University (USA) — 7.7
57. Johns Hopkins University (USA) — 7.6
58. Alibaba (China) — 7.6
59. Mila (Canada) — 7.5
60. University of Minnesota (USA) — 7.3
61. MPI Intelligent Systems (Germany) — 7.2
62. Samsung (South Korea) — 6.9
63. McGill University (Canada) — 6.8
64. University of British Columbia (Canada) — 6.8
65. Hong Kong University of Science and Technology (China) — 6.8
66. Nanjing University (China) — 6.6
67. University of Chicago (USA) — 6.5
68. UC Santa Barbara (USA) — 6.4
69. University of Tuebingen (Germany) — 6.4
70. Rutgers University (USA) — 6.3
71. University of Edinburgh (UK) — 6.3
72. ENS Paris (France) — 6.2
73. University of Technology Sydney (Australia) — 6.0
74. KAUST (Saudi Arabia) — 5.8
75. RIKEN (Japan) — 5.7
76. Nanyang Technological University (China) — 5.5
77. Toyota Technological Institute at Chicago (USA) — 5.4
78. UC Davis (USA) — 5.3
79. CNRS (France) — 5.3
80. Ohio State University (USA) — 5.3
81. Rice University (USA) — 5.2
82. Zhejiang University (China) — 5.1
83. Northwestern University (USA) — 4.9
84. Indiana University (USA) — 4.9
85. University of Alberta (Canada) — 4.7
86. University of Amsterdam (Netherlands) — 4.7
87. Intel (USA) — 4.5
88. MPI Informatics (Germany) — 4.5
89. SUNY Stony Brook (USA) — 4.5
90. Hebrew University (Israel) — 4.5
91. UC Irvine (USA) — 4.4
92. Weizmann Institute (Israel) — 4.4
93. University of Montreal (Canada) — 4.4
94. Aalto University (Finland) — 4.4
95. Ecole Polytechnique (France) — 4.2
96. Adobe (USA) — 4.1
97. Xidian University (China) — 4.1
98. Australian National University (Australia) — 4.0
99. Salesforce (USA) — 4.0
100. KTH Royal Institute of Technology (Sweden) — 3.8

Top 40 American Universities Leading in AI Research at NeurIPS 2020 (with Publication Indices):

Top 40 American Universities Leading in AI Research at NeurIPS 2020

1. Stanford University — 67.0
2. MIT — 61.1
3. UC Berkeley — 52.4
4. Carnegie Mellon University — 47.3
5. Princeton University — 28.0
6. New York University — 26.1
7. UT Austin — 25.7
8. Columbia University — 25.5
9. University of Illinois at Urbana-Champaign — 23.6
10. Cornell University — 23.0
11. Harvard University — 22.3
12. UCLA — 18.7
13. UC San Diego — 18.3
14. Georgia Institute of Technology — 15.1
15. University of Pennsylvania — 14.3
16. University of Maryland — 13.8
17. University of Michigan — 13.6
18. Purdue University — 11.8
19. University of Washington — 11.5
20. Duke University — 10.7
21. Boston University — 10.3
22. UMass Amherst — 9.9
23. University of Southern California — 9.7
24. California Institute of Technology — 8.8
25. Northeastern University — 8.1
26. University of Wisconsin-Madison — 8.0
27. Texas A&M University — 7.7
28. Johns Hopkins University — 7.6
29. University of Minnesota — 7.3
30. University of Chicago — 6.5
31. UC Santa Barbara — 6.4
32. Rutgers University — 6.3
33. Toyota Technological Institute at Chicago — 5.4
34. UC Davis — 5.3
35. Ohio State University — 5.3
36. Rice University — 5.2
37. Northwestern University — 4.9
38. Indiana University — 4.9
39. SUNY Stony Brook — 4.5
40. UC Irvine — 4.4

Top 40 Global Universities Leading in AI Research at NeurIPS 2020 (with Publication Indices):

Top 40 Global Universities Leading in AI Research at NeurIPS 2020

1. Stanford University (USA) — 67.0
2. MIT (USA) — 61.1
3. UC Berkeley (USA) — 52.4
4. Carnegie Mellon University (USA) — 47.3
5. University of Oxford (UK) — 35.5
6. Tsinghua University (China) — 34.5
7. Princeton University (USA) — 28.0
8. ETH (Switzerland) — 26.6
9. New York University (USA) — 26.1
10. UT Austin (USA) — 25.7
11. Columbia University (USA) — 25.5
12. KAIST (South Korea) — 23.8
13. University of Illinois at Urbana-Champaign (USA) — 23.6
14. Cornell University (USA) — 23.0
15. EPFL (Switzerland) — 22.6
16. Harvard University (USA) — 22.3
17. University of Cambridge (UK) — 21.6
18. UCLA (USA) — 18.7
19. UC San Diego (USA) — 18.3
20. Peking University (China) — 18.1
21. University College London (UK) — 15.5
22. Georgia Institute of Technology (USA) — 15.1
23. University of Pennsylvania (USA) — 14.3
24. University of Maryland (USA) — 13.8
25. University of Michigan (USA) — 13.6
26. University of Toronto (Canada) — 13.2
27. National University of Singapore (Singapore) — 12.2
28. Purdue University (USA) — 11.8
29. University of Washington (USA) — 11.5
30. INRIA (France) — 11.1
31. Duke University (USA) — 10.7
32. Boston University (USA) — 10.3
33. UMass Amherst (USA) — 9.9
34. University of Southern California (USA) — 9.7
35. Tel Aviv University (Israel) — 9.4
36. Imperial College London (UK) — 9.1
37. Shanghai Jiao Tong University (China) — 9.1
38. University of Tokyo (Japan) — 8.8
39. Seoul National University (South Korea) — 8.8
40. California Institute of Technology (USA) — 8.8

Top 20 Global Companies Leading in AI Research at NeurIPS 2020 (with Publication Indices):

Top 20 Global Companies Leading in AI Research at NeurIPS 2020

1. Google (USA) — 128.0
2. Microsoft (USA) — 42.9
3. Facebook (USA) — 31.4
4. IBM (USA) — 19.0
5. NVIDIA (USA) — 11.9
6. Huawei (China) — 11.7
7. Amazon (USA) — 8.3
8. Tencent (China) — 8.3
9. Alibaba (China) — 7.6
10. Samsung (South Korea) — 6.9
11. Intel (USA) — 4.5
12. Adobe (USA) — 4.1
13. Salesforce (USA) — 4.0
14. Apple (USA) — 3.7
15. Qualcomm (USA) — 3.5
16. Bosch (Germany) — 3.5
17. Baidu (China) — 3.4
18. SenseTime (China) — 3.0
19. OpenAI (USA) — 2.9
20. Criteo (France) — 2.7

Further Analysis

Word Cloud of Paper Titles at NeurIPS 2020

Changes in rank at Top 10 Global Organizations at NeurIPS 2020 vs. NeurIPS 2019:

Google: stayed at 1st place
Stanford University: stayed at 2nd place
MIT: improved from 4th to 3rd place
UC Berkeley: improved from 6th to 4th place
Carnegie Mellon University: dropped from 3rd to 5th place
Microsoft: dropped from 5th to 6th place
University of Oxford: stayed at 7th place
Tsinghua University: improved significantly from 13th to 8th place
Facebook: dropped from 8th to 9th place
Princeton University: stayed at 10th place

Changes in Publication Indices at Top 50 Global Organizations at NeurIPS 2020 vs. NeurIPS 2019 (positive change means more publications at NeurIPS 2020 than at NeurIPS 2019):

1. Google (USA): up 33.5 (35%) — from 94.5 to 128.0
2. Stanford University (USA): up 9.1 (16%) — from 57.8 to 67.0
3. MIT (USA): up 14.4 (31%) — from 46.7 to 61.1
4. UC Berkeley (USA): up 22.8 (77%) — from 29.7 to 52.4
5. Carnegie Mellon University (USA): down -1.2 (-3%) — from 48.5 to 47.3
6. Microsoft (USA): up 7.8 (22%) — from 35.1 to 42.9
7. University of Oxford (UK): up 11.9 (51%) — from 23.6 to 35.5
8. Tsinghua University (China): up 15.6 (82%) — from 18.9 to 34.5
9. Facebook (USA): up 7.8 (33%) — from 23.6 to 31.4
10. Princeton University (USA): up 7.2 (35%) — from 20.7 to 28.0
11. ETH (Switzerland): up 11.7 (78%) — from 14.9 to 26.6
12. New York University (USA): up 12.1 (86%) — from 14.0 to 26.1
13. UT Austin (USA): up 6.9 (36%) — from 18.9 to 25.7
14. Columbia University (USA): up 2.2 (10%) — from 23.3 to 25.5
15. KAIST (South Korea): up 20.0 (525%) — from 3.8 to 23.8
16. University of Illinois at Urbana-Champaign (USA): up 3.7 (18%) — from 20.0 to 23.6
17. Cornell University (USA): up 2.7 (13%) — from 20.3 to 23.0
18. EPFL (Switzerland): up 10.0 (79%) — from 12.6 to 22.6
19. Harvard University (USA): up 10.1 (82%) — from 12.2 to 22.3
20. University of Cambridge (UK): up 12.5 (138%) — from 9.1 to 21.6
21. IBM (USA): up 3.6 (23%) — from 15.4 to 19.0
22. UCLA (USA): down -0.1 (-1%) — from 18.8 to 18.7
23. UC San Diego (USA): up 6 (48%) — from 12.3 to 18.3
24. Peking University (China): up 2.4 (15%) — from 15.7 to 18.1
25. University College London (UK): up 6.0 (63%) — from 9.5 to 15.5
26. Georgia Institute of Technology (USA): down -0.5 (-3%) — from 15.6 to 15.1
27. University of Pennsylvania (USA): up 5.1 (56%) — from 9.2 to 14.3
28. University of Maryland (USA): up 7.5 (119%) — from 6.3 to 13.8
29. University of Michigan (USA): up 6.9 (103%) — from 6.7 to 13.6
30. University of Toronto (Canada): down -1.7 (-11%) — from 14.9 to 13.2
31. National University of Singapore (Singapore): up 6.8 (128%) — from 5.4 to 12.2
32. NVIDIA (USA): up 7.4 (163%) — from 4.5 to 11.9
33. Purdue University (USA): up 4.1 (52%) — from 7.7 to 11.8
34. Huawei (China): up 9.2 (381%) — from 2.4 to 11.7
35. University of Washington (USA): down -3.4 (-23%) — from 15.0 to 11.5
36. INRIA (France): down -5.6 (-34%) — from 16.7 to 11.1
37. Duke University (USA): down -0.8 (-7%) — from 11.5 to 10.7
38. Boston University (USA): up 5.6 (119%) — from 4.7 to 10.3
39. UMass Amherst (USA): up 0.3 (3%) — from 9.6 to 9.9
40. University of Southern California (USA): up 0.6 (6%) — from 9.1 to 9.7
41. Tel Aviv University (Israel): up 6.2 (193%) — from 3.2 to 9.4
42. Imperial College London (UK): up 3.1 (53%) — from 5.9 to 9.1
43. Shanghai Jiao Tong University (China): up 7.3 (409%) — from 1.8 to 9.1
44. University of Tokyo (Japan): up 4.7 (114%) — from 4.1 to 8.8
45. Seoul National University (South Korea): up 3.5 (66%) — from 5.3 to 8.8
46. California Institute of Technology (USA): up 0.8 (10%) — from 8.0 to 8.8
47. University of Science and Technology of China (China): up 4.5 (118%) — from 3.8 to 8.4
48. Amazon (USA): down -2.8 (-25%) — from 11.1 to 8.3
49. Tencent (China): up 3 (57%) — from 5.3 to 8.3
50. Northeastern University (USA): up 0.8 (11%) — from 7.3 to 8.1

Discussion

The top two positions, Google and Stanford University, remained the same. MIT moved up from 4th to 3rd place. UC Berkeley moved up from 6th to 4th place. Carnegie Mellon dropped from 3rd to 5th place. Microsoft dropped from 5th to 6th place. The University of Oxford stayed at 7th place. Tsinghua University jumped from 13th to 8th place. Facebook dropped from 8th to 9th place. Princeton stayed at 10th place.

To move up in rankings, each organization had to significantly increase their Publication Index: Google published an equivalent of 33.5 more papers, Stanford University is up by 9.1, MIT is up by 14.4, UC Berkeley is up by 22.8, etc. (see table above). Just like in Lewis Carroll’s Red Queen’s race, top organizations need to publish more papers each year just to maintain the lead.

“My dear, here we must run as fast as we can, just to stay in place. And if you wish to go anywhere you must run twice as fast as that.” (Lewis Carroll)

Dataset

About me: My name is Gleb Chuvpilo, and I’m the Managing Partner at Thundermark Capital, a Venture Capital firm that invests in AI and robotics startups. I have a Master’s degree from the MIT Computer Science and Artificial Intelligence Lab and an MBA in Finance and Strategic Management from The Wharton School at the University of Pennsylvania. You can read more about me here. Please drop me a line at gleb@thundermark.com if you’d like to talk about AI, robotics, innovation in general, or your startup idea in particular. 🤖

Serial Entrepreneur & Investor | AI @ MIT & MBA @ Wharton | Thundermark, Peter Thiel, Y Combinator, Palantir, Goldman Sachs