Metaculus

Forecasting AI

AI Fermi Prize Round 2 winners

Round 2 of the AI Fermi Prize closed February 23 at 23:59 GMT. Three prizes have been awarded to commenters who substantially contributed to our mission of tracking and improving the state-of-the-art of forecasts on AI progress.

(Note that several new comments were submitted very close after the deadline. Those will be added to the next prize round.)

The top prize ($400) goes to kokotajlod, for many great question suggestions, clear factorizations, and careful in-depth thinking about AlphaStar.

Discussion topic: what are some suggestions for questions to launch?
kokotajlod commented on Feb 15, 2019 at 2:58pm
Inspired by this (to me) surprising milestone:
--Something about how quickly a team other than OpenAI will be able to replicate this result
--Something about whether the next team to replicate the result will likewise choose to withhold details that would ordinarily be published for ethical reasons.
--Something about whether Xiaoice chatbot and/or Zo chatbot will become significantly better in the next year. (It is thought that OpenAI's breakthrough might lead to better chatbots)
--Something about whether this technology actually will be used maliciously (as reported by credible news source) within the next five years or so
By end of 2020, will another state-of-the-art model be only partially released due to explicit concerns about malicious use?
kokotajlod factored the question on Feb 22, 2019 at 4:56am
I am so confident on this (90%) because:
*There will probably be at least dozens of benchmark-surpassings in the next three years.
* Now that OpenAI has set the example, it will be a live option for each of them to withhold information citing security concerns.
* Besides actual security concerns, there are other motives for doing this (e.g. to obscure weaknesses in the result, or to prevent competition from catching up, or to sound badass).
* There are likely to be actual security concerns sometime in the next three years for some benchmark-surpassings at least.
* Even if there are no good reasons to do this in the next three years, there are going to be enough opportunities that it's likely someone will do it anyway. (Perhaps they mistakenly think their work is dangerous, for example, or perhaps they misjudge the reputation effects of withholding information.)
The best counterargument I can think of is "In the past X years of AI research, OpenAI's announcement is the only example of this so far, so the base rate is low." My reply to that counterargument is that when something happens for the first time the base rate will always be low, yet reliably after something happens once it often happens again.
By end of 2019, will there be an agent at least as good as AlphaStar using non-controversial, human-like APM restrictions?
kokotajlod commented 5 days 18 hours ago
I've written up a gdoc articulating my hypothesis & the evidence for it & the predictions that the hypothesis makes. Comments welcome!
The tl;dr is that I think the best explanation for the Game 11 failure is that AlphaStar isn’t reasoning about the consequences of its actions using a model of the world. Rather, it’s just acting on instincts/heuristics/intuitions that have evolved/been baked into it via millions of previous games played by its ancestors.

Just like in Round 1, the second prize is shared. davidmanheim wins $100 for providing a broad range of brief but careful comments on conference attendance and the the growth of statistics on RL arXiv papers.

James also wins $100 for his two brief comments improving DanielFilan’s model of Starcraft timelines. Remember that a large amount of work is not necessary to win the prize. Two comments that make tangible progress is very prize-worthy as well.

Overall we were excited to see some new users joining in this round. So we also want to extend honourable mentions to Aidan_OGara and Jsevillamol.

Round 3 is now on, and will prize comments written between February 24 and March 10, 23:59 GMT. You can find more examples of prize-worthy commenting here.