This is related to other questions on OpenAI Five, including:
- how much training compute the 2018 version used
- whether it will defeat the world champions
- whether it will defeat the world champions, conditional on AGI in <20 years
OpenAI has announced the final live event for OpenAI five will be a series of matches against OG, the reigning Dota 2 world champions.
Dota 2 is similar in many regards to the game Starcraft 2. Both games involve long-term planning, imperfect information, very large action spaces and real-time interactions.
Following DeepMind's surprise January release of the powerful AlphaStar agent, much of the ensuing discussion focused on the agent's seemingly superhuman action-per-minute stats. It was unclear whether the win was actually due to superhuman Starcraft 2 performance, as opposed to, very roughly... superhuman clicking.
Now, with a significant chance of very powerful performance also for Dota 2, we ask:
Will there be some kind of APM controversy surrounding the OpenAI Five Finals?
The question will resolve positive if, within two weeks of the finals event, there is at least two Reddit posts/comments with at least 100 karma, and a HackerNews post/comment with at least 30 karma, claiming that OpenAI Five's performance is substantially less impressive than it seems due to superhuman APM/micro abilities/something similar. This might come down to a question of judgement, but I expect it to be relatively clear.
Examples sufficient for positive resolution
farmingvillein [Reddit] 101 points · 1 month ago
Great post OP.
What leaves the sourest taste in my mouth is this image: https://i.redd.it/ctuungv1dtc21.png
This is the part that ultimately really bothers me, as it is basically prevarication. I really, really hope that they don't get away with publishing a Nature article with a chart/description like this.
And it is all a little frustrating because what Deepmind showed actually is super cool--beating top players with the APM restrictions that they have in place is a big achievement. But claim your victory within the bounds you were working under, and be really upfront about how it is arguably superhuman in certain areas, and commit to resolving that issue in a more defensible way.
Or, if you're not going to commit to fixing the issue, relax your claims about what your goals are.
wnevets 61 days ago [+43] [HackerNews]
I think a far more interesting limitation would be to cap APM at 150 or so, or to artificially limit action precision with some sort of virtual mouse that reduced accuracy as APM increased.
IIRC OpenAI limits the reaction time to ~200ms when playing DoTA2. AI employing better strategies than humans will always be more interesting than AI that can out click humans.
jgon 61 days ago [+41] [HackerNews]
Even the 200ms reaction time seemed overly slanted towards the AI. I don't think that is the actual reaction time of top pros, in the matches the AI played the human player would teleport in from complete invisibility and try to use an instant cast spell and the AI would have already teleported out. Yes the theoretically may have been constrained to a 200ms reaction time, but in practice the AI was playing at a superhuman level. Even with that advantage in fights, the human team still demolished the AI. Oh well, lots of things to learn still.
Examples not sufficient for positive resolution
gamegoblin 61 days ago  [HackerNews]
In the mass stalker battles, the AI APM exceeded 1000 a few times, and no doubt that most of that was precisely targeted. Whereas a human doing 500 APM micro is obviously going to be far more imprecise. I think a far more interesting limitation would be to cap APM at 150 or so, or to artificially limit action precision with some sort of virtual mouse that reduced accuracy as APM increased.
cm2012 61 days ago [+25] [HackerNews]
In the showmatched they made the computer have to look at a regular screen to control, the stalker micro was much less impressive - and mana won.