Question

Metaculus Help: Spread the word

If you like Metaculus, tell your friends! Share this question via Facebook, Twitter, or Reddit.

Will there be some kind of APM/micro-ability controversy surrounding the OpenAI Five Finals?

This is related to other questions on OpenAI Five, including:


OpenAI has announced the final live event for OpenAI five will be a series of matches against OG, the reigning Dota 2 world champions.

Dota 2 is similar in many regards to the game Starcraft 2. Both games involve long-term planning, imperfect information, very large action spaces and real-time interactions.

Following DeepMind's surprise January release of the powerful AlphaStar agent, much of the ensuing discussion focused on the agent's seemingly superhuman action-per-minute stats. It was unclear whether the win was actually due to superhuman Starcraft 2 performance, as opposed to, very roughly... superhuman clicking.

Now, with a significant chance of very powerful performance also for Dota 2, we ask:

Will there be some kind of APM controversy surrounding the OpenAI Five Finals?


Resolution

The question will resolve positive if, within two weeks of the finals event, there is at least two Reddit posts/comments with at least 100 karma, and a HackerNews post/comment with at least 30 karma, claiming that OpenAI Five's performance is substantially less impressive than it seems due to superhuman APM/micro abilities/something similar. This might come down to a question of judgement, but I expect it to be relatively clear.


Examples sufficient for positive resolution

farmingvillein [Reddit] 101 points · 1 month ago

Great post OP.

What leaves the sourest taste in my mouth is this image: https://i.redd.it/ctuungv1dtc21.png

This is the part that ultimately really bothers me, as it is basically prevarication. I really, really hope that they don't get away with publishing a Nature article with a chart/description like this.

And it is all a little frustrating because what Deepmind showed actually is super cool--beating top players with the APM restrictions that they have in place is a big achievement. But claim your victory within the bounds you were working under, and be really upfront about how it is arguably superhuman in certain areas, and commit to resolving that issue in a more defensible way.

Or, if you're not going to commit to fixing the issue, relax your claims about what your goals are.

wnevets 61 days ago [+43] [HackerNews]

I think a far more interesting limitation would be to cap APM at 150 or so, or to artificially limit action precision with some sort of virtual mouse that reduced accuracy as APM increased.

IIRC OpenAI limits the reaction time to ~200ms when playing DoTA2. AI employing better strategies than humans will always be more interesting than AI that can out click humans.

jgon 61 days ago [+41] [HackerNews]

Even the 200ms reaction time seemed overly slanted towards the AI. I don't think that is the actual reaction time of top pros, in the matches the AI played the human player would teleport in from complete invisibility and try to use an instant cast spell and the AI would have already teleported out. Yes the theoretically may have been constrained to a 200ms reaction time, but in practice the AI was playing at a superhuman level. Even with that advantage in fights, the human team still demolished the AI. Oh well, lots of things to learn still.

Examples not sufficient for positive resolution

gamegoblin 61 days ago [94] [HackerNews]

In the mass stalker battles, the AI APM exceeded 1000 a few times, and no doubt that most of that was precisely targeted. Whereas a human doing 500 APM micro is obviously going to be far more imprecise. I think a far more interesting limitation would be to cap APM at 150 or so, or to artificially limit action precision with some sort of virtual mouse that reduced accuracy as APM increased.

cm2012 61 days ago [+25] [HackerNews]

In the showmatched they made the computer have to look at a regular screen to control, the stalker micro was much less impressive - and mana won.

{{qctrl.predictionString()}}

Metaculus help: Predicting

Predictions are the heart of Metaculus. Predicting is how you contribute to the wisdom of the crowd, and how you earn points and build up your personal Metaculus track record.

The basics of predicting are very simple: move the slider to best match the likelihood of the outcome, and click predict. You can predict as often as you want, and you're encouraged to change your mind when new information becomes available.

The displayed score is split into current points and total points. Current points show how much your prediction is worth now, whereas total points show the combined worth of all of your predictions over the lifetime of the question. The scoring details are available on the FAQ.

Note: this question resolved before its original close time. All of your predictions came after the resolution, so you did not gain (or lose) any points for it.

Note: this question resolved before its original close time. You earned points up until the question resolution, but not afterwards.

This question is not yet open for predictions.

Thanks for predicting!

Your prediction has been recorded anonymously.

Want to track your predictions, earn points, and hone your forecasting skills? Create an account today!

Track your predictions
Continue exploring the site

Community Stats

Metaculus help: Community Stats

Use the community stats to get a better sense of the community consensus (or lack thereof) for this question. Sometimes people have wildly different ideas about the likely outcomes, and sometimes people are in close agreement. There are even times when the community seems very certain of uncertainty, like when everyone agrees that event is only 50% likely to happen.

When you make a prediction, check the community stats to see where you land. If your prediction is an outlier, might there be something you're overlooking that others have seen? Or do you have special insight that others are lacking? Either way, it might be a good idea to join the discussion in the comments.