Universe

Universe is an open-source software platform developed by OpenAI for measuring and training an artificial intelligence (AI) general intelligence, also called strong AI.  It's a middleware program that supports OpenAI's Gym, a toolkit for the development and evaluation of reinforment learning (RL) algorithms.  Games and websites are used to train the AI. Indeed, any task that a person can solve on a computer is theoretically a viable option for training, with researchers being able to plug any application into Universe so AI agents have a common way of interacting with the applications.  

The software environments are instantiated in Docker containers. This way, AI agents are provided with challenges that are interacted with through a virtual keyboard and mouse input using a Virtual Network Computing (VNC) remote desktop. The more interaction the agents have with the environment, the better they become at a specific task.

OpenAI gave an example of a game used to train AI through RL, describing an agent playing Slither, a game in which the player has to avoid collision against multiple snakes. After six days of training time, the agent scored "an average of 1,000 points, with a high score of 9,300 points. As a point of comparison, OpenAI machine-learning researcher Rafal Jozefowicz, with five hours of playing experience, averaged about 1,400 points, with a high score of 7,050."

Universe has been compared to projects like ImageNet, a massive database of hand-labeled sets of images that has been used by researchers to test image recognition systems. In OpenAI's Universe, images are substituted by flash games, web browsers, photo editors, and CAD software.

On release, there were 2500 Atari games, 1000 flash games, and 80 browser environments available.

However, this project seems to have been abandoned by OpenAI in favor of Gym Retro. Several of the upcoming developments that were posted in their blog regarding the Universe launch were never released. This included environment integration tools, so any user could contribute with new environment integrations and the public release of human demonstration data on Universe environments.

Objective

According to OpenAI, the goal of the project was to "develop a single AI agent that can flexibly apply its past experience on Universe environments to quickly master unfamiliar, difficult environments, which would be a major step towards general intelligence," something similar to human brain power. According to Ilya Sutskever, an OpenAI researcher, "an AI should be able to solve any problem you throw at it."  

By expanding the number of training resources, it was expected that the education of AI agents would accelerate. Indeed, before the release of OpenAI's platform, the largest RL resource was the Atari Learning Environment which included 55 games.  

On the contrary, Universe released with the largest library of games and resources ever. These included 1,000 flash games, distributed in Docker image, games like slither.io and Starcraft, browser-based tasks and applications like form filling and fold.it. According to OpenAI, Flash games were a starting point for scaling the platform because they are pervasive on the internet, usually with better graphics than Atari games but are still simple.

OpenAI also mentioned that with the environments running asynchronously inside the Docker image with a local network in the cloud, the games usually ran at 60 fps while over public internet this dropped to 20 fps.

Functionality

Universe allows a user to train and evaluate AI agents, with the AI using a computer like a human would. This provides a wide range of real-time and complex environments. This platform permits any program to become an OpenAI Gym environment easily, without needing special access to the program's internal, source code, or APIs. According to OpenAI's Github, it "does this by packaging the program into a Docker container, and presenting the AI with the same interface a human uses: sending keyboard and mouse events, and receiving screen pixels."  

The AI agent explores different environments visually, observing pixels on a screen and inputting keyboard and mouse commands.  This interface is implemented using the VNC program for remote desktop access. 

Games in the Universe platform provide the feedback loop necessary for the constant improvement of AI skills, gathering experience in small tasks and resolving new ones faster. Ideally, the agent would surpass its specialized knowledge of a specific environment, aiming at a more generalized intelligence that contributes to the resolution of different kinds of problems. Reward functions are Integral to this, necessary for RL. In many games, there's an on-screen score that can be used as a reward.

Besides the game environments, Universe includes browser-based navigation where the agent can interact with the web like people do, learning how to use elements like buttons, lists, and sliders. OpenAI developed a benchmark called Mini World of Bits to understand the challenges of browser interactions in a simplified setting. Consisting of 80 environments that range from simple tasks like clicking a specific button to difficult ones like replying to a contact in a simulated email client, OpenAI believes "that mastering these environments provides valuable signal towards models and training techniques that will perform well on full websites and more complex tasks."  

Properties

During the implementation of this platform, OpenAI had as important properties:

1. Generalization, in which an AI agent can use the interface designed for humans to interact with programs, allowing it to browse the web, play games, interact with a terminal, edit spreadsheets or even operate a photo editing program;

2. Familiarity, since the agents interact with the environments in a way that is usual for humans;

3. VNC as standard, since many implementations of VNC are available online, allowing humans to give demonstrations without installing new software;

4. Easily debugged, the VNC traffic can be saved for analysis and an agent can be observed while in training or being evaluated.