Gym

Figure 1: OpenAI Gym agent-environment. Source: Velotio Perspectives.

Gym is a standard API for developing and testing learning agents developed by OpenAI, originally built to accelerate the company's reinforcement learning (RL) research. While it has been developed for RL, other deep learning approaches can be used. Open AI Gym has a simple interface, pythonic, capable of representing general RL problems, a diverse collection of environments and a site to compare and reproduce results of different RL algorithms. It is also compatible with algorithms written in other frameworks like Google's Tensorflow or University of Montreal's Theano. The beta version was released in 2016.

The toolkit provides an environment and agent arrangement, meaning that there is an agent that can execute actions in an environment. The agent then receives an observation and reward derived from the particular action executed (figure 1).

The environments are virtual spaces where algorithms are tested. OpenAI Gym allows researchers to build algorithms and test them, obtaining feedback relative to an objective test where they can make adjustments and publish their benchmarks for community review.

The algorithms shared with the community are promoted according to those that can generalize well and not through leaderboards based on top scoring algorithms. The versatility in being able to complete other similar tasks is a step towards generalization, considered as the main problem for AI to achieve human level intelligence. According to OpenAI Gym documentation, "It’s not just about maximizing score; it’s about finding solutions which will generalize well. Solutions which involve task-specific hardcoding or otherwise don’t reveal interesting characteristics of learning algorithms are unlikely to pass review."

Using OpenAI Gym

Gym can be used to:

1. Learn RL algorithms;
2. Test new ideas for learning agents;
3. Train an agent on cases without modeling in reality;
4. Increase the rate of learning.

Integrations with OpenAI Gym

Isaac

In 2017, Nvidia announced Isaac. This system uses RL to train robots in virtual environments and employs the OpenAI Gym.

Roboschool

Developed by OpenAI and released in 2017, Roboschool is an open-source software for robot simulation that is integrated with OpenAI Gym. (7) It expands the Gym's environments library for controlling robots in simulation and also eases the process of training multiple agents together in the same environment.

Gym Retro

Gym Retro is a platform for RL research on games. The full version was released in 2018 and increased the released games on the platform to over 1,000 while also providing the tool used to add new games to it.

OpenAI's uses Gym Retro for research on RL algorithms and generalization since it provides the ability to generalize between games with the same concepts but divergent appearances instead of focusing on the agent's optimization to solve a single task.

Safety Gym

Safety Gym is a set of environments and tools released in 2019 by OpenAI to accelerate the study of constrained RL for safe exploration . These environments are richer, featuring an increased range of difficulty when compared to other environments for constrained RL. According to Ray et al. (2019), "Safety Gym includes a benchmark suite of 18 high-dimensional continuous control environments for safe exploration, plus 9 additional environments for debugging task performance separately from safety requirements, and tools for building additional environments."

The two main component of Safety Gym are 1) an environment-builder, allowing the creation of new environments by mixing a variety of physics elements, goals and safety requirements; and 2) pre-configured benchmark environments that help standardize the progress measurement on the safe exploration problem.

Reinforcement Learning

OpenAI Gym centers around reinforcement learning, a subfield of machine learning focused on decision making and motor control. According to OpenAI, it studies "how an agent can learn to achieve goals in a complex, uncertain environment." When an agent performs well, a reward is given. When it fails, due to the absence of reward, the algorithm with try an alternative. This technique has had good results with robots and video games.

RL is not a fully matured technology but it has gained in importance for developing highly-capable AI systems. It has been used in competitive strategy games like Go and Dota, achieving superhuman performance. On the robotics fields, it has been used to control highly-complex systems and on social media to improve the content presented to users.

One advantage of RL is that it is very general, able to be applied to "supervised learning problems with sequential or structured outputs." Also, RL algorithms have achieved good results in difficult environments by using deep RL algorithms that do not make many assumptions about their environment. This way they can be applied to other settings.

Requirements and installation

OpenAI Gym requires Phyton 3.5+ and pip.

Installation

• pip install gym

To install it from source:

• git clone https://github.com/openai/gym && cd gym

• pip install -e

Environments

There are various environments available in OpenAI Gym, differing in difficulty.

• Classic control and toy text
• Algorithmic
• Atari
• Board Games
• 2D and 3D robots

To create an environment, first import Gym and then create the environment:

• import gym
• env = gym.make('MountainCar-v0')

Four values are returned by the environment for every action taken by the agent:

1. Observation: an environment-specific object representing the observation of the environment;
2. Reward: the reward/score achieved by the last action. The goal is to maximize the reward;
3. Done: whether the episode has ended. Then, the environment may need to be reset;
4. Info: information about the environment or diagnostic information for debugging.