OpenAI Gym: Getting Started with CartPole

In [25]:

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:50% !important; }</style>"))

Open AI¶

此notebook是用docker image "eboraas/openai-gym" 跑起來的

以下是完整的OpenAI執行範例

In [14]:

import gym
env = gym.make('CartPole-v0')
for timestep in range(1, 100):
    action = env.action_space.sample()
    observation, reward, done, info = env.step(action)
    if done:
        print("Episode finished after {} timesteps".format(timestep))
        break

[2016-06-21 12:30:34,354] Making new env: CartPole-v0

Episode finished after 12 timesteps

這段code先是利用 gym.make 創了一個環境物件 env make 吃的參數是想創的環境 這裡我們給的 CartPole-v0 對應的就是網站的 https://gym.openai.com/envs/CartPole-v0

env物件會有一些固定的方法 這裡我們用到了 action_space 和 step action_space會給你一個 action 清單，不同環境會有不同的 action 選擇

In [16]:

import gym
env = gym.make('CartPole-v0')
print(env.action_space)

[2016-06-21 12:35:03,149] Making new env: CartPole-v0

Discrete(2)

我們看到 CartPole 還竟洽有兩個動作 動作空間是離算的大小是 我們解讀成 往左 或 往右 移動

通常我們會想跑很多次episode 就是遊戲重複玩很多次

In [21]:

# 遊戲玩20次

import gym
env = gym.make('CartPole-v0')
for i_episode in range(1, 21):
    observation = env.reset()
    for t in range(1, 100):
        env.render()
        action = env.action_space.sample()
        observation, reward, done, info = env.step(action)
        if done:
            print("Episode {} finished after {} timesteps".format(i_episode, t+1))
            break

[2016-06-21 14:22:38,742] Making new env: CartPole-v0

Episode 1 finished after 20 timesteps
Episode 2 finished after 24 timesteps
Episode 3 finished after 43 timesteps
Episode 4 finished after 15 timesteps
Episode 5 finished after 10 timesteps
Episode 6 finished after 24 timesteps
Episode 7 finished after 23 timesteps
Episode 8 finished after 13 timesteps
Episode 9 finished after 31 timesteps
Episode 10 finished after 19 timesteps
Episode 11 finished after 17 timesteps
Episode 12 finished after 10 timesteps
Episode 13 finished after 13 timesteps
Episode 14 finished after 16 timesteps
Episode 15 finished after 37 timesteps
Episode 16 finished after 12 timesteps
Episode 17 finished after 12 timesteps
Episode 18 finished after 22 timesteps
Episode 19 finished after 53 timesteps
Episode 20 finished after 46 timesteps

目前選動作完全沒有策略所以遊戲很快就結束 ... 很快就死

注意:回合與回合之間需要一個 env.reset()

上傳結果¶

官網寫

CartPole-v0 defines "solving" as getting average reward of 195.0 over 100 consecutive trials.

每一次step() 會回傳一個reward數值 一個episode重開始到結束的總reward就是每一步reward的合 CartPole 的reward都是1，意思是說一個episode的reward是存活的時刻數

網站要求我們100回合平均存活195時刻

作弊，手動破解¶

我們應該做的是寫一個學習城市，他會透過這100回的遊戲學會如何選最好的動作，目的是在這100回遊戲結束前 快速學會正確玩法，導致後面的存活時間可提高整個平均分數

但因為cartpole還竟已經是物理有名的問題，我們可以先作弊，寫好一個已經知道正確答案的城市，跑100回，把這100回合的紀錄上傳至OpenAI網站

env.monitor.start 開始紀錄，env.monitor.close() 結束

In [24]:

import gym
env = gym.make('CartPole-v0')
env.monitor.start('cartpole-manual-parameters-100-trials/', force=True) # force deletes existing record if it exists

for _ in range(100):
    observation = env.reset()
    
    for t in range(1, 50000):
        left_or_right = int((observation[2] + observation[3]*0.1 + observation[0]*0.001) > 0)
        observation, reward, done, info = env.step(left_or_right) # take a random action
        if done:
            break
env.monitor.close()

[2016-06-21 14:40:12,809] Making new env: CartPole-v0
[2016-06-21 14:40:12,855] Clearing 12 monitor files from previous run (because force=True was provided)
[2016-06-21 14:40:12,947] Starting new video recorder writing to /mnt/notebooks/cartpole-manual-parameters-100-trials/openaigym.video.0.27.video000000.mp4
[2016-06-21 14:40:16,844] Ending episode 1 because it reached the timestep limit of 200.
[2016-06-21 14:40:16,977] Starting new video recorder writing to /mnt/notebooks/cartpole-manual-parameters-100-trials/openaigym.video.0.27.video000001.mp4
[2016-06-21 14:40:20,670] Ending episode 2 because it reached the timestep limit of 200.
[2016-06-21 14:40:20,812] Ending episode 3 because it reached the timestep limit of 200.
[2016-06-21 14:40:20,870] Ending episode 4 because it reached the timestep limit of 200.
[2016-06-21 14:40:20,899] Ending episode 5 because it reached the timestep limit of 200.
[2016-06-21 14:40:20,924] Ending episode 6 because it reached the timestep limit of 200.
[2016-06-21 14:40:21,051] Ending episode 7 because it reached the timestep limit of 200.
[2016-06-21 14:40:21,076] Ending episode 8 because it reached the timestep limit of 200.
[2016-06-21 14:40:21,112] Starting new video recorder writing to /mnt/notebooks/cartpole-manual-parameters-100-trials/openaigym.video.0.27.video000008.mp4
[2016-06-21 14:40:24,655] Ending episode 9 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,802] Ending episode 10 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,831] Ending episode 11 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,854] Ending episode 12 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,886] Ending episode 13 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,906] Ending episode 14 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,931] Ending episode 15 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,953] Ending episode 16 because it reached the timestep limit of 200.
[2016-06-21 14:40:24,977] Ending episode 17 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,013] Ending episode 18 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,044] Ending episode 19 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,169] Ending episode 20 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,194] Ending episode 21 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,221] Ending episode 22 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,261] Ending episode 23 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,281] Ending episode 24 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,308] Ending episode 25 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,329] Ending episode 26 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,360] Ending episode 27 because it reached the timestep limit of 200.
[2016-06-21 14:40:25,373] Starting new video recorder writing to /mnt/notebooks/cartpole-manual-parameters-100-trials/openaigym.video.0.27.video000027.mp4
[2016-06-21 14:40:29,070] Ending episode 28 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,258] Ending episode 29 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,288] Ending episode 30 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,313] Ending episode 31 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,342] Ending episode 32 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,363] Ending episode 33 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,394] Ending episode 34 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,426] Ending episode 35 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,455] Ending episode 36 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,503] Ending episode 37 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,526] Ending episode 38 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,554] Ending episode 39 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,575] Ending episode 40 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,600] Ending episode 41 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,620] Ending episode 42 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,641] Ending episode 43 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,666] Ending episode 44 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,686] Ending episode 45 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,714] Ending episode 46 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,745] Ending episode 47 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,769] Ending episode 48 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,790] Ending episode 49 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,815] Ending episode 50 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,883] Ending episode 51 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,912] Ending episode 52 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,971] Ending episode 53 because it reached the timestep limit of 200.
[2016-06-21 14:40:29,990] Ending episode 54 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,019] Ending episode 55 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,075] Ending episode 56 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,095] Ending episode 57 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,121] Ending episode 58 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,140] Ending episode 59 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,164] Ending episode 60 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,189] Ending episode 61 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,214] Ending episode 62 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,242] Ending episode 63 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,275] Ending episode 64 because it reached the timestep limit of 200.
[2016-06-21 14:40:30,286] Starting new video recorder writing to /mnt/notebooks/cartpole-manual-parameters-100-trials/openaigym.video.0.27.video000064.mp4
[2016-06-21 14:40:33,775] Ending episode 65 because it reached the timestep limit of 200.
[2016-06-21 14:40:33,942] Ending episode 66 because it reached the timestep limit of 200.
[2016-06-21 14:40:33,965] Ending episode 67 because it reached the timestep limit of 200.
[2016-06-21 14:40:33,997] Ending episode 68 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,026] Ending episode 69 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,151] Ending episode 70 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,174] Ending episode 71 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,203] Ending episode 72 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,225] Ending episode 73 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,250] Ending episode 74 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,273] Ending episode 75 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,301] Ending episode 76 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,331] Ending episode 77 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,363] Ending episode 78 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,391] Ending episode 79 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,415] Ending episode 80 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,441] Ending episode 81 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,472] Ending episode 82 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,496] Ending episode 83 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,522] Ending episode 84 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,559] Ending episode 85 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,587] Ending episode 86 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,617] Ending episode 87 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,644] Ending episode 88 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,672] Ending episode 89 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,701] Ending episode 90 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,734] Ending episode 91 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,763] Ending episode 92 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,791] Ending episode 93 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,829] Ending episode 94 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,861] Ending episode 95 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,883] Ending episode 96 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,914] Ending episode 97 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,936] Ending episode 98 because it reached the timestep limit of 200.
[2016-06-21 14:40:34,967] Ending episode 99 because it reached the timestep limit of 200.
[2016-06-21 14:40:35,003] Ending episode 100 because it reached the timestep limit of 200.
[2016-06-21 14:40:35,038] Finished writing results. You can upload them to the scoreboard via gym.upload('/mnt/notebooks/cartpole-manual-parameters-100-trials')

注意: 每個episode會在第200時刻自動結束

Finished writing results. You can upload them to the scoreboard via gym.upload('/mnt/notebooks/cartpole-manual-parameters-100-trials')

跑完後，把monitor存的紀錄上傳上去 注意:upload需要你的帳號的api key，否則會發生錯誤 (登入後點你的名字拜訪你的主要頁面，會顯示你的API Key)

In [2]:

gym.upload('/mnt/notebooks/cartpole-manual-parameters-100-trials/', api_key='sk_XIkAsixqQPSItwhbgYXQMw')

[2016-06-20 14:11:26,039] [CartPole-v0] Uploading 100 episodes of training data
[2016-06-20 14:11:27,958] [CartPole-v0] Uploading videos of 5 training episodes (32896 bytes)
[2016-06-20 14:11:28,541] [CartPole-v0] Creating evaluation object from /mnt/notebooks/cartpole-manual-parameters-100-trials/ with learning curve and training video
[2016-06-20 14:11:28,829] 
****************************************************
You successfully uploaded your evaluation on CartPole-v0 to
OpenAI Gym! You can find it at:

    https://gym.openai.com/evaluations/eval_iZDTa8JeQmKT1eZxufJbg

****************************************************

Out[2]:

<Evaluation evaluation id=eval_iZDTa8JeQmKT1eZxufJbg at 0x7feb61cab618> JSON: {
  "created": 1466431888, 
  "env": "CartPole-v0", 
  "id": "eval_iZDTa8JeQmKT1eZxufJbg", 
  "object": "evaluation"
}