RLLAB 入门


执行实验

我们对不同的实验模块使用面向对象抽象。进行实验,可以为环境、算法等等构造对应的对象,然后调用合适的训练方法。简单示例如examples/trpo_cartpole.py。代码如下:

from rllab.algos.trpo import TRPO
from rllab.baselines.linear_feature_baseline import LinearFeatureBaseline
from rllab.envs.box2d.cartpole_env import CartpoleEnv
from rllab.envs.normalized_env import normalize
from rllab.policies.gaussian_mlp_policy import GaussianMLPPolicy

env = normalize(CartpoleEnv())

policy = GaussianMLPPolicy(
env_spec=env.spec,
# The neural network policy should have two hidden layers, each with 32 hidden units.
hidden_sizes=(32, 32)
)

baseline = LinearFeatureBaseline(env_spec=env.spec)

algo = TRPO(
env=env,
policy=policy,
baseline=baseline,
batch_size=4000,
whole_paths=True,
max_path_length=100,
n_itr=40,
discount=0.99,
step_size=0.01,
)
algo.train()

执行第一次需要初始化 Theano 并且编译计算图,可能需要登上几分钟。后续执行会非常快,因为编译已经被缓存了。你可以看到下面的后续信息:

using seed 1
instantiating rllab.envs.box2d.cartpole_env.CartpoleEnv
instantiating rllab.policy.mean_std_nn_policy.MeanStdNNPolicy
using argument hidden_sizes with value [32, 32]
instantiating rllab.baseline.linear_feature_baseline.LinearFeatureBaseline
instantiating rllab.algo.trpo.TRPO
using argument batch_size with value 4000
using argument whole_paths with value True
using argument n_itr with value 40
using argument step_size with value 0.01
using argument discount with value 0.99
using argument max_path_length with value 100
using seed 0
0% 100%
[##############################] | ETA: 00:00:00
Total time elapsed: 00:00:02
2016-02-14 14:30:56.631891 PST | [trpo_cartpole] itr #0 | fitting baseline...
2016-02-14 14:30:56.677086 PST | [trpo_cartpole] itr #0 | fitted
2016-02-14 14:30:56.682712 PST | [trpo_cartpole] itr #0 | optimizing policy
2016-02-14 14:30:56.686587 PST | [trpo_cartpole] itr #0 | computing loss before
2016-02-14 14:30:56.698566 PST | [trpo_cartpole] itr #0 | performing update
2016-02-14 14:30:56.698676 PST | [trpo_cartpole] itr #0 | computing descent direction
2016-02-14 14:31:26.241657 PST | [trpo_cartpole] itr #0 | descent direction computed
2016-02-14 14:31:26.241828 PST | [trpo_cartpole] itr #0 | performing backtracking
2016-02-14 14:31:29.906126 PST | [trpo_cartpole] itr #0 | backtracking finished
2016-02-14 14:31:29.906335 PST | [trpo_cartpole] itr #0 | computing loss after
2016-02-14 14:31:29.912287 PST | [trpo_cartpole] itr #0 | optimization finished
2016-02-14 14:31:29.912483 PST | [trpo_cartpole] itr #0 | saving snapshot...
2016-02-14 14:31:29.914311 PST | [trpo_cartpole] itr #0 | saved
2016-02-14 14:31:29.915302 PST | ----------------------- -------------
2016-02-14 14:31:29.915365 PST | Iteration 0
2016-02-14 14:31:29.915410 PST | Entropy 1.41894
2016-02-14 14:31:29.915452 PST | Perplexity 4.13273
2016-02-14 14:31:29.915492 PST | AverageReturn 68.3242
2016-02-14 14:31:29.915533 PST | StdReturn 42.6061
2016-02-14 14:31:29.915573 PST | MaxReturn 369.864
2016-02-14 14:31:29.915612 PST | MinReturn 19.9874
2016-02-14 14:31:29.915651 PST | AverageDiscountedReturn 65.5314
2016-02-14 14:31:29.915691 PST | NumTrajs 1278
2016-02-14 14:31:29.915730 PST | ExplainedVariance 0
2016-02-14 14:31:29.915768 PST | AveragePolicyStd 1
2016-02-14 14:31:29.921911 PST | BacktrackItr 2
2016-02-14 14:31:29.922008 PST | MeanKL 0.00305741
2016-02-14 14:31:29.922054 PST | MaxKL 0.0360272
2016-02-14 14:31:29.922096 PST | LossBefore -0.0292939
2016-02-14 14:31:29.922146 PST | LossAfter -0.0510883
2016-02-14 14:31:29.922186 PST | ----------------------- -------------

桩模式实验

rllab 同样提供一种“桩”模式来执行实验,这样能够支持更多的配置,例如记录日志和并行化。简单的脚本可在 examples/trpo_cartpole_stub.py 中看到。内容如下:

from rllab.algos.trpo import TRPO
from rllab.baselines.linear_feature_baseline import LinearFeatureBaseline
from rllab.envs.box2d.cartpole_env import CartpoleEnv
from rllab.envs.normalized_env import normalize
from rllab.misc.instrument import stub, run_experiment_lite
from rllab.policies.gaussian_mlp_policy import GaussianMLPPolicy

stub(globals())

env = normalize(CartpoleEnv())

policy = GaussianMLPPolicy(
env_spec=env.spec,
# The neural network policy should have two hidden layers, each with 32 hidden units.
hidden_sizes=(32, 32)
)

baseline = LinearFeatureBaseline(env_spec=env.spec)

algo = TRPO(
env=env,
policy=policy,
baseline=baseline,
batch_size=4000,
whole_paths=True,
max_path_length=100,
n_itr=40,
discount=0.99,
step_size=0.01,
)

run_experiment_lite(
algo.train(),
# Number of parallel workers for sampling
n_parallel=1,
# Only keep the snapshot parameters for the last iteration
snapshot_mode="last",
# Specifies the seed for the experiment. If this is not provided, a random seed
# will be used
seed=1,
# plot=True,
)

第一个需要注意的差异是代码的所有 import 语句后的这一行:stub(globals()),这个可以将所有已经导入的类的构造器用桩式方法替代。在这个调用后,类构造器如 TRPO() 将会返回一个序列化的桩对象,所有方法 invocation 和 属性获取方法同样会变成序列化的桩式调用和桩式属性。接着,run_experiment_lite 调用序列化最终的桩式方法调用,并启动一个脚本实际运行试验。

按照这样的方式启动试验的好处是,我们将试验参数的配置和试验的实际执行分割开来。run_experiment_lite 支持多种执行方式,本地式,在一个 docker 容器本地式,或者在 ec2 远程。按照这样的抽象,不同超参数的多个试验可以被快速构造并在多个 ec2 的机器上同时执行。
另一个微妙的地方是我们使用 Theano 来实现算法,这对混合 GPU 及 CPU 使用支持较弱。当主进程对批量优化使用 GPU 而多个 worker 进行想要使用 GPU 产生轨迹 rollout 的时候非常麻烦。所以分割实验让 worker 进程可以合适地进行 CPU 下的 Theano 的初始化。

run_experiment_lite 的另外参数:
* exp_name:如果该参数设置了,实验数据会被存放在 data/local/{exp_name} 中。默认来说,该文件夹名被设置为 experiment_{timestamp}
* exp_prefix:如果该参数设置了,并且 exp_name 没有指定,实验文件夹名将会被设置为 {exp_prefix}_{timestamp}

实现新的环境

本节我们会介绍如何通过框架实现一个点机器人环境。

每个环境必须实现定义在文件 rllab/envs/base.py 中的方法、属性:

class Env(object):
def step(self, action):
"""
Run one timestep of the environment's dynamics. When end of episode
is reached, reset() should be called to reset the environment's internal state. 执行环境转移动态的一个时间步,当回合的结尾到达,reset() 必须调用重置环境内部状态
Input
-----
action : an action provided by the environment
Outputs
-------
(observation, reward, done, info)
observation : agent's observation of the current environment
reward [Float] : amount of reward due to the previous action
done : a boolean, indicating whether the episode has ended
info : a dictionary containing other diagnostic information from the previous action
"""
raise NotImplementedError

def reset(self):
"""
Resets the state of the environment, returning an initial observation. 重置环境状态,返回初始观察状态
Outputs
-------
observation : the initial observation of the space. (Initial reward is assumed to be 0.) 初始状态,初始回报为 0
"""
raise NotImplementedError

@property
def action_space(self):
"""
Returns a Space object 返回一个行动空间对象
"""
raise NotImplementedError

@property
def observation_space(self):
"""
Returns a Space object 返回一个状态空间对象
"""
raise NotImplementedError

我们会实现一个简单的 2D 状态和 2D 行动的环境。其目标是控制点机器人在 2D 中移动到源点。我们接受在 2D 平面 (x,y)∈ℝ<sup>2</sup> 点机器人的位置。行动是他的速度 (x˙,y˙)∈ℝ<sup>2</sup>,满足 |x˙|≤0.1 和 |y˙|≤0.1。我们通过定义奖励为负的到原点的距离鼓来励机器人往源点移动。r(x,y)=−sqrt(x<sup>2</sup>+y<sup>2</sup>)。

我们现在来创建环境的文件,假设被放在 example/point_env.py 中。首先,声明一个继承自基环境的类,加入一些 import 内容:

from rllab.envs.base import Env
from rllab.envs.base import Step
from rllab.spaces import Box
import numpy as np
class PointEnv(Env):

# ...

对每个环境,我们需要指定合法状态集合和合法行动集合。这个是通过下面的属性方法实现:

class PointEnv(Env):

# ...

@property
def observation_space(self):
return Box(low=-np.inf, high=np.inf, shape=(2,))

@property
def action_space(self):
return Box(low=-0.1, high=0.1, shape=(2,))

现在已经做到了!然后通过下面给诊断脚本来模拟环境:

python scripts/sim_env.py examples.point_env --mode random

随机行动模拟下,从均匀分布中采样。

也可以使用一个神经网络策略来解决这个问题,当然是远远胜任了。可以创建下面的代码的脚本:

from rllab.algos.trpo import TRPO
from rllab.baselines.linear_feature_baseline import LinearFeatureBaseline
from examples.point_env import PointEnv
from rllab.envs.normalized_env import normalize
from rllab.policies.gaussian_mlp_policy import GaussianMLPPolicy

env = normalize(PointEnv())
policy = GaussianMLPPolicy(
env_spec=env.spec,
)
baseline = LinearFeatureBaseline(env_spec=env.spec)
algo = TRPO(
env=env,
policy=policy,
baseline=baseline,
)
algo.train()

假设这个文件是 examples/trpo_point.py。你可以运行下面的脚本:

python examples/trpo_point.py
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s