通过简易迷宫问题，理解Monte-Carlo, Sarsa和Q-learning算法
3×33\times33×3的迷宫如下
可以通过每一步都随机地走，直到走到S8为止
这里主要写通过强化学习，找到最佳路线
MC算法：
首先定义一个policy，它是一个矩阵，行代表S1～S8
列代表action的4个方向，这里定义方向为↑、→、↓、←的顺序
policy定义为πθ(s,a)\pi_{\theta}(s, a)πθ(s,a), 其中θ\thetaθ是决定π\piπ的参数，首先定义初始θ0\theta_{0}θ0
θ0\theta_{0}θ0矩阵中值为1代表这个方向可以走，如果是墙壁，就定义为np.nan
#row: S0~S7, column: ↑、→、↓、←
#theta: parameter of policy pi(s, a)
theta_0 = np.array([[np.nan, 1, 1, np.nan], #S0
[np.nan, 1, np.nan, 1], #S1
[np.nan, np.nan, 1, 1], #S2
[1, 1, 1, np.nan], #S3
[np.nan, np.nan, 1, 1], #S4
[1, np.nan, np.nan, np.nan], #S5
[1, np.nan, np.nan, np.nan], #S6
[1, 1, np.nan, np.nan], #S7
#S8 is goal, no policy
])
下面由θ0\theta_{0}θ0求πθ(s,a)\pi_{\theta}(s, a)πθ(s,a)，可以单纯求比例，也可以用softmax
soft公式如下：
P(θi)=exp(βθi)∑i=1Naexp(βθi)P(\theta_{i}) = \frac{exp(\beta\theta_{i})}{\sum_{i=1}^{N_{a}}exp(\beta\theta_{i})}P(θi)=∑i=1Naexp(βθi)exp(βθi)
NaN_{a}Na表示action的种类数（4个）
#use softmax to get policy
def softmax_convert_into_pi_from_theta(theta):
beta = 1.0
[m, n] = theta.shape
pi = np.zeros((m,n))
exp_theta = np.exp(beta * theta)
for i in range(0, m):
pi[i, :] = exp_theta[i, :] /np.nansum(exp_theta[i, :])
pi = np.nan_to_num(pi)
return pi
打印初期πθ\pi_{\theta}πθ可以看到在不同的state下不同action的概率
pi_0 = softmax_convert_into_pi_from_theta(theta_0)
print(pi_0)
#output:
#[[ 0. 0.5 0.5 0. ]
# [ 0. 0.5 0. 0.5 ]
# [ 0. 0. 0.5 0.5 ]
# [ 0.33333333 0.33333333 0.33333333 0. ]
# [ 0. 0. 0.5 0.5 ]
# [ 1. 0. 0. 0. ]
# [ 1. 0. 0. 0. ]
# [ 0.5 0.5 0. 0. ]]
通过policy来确定agent的动作
通过π\piπ的概率来选择下一步的action和state
def get_action_and_next_s(pi, s):
direction = ["up", "right", "down", "left"]
next_direction = np.random.choice(direction, p=pi[s, :])
if next_direction == "up":
action = 0
s_next = s - 3
elif next_direction == "right":
action = 1
s_next = s + 1
elif next_direction == "down":
action = 2
s_next = s + 3
elif next_direction == "left":
action = 3
s_next = s - 1
return [action, s_next]
直到走到Goal，一直不停探索路径，到达goal的路径保存在s_a_history
def goal_maze_ret_s_a(pi):
s = 0 #start
s_a_history = [[0, np.nan]]
while(1):
[action, next_s] = get_action_and_next_s(pi, s)
s_a_history[-1][1] = action
#print([action, next_s])
s_a_history.append([next_s, np.nan]) #now don't know next action, add np.nan as next action
if next_s == 8:
break
else:
s = next_s
return s_a_history
随机走一条路径
s_a_history = goal_maze_ret_s_a(pi_0)
print(s_a_history)
#output:
#[[0, 2], [3, 0], [0, 2], [3, 0], [0, 1], [1, 3], [0, 2], [3, 0], [0, 2], [3, #1], [4, 2], [7, 0], [4, 3], [3, 1], [4, 2], [7, 1], [8, nan]]
#17 steps
MC算法需要完整的路径来更新policy，所以每次得到完整路径后去更新θ\thetaθ，通过θ\thetaθ得到π\piπ，
重复上面的步骤直到policy收敛
θ\thetaθ的更新公式：
θsi,aj=θsi,aj+η.Δθsi,aj\theta_{s_{i},a_{j}} = \theta_{s_{i},a_{j}} + \eta.\Delta\theta_{s_{i},a_{j}}θsi,aj=θsi,aj+η.Δθsi,aj
Δθsi,aj={N(si,aj)+P(si,aj)N(si,a)}/T\Delta\theta_{s_{i},a_{j}} = \{N(s_{i}, a_{j}) + P(s_{i}, a_{j})N(s_{i}, a)\}/TΔθsi,aj={N(si,aj)+P(si,aj)N(si,a)}/T
NNN表示次数，TTT表示到达goal的总step数
def update_theta(theta, pi, s_a_history):
eta = 0.1 #learning rate
T = len(s_a_history) - 1
[m, n] = theta.shape
delta_theta = theta.copy() #deep copy
for i in range(0, m):
for j in range(0, n):
if not(np.isnan(theta[i, j])):
SA_i = [SA for SA in s_a_history if SA[0] == i] #the (s,a) pair with state i
SA_ij = [SA for SA in s_a_history if SA == [i, j]] #the (s,a) pair with state i, action j
N_i = len(SA_i)
N_ij = len(SA_ij)
delta_theta[i, j] = (N_ij + pi[i, j] * N_i) / T
new_theta = theta + eta * delta_theta
return new_theta
MC算法迭代，定义收敛标准
stop_epsilon = 10**-8
theta = theta_0
pi = pi_0
is_continue = True
count = 1
while is_continue:
s_a_history = goal_maze_ret_s_a(pi)
new_theta = update_theta(theta, pi, s_a_history)
new_pi = softmax_convert_into_pi_from_theta(new_theta)
print(np.sum(np.abs(new_pi - pi)))
print("steps to reach goal is: " + str(len(s_a_history) - 1))
if np.sum(np.abs(new_pi - pi)) < stop_epsilon:
is_continue = False
else:
theta = new_theta
pi = new_pi
最后确认下收敛后的policy
np.set_printoptions(precision=3, suppress=True) #not display by e
print(pi)
#output
#[[ 0. 0. 1. 0. ]
# [ 0. 0.38 0. 0.62 ]
# [ 0. 0. 0.474 0.526]
# [ 0. 1. 0. 0. ]
# [ 0. 0. 1. 0. ]
# [ 1. 0. 0. 0. ]
# [ 1. 0. 0. 0. ]
# [ 0. 1. 0. 0. ]]
可以看到policy已经可以找到最佳路径，而未收敛的概率处是因为通过最优路径不会再采取那个action，所以仍保留初始附近的概率
Sarsa算法
名称解释，Sarsa指当前State(S), Action(a)得到即时Reward( r), 并通过下一步的State(s), Action(a)得到预期reward
定义Q矩阵，通过ϵ−greedy\epsilon-greedyϵ−greedy方法选择下一步的action, 在每一个step都更新Q值，而不是MC算法的每个episode更新policy
最后Q矩阵收敛可以得到最优路径
Q矩阵的size和上面policy一样，行代表state，列代表action，初始值为random
但是为了保持墙壁的限制条件，让Q乘θ0\theta_{0}θ0
将Q的terminal state设为0
#row: S0~S7, column: ↑、→、↓、←
#theta: policy pi(s, a)
theta_0 = np.array([[np.nan, 1, 1, np.nan], #S0
[np.nan, 1, np.nan, 1], #S1
[np.nan, np.nan, 1, 1], #S2
[1, 1, 1, np.nan], #S3
[np.nan, np.nan, 1, 1], #S4
[1, np.nan, np.nan, np.nan], #S5
[1, np.nan, np.nan, np.nan], #S6
[1, 1, np.nan, np.nan], #S7
#S8 is goal, no policy
])
[a, b] = theta_0.shape
Q = np.random.rand(a, b) * theta_0
Q[7,3]=0
print(Q)
#output:
[[ nan 0.745 0.711 nan]
[ nan 0.286 nan 0.385]
[ nan nan 0.513 0.056]
[ 0.501 0.921 0.993 nan]
[ nan nan 0.891 0.01 ]
[ 0.875 nan nan nan]
[ 0.439 nan nan nan]
[ 0.958 0.936 nan 0. ]]
ϵ−greedy\epsilon-greedyϵ−greedy（选择下一步的action）：
greedy表示每次选择Q[s, :]中的最大值作为下一步的action，但是它有一个问题，就是容易陷入local optimal，所以要平衡随机探索和greedy
于是以ϵ\epsilonϵ的概率进行随机探索，1−ϵ1-\epsilon1−ϵ的概率采取greedy
ϵ\epsilonϵ随着step增加而衰减
而且有一个理论，也就是说ϵ−greedy\epsilon-greedyϵ−greedy是可以改进policy的：
for any ϵ−greedy\epsilon-greedyϵ−greedy policy π\piπ, ϵ−greedy\epsilon-greedyϵ−greedy policy π′\pi'π′ with respect to qπq_{\pi}qπ is an improvement. Vπ′(s)≥Vπ(s)V_{\pi'}(s) \geq V_{\pi}(s)Vπ′(s)≥Vπ(s)
证明就不写了
由于涉及到随机探索，所以仍然需要探索用的policy
这里不用softmax，而是用简单的概率
def simple_convert_into_pi_from_theta(theta):
[m, n] = theta.shape
pi = np.zeros((m, n))
for i in range(0, m):
pi[i, :] = theta[i, :] / np.nansum(theta[i, :])
pi = np.nan_to_num(pi)
return pi
pi_0 = simple_convert_into_pi_from_theta(theta_0)
实现ϵ−greedy\epsilon-greedyϵ−greedy：
#epsilon-greedy
#get the action
def get_action(s, Q, epsilon, pi_0):
direction = ["up", "right", "down", "left"]
#print("s = " + str(s))
#probability epsilon to random search
if np.random.rand() < epsilon:
next_direction = np.random.choice(direction, p=pi_0[s, :])
else:
#move by the maximum Q
next_direction = direction[np.nanargmax(Q[s, :])]
if next_direction == "up":
action = 0
elif next_direction == "right":
action = 1
elif next_direction == "down":
action = 2
elif next_direction == "left":
action = 3
return action
#get next state by action
def get_s_next(s, a, Q, epsilon, pi_0):
direction = ["up", "right", "down", "left"]
next_direction = direction[a]
if next_direction == "up":
s_next = s - 3
elif next_direction == "right":
s_next = s + 1
elif next_direction == "down":
s_next = s + 3
elif next_direction == "left":
s_next = s - 1
return s_next
通过state, action更新Q矩阵的Sarsa算法：
推导：
μk=1k∑j=1kxj\mu_{k} = \frac{1}{k}\sum_{j=1}^{k}x_{j}μk=k1∑j=1kxj
=1k(xk+∑j=1k−1xj)=\frac{1}{k}(x_{k} + \sum_{j=1}^{k-1}x_{j})=k1(xk+∑j=1k−1xj)
=1k(xk+(k−1)μk−1)=\frac{1}{k}(x_{k} + (k-1)\mu_{k-1})=k1(xk+(k−1)μk−1)
=μk−1+1k(xk−μk−1)=\mu_{k-1} + \frac{1}{k}(x_{k} -\mu_{k-1})=μk−1+k1(xk−μk−1)
for each state StS_{t}St with GtG_{t}Gt:
N(St)←N(St)+1N(S_{t})\leftarrow N(S_{t})+1N(St)←N(St)+1
V(St)←V(St)+1N(St)(Gt−V(St))V(S_{t})\leftarrow V(S_{t}) + \frac{1}{N(S_{t})}(G_{t}-V(S_{t}))V(St)←V(St)+N(St)1(Gt−V(St))
Gt=Rt+1+γV(St+1)G_{t}=R_{t+1} +\gamma V(S_{t+1})Gt=Rt+1+γV(St+1)
得到：
Q(st,at)=Q(st,at)+η∗(Rt+1+γQ(st+1,at+1)−Q(st,at))Q(s_{t}, a_{t}) = Q(s_{t}, a_{t}) + \eta*(R_{t+1} +\gamma Q(s_{t+1},a_{t+1})-Q(s_{t}, a_{t}))Q(st,at)=Q(st,at)+η∗(Rt+1+γQ(st+1,at+1)−Q(st,at))
其中Rt+1+γQ(st+1,at+1)−Q(st,at)R_{t+1} +\gamma Q(s_{t+1},a_{t+1})-Q(s_{t}, a_{t})Rt+1+γQ(st+1,at+1)−Q(st,at)叫做TD error
根据上述公式更新Q：
def Sarsa(s, a, r, s_next, a_next, Q, eta, gamma):
if s_next == 8: #the goal, next state not exist
Q[s, a] = Q[s, a] + eta * (r - Q[s, a])
else:
Q[s, a] = Q[s, a] + eta * (r + gamma*Q[s_next, a_next] - Q[s, a])
return Q
一个episode:
def goal_maze_ret_s_a_Q(Q, epsilon, eta, gamma, pi):
s = 0 #start state
a = a_next = get_action(s, Q, epsilon, pi) #get action by epsilon-greedy
s_a_history = [[0, np.nan]]
while(1): #until reach the goal
a = a_next
s_a_history[-1][1] = a
s_next = get_s_next(s, a, Q, epsilon, pi)
s_a_history.append([s_next, np.nan])
if s_next == 8:
r = 1 #get the reward
a_next = np.nan
else:
r = 0
a_next = get_action(s_next, Q, epsilon, pi)
Q = Sarsa(s, a, r, s_next, a_next, Q, eta, gamma) #update Q
if s_next == 8:
break
else:
s = s_next
return [s_a_history, Q]
完整迭代：
eta = 0.1
gamma = 0.9
epsilon = 0.5
v = np.nanmax(Q, axis=1) #select the maximum Q value for each state
is_continue = True
episode = 1
while is_continue:
print("episode: " + str(episode))
epsilon = epsilon / 2 #epsilon-greedy
[s_a_history, Q] = goal_maze_ret_s_a_Q(Q, epsilon, eta, gamma, pi_0)
new_v = np.nanmax(Q, axis=1) #maximum value for each state
print(np.sum(np.abs(new_v - v)))
v = new_v
print("steps to reach goal is: " + str(len(s_a_history) - 1))
episode = episode + 1
if episode > 100:
break
虽然写了到100 episode，但是从output看出很快就收敛到最优路径
episode: 1
0.227489819094
steps to reach goal is: 14
episode: 2
0.154785251991
steps to reach goal is: 10
episode: 3
0.0105243446365
steps to reach goal is: 4
episode: 4
0.00946282714278
steps to reach goal is: 4
episode: 5
0.00848520483778
steps to reach goal is: 4
episode: 6
0.00758386849718
steps to reach goal is: 4
episode: 7
0.0051117594454
steps to reach goal is: 4
episode: 8
看下收敛后的路径及Q矩阵
print(s_a_history)
#output:
[[0, 2], [3, 1], [4, 2], [7, 1], [8, nan]]
print(Q)
#output:
[[ nan 0.705 0.729 nan]
[ nan 0.286 nan 0.41 ]
[ nan nan 0.513 0.056]
[ 0.501 0.81 0.775 nan]
[ nan nan 0.9 0.088]
[ 0.875 nan nan nan]
[ 0.575 nan nan nan]
[ 0.928 1. nan 0. ]]
标准Sarsa只能backward一步，而Sarsa(λ)\lambda)λ)算法一次可以更新整个路径，与Sarsa不同的是加了一个EEE矩阵
（这个单独写）
Q-learning
Q-learning与Sarsa的区别：
对比Q的更新公式：
Sarsa：
Q(st,at)=Q(st,at)+η∗(Rt+1+γQ(st+1,at+1)−Q(st,at))Q(s_{t}, a_{t}) = Q(s_{t}, a_{t}) + \eta*(R_{t+1} +\gamma Q(s_{t+1},a_{t+1})-Q(s_{t}, a_{t}))Q(st,at)=Q(st,at)+η∗(Rt+1+γQ(st+1,at+1)−Q(st,at))
Q-learning:
Q(st,at)=Q(st,at)+η∗(Rt+1+γmaxaQ(st+1,a)−Q(st,at))Q(s_{t}, a_{t}) = Q(s_{t}, a_{t}) + \eta*(R_{t+1} +\gamma \mathop{\max}\limits_{a }Q(s_{t+1},a)-Q(s_{t}, a_{t}))Q(st,at)=Q(st,at)+η∗(Rt+1+γamaxQ(st+1,a)−Q(st,at))
可以看到Sarsa更新时需要求并且使用at+1a_{t+1}at+1，而Q-learning需要用到st+1s_{t+1}st+1对应的最大Q值。
at+1a_{t+1}at+1是由决定下一步action的policy决定的，所以Q的更新依赖于policy，称为online-policy
而Q-learning不依赖于决定action的policy，称为offline-policy
Sarsa中用到的ϵ−greedy\epsilon-greedyϵ−greedy中random的部分Q-learning中没有，所以Q-learning有比Sarsa收敛更快的特征
按照Q-learning公式：
def Q_learning(s, a, r, s_next, Q, eta, gamma):
if s_next == 8: #goal
Q[s, a] = Q[s, a] + eta*(r - Q[s, a])
else:
Q[s, a] = Q[s, a] + eta*(r + gamma*np.nanmax(Q[s_next,:]) - Q[s, a])
return Q
因为不用ϵ−greedy\epsilon-greedyϵ−greedy，所以action函数去掉随机的部分：
#get the action
def get_action(s, Q):
direction = ["up", "right", "down", "left"]
#print("s = " + str(s))
next_direction = direction[np.nanargmax(Q[s, :])]
if next_direction == "up":
action = 0
elif next_direction == "right":
action = 1
elif next_direction == "down":
action = 2
elif next_direction == "left":
action = 3
return action
def get_s_next(s, a):
direction = ["up", "right", "down", "left"]
next_direction = direction[a]
if next_direction == "up":
s_next = s - 3
elif next_direction == "right":
s_next = s + 1
elif next_direction == "down":
s_next = s + 3
elif next_direction == "left":
s_next = s - 1
return s_next
一次episode:
def goal_maze_ret_Q_learning(Q, eta, gamma):
s = 0 #start state
a = a_next = get_action(s, Q) #get action by maximum Q value
s_a_history = [[0, np.nan]]
while(1): #until reach the goal
a = a_next
s_a_history[-1][1] = a
s_next = get_s_next(s, a)
s_a_history.append([s_next, np.nan])
if s_next == 8:
r = 1 #get the reward
a_next = np.nan
else:
r = 0
a_next = get_action(s_next, Q)
Q = Q_learning(s, a, r, s_next, Q, eta, gamma) #update Q
if s_next == 8:
break
else:
s = s_next
return [s_a_history, Q]
完整迭代到100次：
eta = 0.1 #learning-rate
gamma = 0.9 #decrease rate
v = np.nanmax(Q, axis=1) #maximum value for each state
is_continue = True
episode = 1
V = [] #state value for each episode
V.append(np.nanmax(Q, axis=1)) #get the maximum value for each state
while is_continue:
print("episode " + str(episode))
[s_a_history, Q] = goal_maze_ret_Q_learning(Q, eta, gamma) #get one path
new_v = np.nanmax(Q, axis=1)
print(np.sum(np.abs(new_v - v))) #get the error
v = new_v
V.append(v)
print("steps to reach goal: " + str(len(s_a_history) - 1))
episode = episode + 1
if episode > 100:
is_continue = False
收敛结果
episode 1
0.100490460992
steps to reach goal: 20
episode 2
0.0969381947436
steps to reach goal: 16
episode 3
0.0943713455446
steps to reach goal: 8
episode 4
0.0932547392558
steps to reach goal: 6
episode 5
0.0918454990963
steps to reach goal: 6
episode 6
0.0904328729472
steps to reach goal: 4
episode 7
0.0895058129093
steps to reach goal: 4
episode 8
0.0885468091866
steps to reach goal: 4
episode 9
0.0875441463751
steps to reach goal: 4
episode 10
0.0864873768061
steps to reach goal: 4
episode 11
0.0853677368575
steps to reach goal: 4
episode 12
0.0841783707317
steps to reach goal: 4
episode 13
0.0829144098366
steps to reach goal: 4
episode 14
0.0815729464098
steps to reach goal: 4
episode 15
0.0801529321705
steps to reach goal: 4
episode 16
0.0786550263091
steps to reach goal: 4
episode 17
0.0770814118009
steps to reach goal: 4
episode 18
0.0754355946909
steps to reach goal: 4
episode 19
0.0737221974552
steps to reach goal: 4
episode 20
0.0719467546908
steps to reach goal: 4
436
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
