Reinforcement Learning Exercise 4.7

Exercise 4.7 (programming) Write a program for policy iteration and re-solve Jack’s car rental problem with the following changes. One of Jack’s employees at the first location rides a bus home each night and lives near the second location. She is happy to shuttle one car to the second location for free. Each additional car still costs $2, as do all cars moved in the other direction. In addition, Jack has limited parking space at each location.If more than 10 cars are kept overnight at a location (after any moving of cars), then an additional cost of $4 must be incurred to use a second parking lot (independent of how many cars are kept there). These sorts of nonlinearities and arbitrary dynamics often occur in real problems and cannot easily be handled by optimization methods other than dynamic programming. To check your program, first replicate the results given for the original problem.

First we can give out the code of Jack’s car rental problem, then just a few modification is needed to solve this problem.
Code of Jack’s car rental problem:

#[car_rental.py]
#==================================================================
# Python3
# Copyright
# 2019 Ye Xiang (xiang_ye@outlook.com)
#==================================================================

import scipy.stats as sci_stats
import numpy as np
import matplotlib as mpltlib
import matplotlib.pyplot as mtplt
import seaborn as sbn
import math

class car_rental:
    def __init__(self):
        self.__init__all()

    def __init__all(self):
        self._max_car_num = 20 # maximum car number for each location
        self._gamma = 0.9      # discount rate
        self.__max_transfer_car_num = 5
        self.__accuracy = 0.01
        self._transfer_cost_per_car = 2
        self.__rental_credit_per_car = 10

        #Initialize policy and V(s) as zeros in 21X21 table, becauseof no more than 20 cars for each loaction
        self.__Vs = np.zeros((self._max_car_num + 1,self._max_car_num + 1))
        self.__policy = np.zeros(self.__Vs.shape, dtype = np.int32)

        #Record policy changes in iteration
        self.__history_policy = []

        self._requested_prob1 = self.__init_probabilities(self._max_car_num + 1, 3)
        self._returned_prob1  = self.__init_probabilities(self._max_car_num + 1, 3)
        self._requested_prob2 = self.__init_probabilities(self._max_car_num + 1, 4)
        self._returned_prob2  = self.__init_probabilities(self._max_car_num + 1, 2)

        #The probability that the request car number >= the car number in the morning.
        self._req_over_boundary_prob1 = self.__init_over_boundary_prob(self._requested_prob1)
        self._req_over_boundary_prob2 = self.__init_over_boundary_prob(self._requested_prob2)
        #The probability that the returned car number makes the car number >= the maximum car num in the evening.
        self._ret_over_boundary_prob1 = self.__init_over_boundary_prob(self._returned_prob1)
        self._ret_over_boundary_prob2 = self.__init_over_boundary_prob(self._returned_prob2)

    def __init_probabilities(self, array_size, poisson_lambda):
        arr = np.zeros(array_size)
        for i in range(0, array_size):
            arr[i] = sci_stats.poisson.pmf(k = i, mu = poisson_lambda)
        return arr

    def __init_over_boundary_prob(self, prob_in_boundary):
        # For example, when requested car number is 3, the car number in morning is 2, the self._req_over_boundary_prob1[2] stands for 
        # all the request car number that makes the remain car number is 0, include requested car number is 2, 3, 4 & ...
        # so self._req_over_boundary_prob1[2] = 1 - self._requseted_prob1[0]- self._requseted_prob1[1]
        prob = np.zeros(len(prob_in_boundary) + 1)
        tmp = 0
        prob[0] = 1
        for i in range(1, len(prob_in_boundary) + 1):
            tmp += prob_in_boundary[i - 1]
            prob[i] = 1 - tmp
        return prob

    def _get_rental_reward(self, request_num, car_num):
        return min(self._max_car_num, min(request_num, car_num)) * self.__rental_credit_per_car

    def _get_action_cost(self, action):
        return abs(action) * self._transfer_cost_per_car
    

    def _calculate_Vs_by_policy(self, Vs,car_num1_in_evening, car_num2_in_evening, action):
        tmp_vs = 0.0