Defining a Gridworld
Many gridworlds represent mazes with start and goal states that the agent needs to solve. Mazes can be easily defined. Here we create the Dyna Maze from Chapter 8 in (Sutton and Barto 2018).
x <- gridworld_maze_MDP(
dim = c(6,9),
start = "s(3,1)",
goal = "s(1,9)",
walls = c("s(2,3)", "s(3,3)", "s(4,3)",
"s(5,6)",
"s(1,8)", "s(2,8)", "s(3,8)"),
goal_reward = 1,
step_cost = 0,
restart = TRUE,
discount = 0.95,
name = "Dyna Maze",
)
x
#> MDP, list - Dyna Maze
#> Discount factor: 0.95
#> Horizon: Inf epochs
#> Size: 47 states / 5 actions
#> Start: s(3,1)
#>
#> List components: 'name', 'discount', 'horizon', 'states', 'actions',
#> 'transition_prob', 'reward', 'info', 'start'Gridworlds are implemented with state names
"s(<row>,<col>)", where row and
col are locations in the matrix representing the gridworld.
The actions are "up", "right",
"down", and "left". Conversion between state
labels and the position in the matrix (row and column index) can be done
with gridworld_s2rc() and gridworld_rc2s(),
respectively.
The transition graph can be visualized. Note, the transition from the state below the goal state back to the start state shows that the maze restarts the agent once it reaches the goal and collects the goal reward.
A more general way to create gridworlds is implemented in the
function gridworld_init() which initializes a new gridworld
creating a matrix of states with given dimensions. Unreachable stats and
absorbing state can be defined. The returned information can be used to
build a custom gridworld MDP.
The solver creates a warning indicating that the solution has not
converged after only 5 iterations. In the visualization, we see that
value iteration has expanded values from the goal state up to 5 squares
away. To make this analysis easier, we can use
R markdown documents can use 
