Final QL MDP Results

Here are the final results for the QL MDP. I have also plotted on the same graph the results from the Exact MDP for comparison. As you can see, the QL MDP gives up some performance, and tends to result in a slightly higher total cost than the Exact MDP.

One peculiar behavior that was observed was the that the QL MDP always exhibited compute-move-compute-move-compute-move policies. I wonder if moving helps it eliminate some of the aliasing of the states? Is it intelligently deciding to move to a less ambiguous state?