This paper proposes an action representation (called ‘spatial action maps’) for robots learning to manipulate objects using deep reinforcement learning. This work is inspired by previous works using dense action representations. An agent is trained, using simulation, to push objects to a target location. While a standard algorithm is used for training (DDQN), the policy is represented using a Fully Convolutional neural net. Experimental results using a few baselines show some promise of the proposed approach. The paper in general is well-written, and I was very excited when reading sections I and II. This excitement decreased from section III. For example, the reward function assumes that the distance between objects and the target location is known — which may not be the case in more complex scenarios. In addition and most importantly, the experimental setting is very toy-like environment. This environment assumes that the action representation is available in one image, which is a strong assumption in more complex (real-world) environments. Last but not least, the experiments do not take into account strong baselines to compare against. It would have been interesting to see methods X, Y and Z with and without spatial action maps, where the use of spatial action maps makes a substantial difference — not only in a toy environment but in a more realistic one.

Originality The authors propose a novel representation for actions in mobile manipulation settings and discuss several advantages of the proposed representation. They also present an empirical study that showcases the advantages of the approach. The action representation is indeed novel, to the best of my knowledge. Quality The results are impressive and the evaluation is thorough, especially the ablations. Figure 7 clearly shows the value of the proposed action space. Taken together, Tables 1,2,3, and 4 all show the effect of different kinds of ablations (using straight line paths instead of shortest paths in the movement primitives, using a fixed step size, etc). A consistent trend is that the design choices make the most difference in the Large Divider environment - this makes sense since the agent must navigate around the large divider in order to push all of the blocks successfully. The section on limitations of the approach is an important inclusion and is appreciated. As for the supplementary website, the videos are useful to watch. The emergent behavior of grouping items against the wall and then sweeping multiple objects with long trajectory is indeed interesting, as discussed by the authors. Clarity Overall, the authors provide a very thorough explanation of their method, the state and action representations used, and their results. One minor point - the paper could use more details on how gradients are passed only through the state pixel corresponding to a selected action pixel. Significance Overall, this is a good paper that proposes a nice idea for an action space, and presents thorough validation that the proposed action space outperforms other choices.