The paper presents a method for using differentiable physics to perform planar pushing tasks. By utilizing a differentiable formulation of planar pushing equations of motion, the robot can simultaneously push an object to the desired target and identify the internal mass distribution of the object. The derivations appear to be correct (though I admit I did not check them in detail), and the overall method comes across as well thought-out and very reasonable. The complete system is also logically designed, and evaluated well. The results seem compelling. Overall, I think this is a reasonably strong paper. My main reservations about this work are somewhat higher-level. First, with modern automatic differentiation tools, actually building a differentiable simulator, especially under the restricted settings considered in this paper, is not actually all that difficult. That's not a bad thing -- a contribution does not have to be difficult to be valuable. But I'm finding it hard not to think that the paper somewhat overcomplicates matters with two pages of dense linear algebra. Would it not be enough to describe the forward simulation formulation (which is not new, and based on textbook-standard equations), and then just state that the method relies on differentiating through these equations of motion? The derivatives are complicated, but not hard to automate. Second, I can't help but think that, though the notion of differentiable physics for robotic control is very promising, in some sense this paper takes the "easy way out" by considering a restricted setting where many of the most challenging facets of this problem (establishing and breaking contacts in 3D, non-convex and complex collision shapes, etc.) are removed by construction, and therefore the lessons from this paper might be difficult to generalize to more general 3D manipulation scenarios. Lastly, I can't help but think that, for the demo that is actually shown in the accompanying video, this method is a bit overkill -- while I really do appreciate the technical aspects of the approach and I think it's interesting and valuable, in the end it enables a robot to push a hammer to the edge of a table. Somehow it seems like tasks like this can be solved more easily in other ways, for example by relying more heavily on feedback as opposed to detailed modeling and system identification. This is of course a bit of a digression, and I do think the paper should be judged on its merits, but perhaps this remark might suggest to the authors that a more convincing evaluation would increase the impact of the work.

Originality =========== The idea is original. Representing a non-homogeneous object as a collection of homogeneous cubes is not very novel, but the application to estimating friction and mass seems original. Quality ======= The theory seems sound, the experiments are convincing. Clarity ======= The paper reads OK. The detailed derivations make the paper hard to read. I'd suggest shifting some of them to an appendix and add a few illustrations instead in the main paper. The assumptions and limitations need to be pointed out more clearly. The planning and control approach is also somewhat unclear. Significance ============ An important step towards efficient system identification for sliding objects. Major Comments: =============== - Sect. III: If I understand that correctly, you take the maximum extensions of the object as seen from above and project them down on the surface to get the shape for the cuboids. An illustration would help here. Why do we need a partial 3D view, in the end it seems like everything is done with a single layer of cuboids (so just background subtraction and an overhead camera would suffice). - I'd make more explicit earlier on that everything is purely 2D, i.e., you don't estimate the height of the center of mass etc. - Please point the assumptions and limitations of the model out more explicitly. In the end we only have a friction coefficient for Coulomb friction. So the model cannot identify more complex friction effects like stiction, which might actually make a big difference between the careful, small pokes for system identification and the long pushes for moving the object in the end. - It should also be pointed out earlier that the forces are always parallel to the surface. - Sect. VI: Is a bit vague, if this is about a policy gradient approach, explain it in RL terms (reward, state, action, policy). Maybe just call it gradient-based optimization. It is also very vague what the gradient actually is. - Fig. 3: The mass distribution of the book seems rather strange, you'd expect almost no mass on the left half (cover) while the estimated distribution has a clear diagonal structure. This diagonal structure is also present in the other objects. Where does this come from? An artefact of how data is collected? - Sect. VIII: It would also be nice to include a figure illustrating the results of planning and control, e.g., push arrows around an object that are color-coded according to their quality. Minor Comments: =============== - Equation 1 -> Eqaution (1) , use \eqref{} - Algo 2: the formatting of multi-letter subscripts is very inconsistent (e.g. _waypoint vs. _left), as is the formatting of the variable "improvement"

This is a beautiful paper. I think the experimental evaluation of contributions (1) and (2) is compellingly shown in Fig 3. The results section is comprehensive, the paper is well written. As far as I can tell contribution (1) is original, but the idea of differentiating a simulator with LCP constraints was recently shown by Amos and Kolter (https://arxiv.org/abs/1810.13400). It is worth clarifying how high-resolution can we make the 3D voxel grids representing the object, while still being able to estimate the mass and the friction.