Abstract: Designing dynamically feasible trajectories for rigid bodies is a fundamental problem in robotics. Although direct trajectory optimization is widely applied to solve this problem, state-of-the-art methods overlook the manifold structures of rigid bodies, resulting in slow convergence. This paper introduces a Riemannian optimization framework for direct trajectory optimization of rigid bodies on matrix Lie groups. The proposed approach first leverages the Lie Group Variational Integrator to formulate the discrete rigid body dynamics. The paper then derives the exact first- and second-order Riemannian derivatives of the dynamics using the matrix Lie group structure. Finally, this work applies a line-search Riemannian Interior Point Method (RIPM) to perform trajectory optimization. The paper demonstrates that both the derivative evaluations and Newton steps required to solve the RIPM exhibit linear complexity with respect to the planning horizon and system degrees of freedom. Simulation results illustrate that the proposed method is an order of magnitude faster than conventional methods.