As one of the most promising stimulus-responsive and smart polymer materials, shape memory polymer (SMP) has attracted great attention in the research field. There are three major research topics on the physics and mechanics of SMP, summarized as: exchange between the SMP and the surrounding (e.g. heat, energy, etc.), control of shape change (e.g. actuation approach, constructive modeling, etc.) and its soft matter (e.g. thermodynamics, composite, etc.). The shape memory effect (SME) in SMPs results from the thermodynamic behavior of intrinsic molecular structures, is induced (or triggered) by external stimuli, obeys the Arrhenius rule, and is the macroscopic appearance of segmental relaxation behavior. In this review, the physical and mechanical mechanisms of SMPs are presented and discussed, including thermodynamics, entropy effect, mechanical behavior and stimulus-responsive shape recovery behavior, transition temperature, temperature memory effect, multi-field effect and multi-shape memory effect. Finally, the future work of SMP mechanics is forecasted and discussed.