Rutting is one of the main failure modes of asphalt pavement. It is very important to study the rutting deformation evolution for the anti-rutting design of pavement. Burgers model is adopted to describe the viscoelasticity of asphalt matrix, and the model parameters are determined according to the laboratory data. The geometric models of test-pieces are built with the help of the digital image processing method. The discrete element method is used to investigate the rutting depth variation of asphalt mixture with aggregate distribution, load and temperature. It is found that temperature has an evident effect on rutting deformation near the softening point of asphalt but only a slight effect when far away from the softening point. At lower temperature, the mixture behaves good integrity, the aggregate distribution has a tiny effect on local rutting deformation, and the rutting increment caused by overloading is unobvious. However, when the softening point is reached or exceeded, the integrity of the mixture declines distinctly, the aggregate distribution has a significant effect on local rutting deformation, and the rutting increment caused by overloading is pretty remarkable. Therefore, rutting deformation will increase markedly when overloading and high temperature are combined, which can pose serious potential hazards to asphalt pavement.