Accurate and robust short-term traffic prediction is an important part of advanced traveler information systems. With the development of intelligent navigation and autonomous driving, it is necessary to explore the lane-level predictions of traffic speeds. However, most existing traffic prediction models concentrate on forecasting the traffic flow characteristics of the entire road sections rather than those of certain lanes. This paper proposes a fusion deep learning (FDL) model to predict lane-level traffic speed. First, the entropy-based grey relation analysis is introduced to choose lane sections that are strongly correlated with the lane section to be predicted. Second, a two-layer deep learning framework is established by combining the long short-term memory (LSTM) neural network and the gated recurrent unit (GRU) neural network. Third, the ground-truth data of several lane sections captured by remote traffic microwave sensors (RTMS) on the 2nd Ring Road of Beijing are utilized to examine the FDL model and compare it with several benchmark models. The experimental result indicates that in addition to capturing the fluctuations of traffic speed at the lane level, the FDL model has better performance than the benchmark models in terms of prediction accuracy and stability.