Abstract: Deep learning has received a growing interest in recent years for detecting different types of pavement distresses and automating pavement condition assessment. A proper choice of deep learning models is key for successful pavement assessment applications. In this study, we first present a comprehensive experimental comparison of state-of-the-art image classification models to evaluate their performances on 11 pavement objects classification. Our experiments are conducted in different dimensions of comparison, including deep classifier architecture, effects of network depth, and computational costs. Five convolutional neural network (CNN) classifiers widely used in transportation applications, including VGG16, VGG19, ResNet50, DenseNet121, and a generic CNN (as the control model), are tested with a comprehensive pixel-level annotated dataset for 11 different distress and non-distress classes (UCF-PAVE 2017). In addition, we investigate a simple yet effective approach of encoding contextual information with multi-scale input tiles to classify highly random pavement objects in size, shape, intensity, texture, and direction. Our comparison results show that the multi-scale approach significantly improves the classification accuracy for all compared deep classifiers at a negligible extra computational cost. Finally, we provide recommendations of how to improve the classification performance of deep CNNs for automated pavement condition assessment based on the comparison results.