Modulation classification, an intermediate step between signal detection and demodulation, is commonly deployed in many modern wireless communication systems. Although many approaches have been introduced in the last decades for identifying the modulation format of the incoming signal, they have the obstacle of mining radio characteristics for most traditional machine learning algorithms. To effectively handle this limitation, we propose an accurate modulation classification method by exploiting deep learning for being compatible with constellation diagram. A convolutional neural network (CNN), namely CRNet, is developed to proficiently learn the most relevant radio characteristics from transformed gray-scale constellation image by cross-residual connection, a novel structure for associating the intrinsic information between two processing flows specified by regular and grouped convolutional layers. Based on the experimental evaluation, CRNet achieves the classification rate of approximately 90% at +10 dB signal-to-noise ratio (SNR) under a multipath Rayleigh fading channel and further performs more accurately than some existing deep models for constellation-based modulation classification.