Multispectral images have numerous features and a wide range of applications. However, traditional image compression methods, such as JPEG2000 and 3D-SPIHT, do not make effective use of spectral information. We propose a deep compression framework based on interspectral prediction to take full advantage of spectral correlation when using temporal correlation for interframe prediction in video compression. First, two-dimensional and three-dimensional convolutions were used to obtain spatial and spectral information for predicting the original image. Then, we applied a residual neural network to compress the residual information of the image. Subsequently, a decoder was employed to reconstruct the multispectral image based on the compressed image and residual information. All components were jointly trained by a single loss function that considered the tradeoff between the compression bit rate and decoded image quality. The experimental results showed that our proposed method outperformed other traditional compression algorithms, including JPEG2000, 3D-SPIHT, and PCA+JPEG2000, in terms of peak signal-to-noise ratio and spectral angle and is equivalent to or even better than some image compression algorithms based on deep neural networks.