Skip to main content
SpringerLink
Log in

Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Dual-view snapshot compressive imaging (SCI) aims to capture videos from two field-of-views (FoVs) using a 2D sensor (detector) in a single snapshot, achieving joint FoV and temporal compressive sensing, and thus enjoying the advantages of low-bandwidth, low-power and low-cost. However, it is challenging for existing model-based decoding algorithms to reconstruct each individual scene, which usually require exhaustive parameter tuning with extremely long running time for large scale data. In this paper, we propose an optical flow-aided recurrent neural network for dual video SCI systems, which provides high-quality decoding in seconds. Firstly, we develop a diversity amplification method to enlarge the differences between scenes of two FoVs, and design a deep convolutional neural network with dual branches to separate different scenes from the single measurement. Secondly, we integrate the bidirectional optical flow extracted from adjacent frames with the recurrent neural network to jointly reconstruct each video in a sequential manner. Extensive results on both simulation and real data demonstrate the superior performance of our proposed model in short inference time. The code and data are available at https://github.com/RuiyingLu/OFaNet-for-Dual-view-SCI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://github.com/NVIDIA/flownet2-pytorch/

References

  • Angayarkanni, V., Radha, S., & Akshaya, V. (2019). Multi-view video codec using compressive sensing for wireless video sensor networks. International Journal of Mobile Communications, 17(6), 727–745.

    Article  Google Scholar 

  • Caballero, J., Ledig, C., Aitken, A.P., Acosta, A., Totz, J., Wang, Z., & Shi, W. (2017). Real-time video super-resolution with spatio-temporal networks and motion compensation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 2848–2857.

  • Cheng, J., Tsai, Y., Wang, S., & Yang, M. (2017) Segflow: Joint learning for video object segmentation and optical flow. In: IEEE International conference on computer vision (ICCV), pp. 686–695.

  • Cheng, Z., Lu, R., Wang, Z., Zhang, H., Chen, B., Meng, Z., & Yuan, X. (2020). BIRNAT: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In: European conference on computer vision (ECCV).

  • Donoho, D. L. (2006). Compressed sensing. IEEE Transactions on Information Theory, 52(4), 1289–1306.

    Article  MathSciNet  Google Scholar 

  • Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015). Flownet: Learning optical flow with convolutional networks. In: 2015 IEEE International conference on computer vision (ICCV), pp. 2758–2766.

  • Emmanuel, C., Romberg, J., & Tao, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2), 489–509.

    Article  MathSciNet  Google Scholar 

  • Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., & Nayar, S.K. (2011). Video from a single coded exposure photograph using a learned over-complete dictionary. In: International conference on computer vision (ICCV), pp. 287–294. IEEE.

  • Hui, T., Tang, X., & Loy, C.C. (2018). Liteflownet: A lightweight convolutional neural network for optical flow estimation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 8981–8989. IEEE Computer Society.

  • Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017). Flownet 2.0: Evolution of optical flow estimation with deep networks. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 1647–1655.

  • Iliadis, M., Spinoulas, L., & Katsaggelos, A. K. (2018). Deep fully-connected networks for video compressive sensing. Digital Signal Processing, 72, 9–18.

    Article  Google Scholar 

  • Jalali, S., & Yuan, X. (2019). Snapshot compressed sensing: Performance bounds and algorithms. IEEE Transactions on Information Theory, 65(12), 8005–8024.

    Article  MathSciNet  Google Scholar 

  • Kingma, D., & Ba, J. (2015). Adam: A method for stochastic optimization. In: The international conference on learning representations (ICLR).

  • Liu, Y., Yuan, X., Suo, J., Brady, D., & Dai, Q. (2019). Rank minimization for snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12), 2990–3006.

    Article  Google Scholar 

  • Llull, P., Liao, X., Yuan, X., Yang, J., Kittle, D., Carin, L., et al. (2013). Coded aperture compressive temporal imaging. Optics Express, 21(9), 10526–10545.

    Article  Google Scholar 

  • Lu, S., Yuan, X., & Shi, W. (2020). An integrated framework for compressive imaging processing on CAVs. In: ACM/IEEE Symposium on edge computing (SEC).

  • Ma, J., Liu, X., Shou, Z., & Yuan, X. (2019). Deep tensor admm-net for snapshot compressive imaging. In: IEEE/CVF Conference on computer vision (ICCV).

  • Meng, Z., Jalali, S., & Yuan, X. (2020). Gap-net for snapshot compressive imaging.arXiv:2012.08364.

  • Miao, X., Yuan, X., Pu, Y., & Athitsos, V. (2019) \(\lambda \)-net: Reconstruct hyperspectral images from a snapshot measurement. In: IEEE/CVF conference on computer vision (ICCV).

  • Mait, N. J., Euliss, G. W., & Athale, R. A. (2018). Computational imaging. Advances in Optics and Photonics, 10(2), 409–483.

    Article  Google Scholar 

  • Nakamura, T., Kagawa, K., Torashima, S., & Yamaguchi, M. (2019). Super field-of-view lensless camera by coded image sensors. Sensors, 19(6), 1329.

    Article  Google Scholar 

  • Ng, J.Y., Hausknecht, M.J., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015). Beyond short snippets: Deep networks for video classification. In: IEEE Conference on computer vision and pattern recognition, (CVPR), pp. 4694–4702.

  • Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., & Sorkine-Hornung, A. (2017) Learning video object segmentation from static images. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 3491–3500.

  • Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., & Van Gool, L. (2017). The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675

  • Qiao, M., Liu, X., & Yuan, X. (2020). Snapshot spatial-temporal compressive imaging. Optics Letters, 45(7), 1659–1662.

    Article  Google Scholar 

  • Qiao, M., Meng, Z., Ma, J., & Yuan, X. (2020). Deep learning for video compressive sensing. APL Photonics, 5(3), 030801.

    Article  Google Scholar 

  • Reddy, D., Veeraraghavan, A., & Chellappa, R. (2011). P2c2: Programmable pixel compressive camera for high speed imaging. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 329–336. IEEE.

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), vol. 9351, pp. 234–241.

  • Spacek, L. (2005). A catadioptric sensor with multiple viewpoints. Robotics and Autonomous Systems, 51(1), 3–15.

    Article  Google Scholar 

  • Sun, D., Yang, X., Liu, M., & Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 8934–8943. IEEE Computer Society.

  • Sun, Y., Yuan, X., & Pang, S. (2017). Compressive high-speed stereo imaging. Optics Express, 25(15), 18182–18190.

    Article  Google Scholar 

  • Teed, Z., & Deng, J. (2020). RAFT: recurrent all-pairs field transforms for optical flow. In: A. Vedaldi, H. Bischof, T. Brox, J. Frahm (eds.) The European conference on computer vision (ECCV), vol. 12347, pp. 402–419.

  • Wagadarikar, A., John, R., Willett, R., & Brady, D. (2008). Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47(10), B44–B51.

    Article  Google Scholar 

  • Wagadarikar, A. A., Pitsianis, N. P., Sun, X., & Brady, D. J. (2009). Video rate spectral imaging using a coded aperture snapshot spectral imager. Optics Express, 17(8), 6368–6388.

    Article  Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P., et al. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Xu, K., & Ren, F. (2016). CSVideoNet: A real-time end-to-end learning framework for high-frame-rate video compressive sensing. arXiv: 1612.05203.

  • Xu, R., Li, X., Zhou, B., & Loy, C.C. (2019). Deep flow-guided video inpainting. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 3723–3732.

  • Yang, J., Yuan, X., Liao, X., Llull, P., Brady, D. J., Sapiro, G., & Carin, L. (2014). Video compressive sensing using gaussian mixture models. IEEE Transactions on Image Processing, 23(11), 4863–4878.

    Article  MathSciNet  Google Scholar 

  • Yoshida, M., Torii, A., Okutomi, M., Endo, K., Sugiyama, Y., Taniguchi, R.i., & Nagahara, H. (2018). Joint optimization for compressive video sensing and reconstruction under hardware constraints. In: The European conference on computer vision (ECCV).

  • Yuan, X. (2016). Generalized alternating projection based total variation minimization for compressive sensing. In: 2016 IEEE International conference on image processing (ICIP), pp. 2539–2543.

  • Yuan, X. (2020). Various total variation for snapshot video compressive imaging. arXiv: 2005.08028.

  • Yuan, X., Brady, D. J., & Katsaggelos, A. K. (2021). Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 38(2), 65–88.

    Article  Google Scholar 

  • Yuan, X., Liu, Y., Suo, J., & Dai, Q. (2020). Plug-and-play algorithms for large-scale snapshot compressive imaging. In: IEEE Conference on computer vision and pattern recognition (CVPR).

  • Yuan, X., Liu, Y., Suo, J., Durand, F., & Dai, Q. (2021). Plug-and-play algorithms for video snapshot compressive imaging. arXiv: 2101.04822.

  • Yuan, X., Llull, P., Liao, X., Yang, J., Brady, D.J., Sapiro, G., & Carin, L. (2014). Low-cost compressive sensing for color video and depth. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 3318–3325.

  • Yuan, X., Sun, Y., & Pang, S. (2017). Compressive video sensing with side information. Applied Optics, 56(10), 2697–2704.

    Article  Google Scholar 

  • Zhang, K., Zuo, W., & Zhang, L. (2018). FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 27(9), 4608–4622.

    Article  MathSciNet  Google Scholar 

Download references

Funding

The funding was provided by National Natural Science Foundation of China (Grand No. 61771361), the 111 Project (Grand No. B18039), Young Thousand Talent by Chinese Central Government.

Author information

Authors and Affiliations

  1. National Lab of Radar Signal Processing, Xidian University, Xi’an, Shaanxi, China

    Ruiying Lu, Bo Chen, Guanliang Liu & Ziheng Cheng

  2. School of Physical Science and Technology, Ningbo University, Ningbo, Zhejiang, China

    Mu Qiao

  3. Westlake University, Hangzhou, Zhejiang, China

    Xin Yuan

Authors
  1. Ruiying Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guanliang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ziheng Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mu Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xin Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Bo Chen or Xin Yuan.

Additional information

Communicated by Stephen Lin.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 1796 KB)

Supplementary material 2 (mp4 1813 KB)

Supplementary material 3 (mp4 1821 KB)

Supplementary material 4 (mp4 1848 KB)

Supplementary material 5 (mp4 1931 KB)

Supplementary material 6 (mp4 1900 KB)

Supplementary material 7 (mp4 15441 KB)

Supplementary material 8 (mp4 786 KB)

Supplementary material 9 (mp4 973 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, R., Chen, B., Liu, G. et al. Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network. Int J Comput Vis 129, 3279–3298 (2021). https://doi.org/10.1007/s11263-021-01532-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01532-1

Keywords

Search

Navigation