Elsevier

Journal of Computer Languages

Volume 52, June 2019, Pages 103-112
Journal of Computer Languages

Mobile photo recommendation system of continuous shots based on aesthetic ranking

https://doi.org/10.1016/j.cola.2019.04.005Get rights and content

Abstract

Image aesthetics have been a popular topic in the recent years. The users can obtain aesthetic score of any image by using the computational approaches based on photography rules. However, most approaches consider the task as an off-line process because those methods usually require high computational complexity. On the other hand, mobile devices (tablet and smart phone) have become an indispensable tool in our daily life. Many users are more willing to take photos with them instead of digital cameras. Thus, developing practical camera applications (APPs) for mobile devices has high application potential. In this paper, respect to the issue of hardware computation limitation of mobile devices, we first design an instant photo aesthetics evaluation method. It adopts several simple and effective features to predict photo scores. The preliminary results conducted on benchmark datasets verify its performance and efficiency. Based on this achievement, we develop a continuous-shot photo recommendation system. It includes two steps: dominant photo selection performed in partially-decoded compression domain and instant photo aesthetics ranking. The system can automatically list the photos with higher scores, and help users to select desired ones. Crowd sourcing tests are employed to evaluate system performance, which show that our system is effective for instant aesthetic rating and photo recommendation.1

Introduction

As the progress of imaging hardware, a new shooting mode called the continuous shooting (or burst shooting) becomes widespread in most modern mobile cameras. In the continuous shooting mode, numerous photos are taken by a single button-press. In the past, this mode is a luxurious function that is generally equipped with high-level digital cameras or DSLRs. Nowadays, it has grown into a common and popular functionality. Since multiple photos can be produced in quick succession by pressing a button or holding it down, the photographer is allowed to take a collection of photos for a scene by panning or moving the camera in a short period. He or she can then select the most visually appealing photos from the collection for sharing and preservation. However, selecting visually favorable shots from a set of collections is often tedious and time-consuming to end users. People have to browse the entire set of collections and then select/remove unwanted ones. In this paper, we develop a photo recommendation system that can help mobile users reduce the burden of photo selection process.

This system starts from finding the dominant photos with a hierarchical clustering algorithm in the compression domain. Then, we apply an efficient aesthetic quality (AQ) assessment algorithm on the dominant photos to calculate their AQ scores. Finally, the dominant photos are displayed on the screen in a descending order according to the AQ scores to enable the photo selection process. To evaluate the feasibility and effectiveness of the proposed system, we create a mobile APP and conduct a user-study experiment via crowd sourcing. We develop a system for automatic selection and recommendation of the photos taken by a mobile camera with the continuous mode. Our system is designed by leveraging the progress of AQ estimation of photos [1], [2], [3], [4] in computational aesthetics.

We summarize the main contributions as follows:

  • 1.

    We design a set of efficient aesthetic features for photo quality evaluation. The features can be implemented efficiently on portable devices of limited computation resources. The features are verified on public benchmarks to show their effectiveness.

  • 2.

    Leveraging the aesthetics scores evaluated by the proposed features, we develop a camera APP that can recommend the representative photos automatically from the continuous photos taken by the camera.

The paper is organized as follows: Section 2 surveys AQ assessment works and gives an overview of our system. Section 3 presents the AQ feature extraction method in our work. Section 4 presents our continuous-shot photo ranking and recommendation system, and the subjective experiments. The conclusion is drawn in Section 5.

Section snippets

AQ assessment

Six factors affecting the aesthetic impression of photos include color, form (shape), motion, spatial layout, depth, and human body, from the cognitive neural science [5]. Expert photographer can take good photos by judging these factors, and favorable photography rules include the rule-of-third, clarity contrast, color harmony, lightness, shape convexity, color saturation [6], [7]. In addition to coding these rules in an algorithm, training a classifier based on the aesthetic features via

Aesthetic feature extraction

The AQ features and the AQ classifier designed in this work are depicted as follows.

Continuous-shot photo recommendation and AQ ranking system

Our proposed photo ranking system for the continuous shooting mode includes two stages. Stage I selects the dominant images from the burst photo pool, and Stage II ranks the dominant images with the trained aesthetics classifier. They are described in the following.

Conclusion

Photo AQ assessment has been a popular research topic. Many previous works achieved high classification rates by designing new aesthetic features. However, most features are not describable or are very time-consuming to compute and thus not applicable for real-time applications. This paper developed a photo recommendation system of continuous photos based on aesthetic ranking, which can assist mobile users to select the representative photos with higher aesthetic quality from the continuous

References (42)

  • J. Yang et al.

    A fast image retrieval system using index lookup table on mobile device

    Proc. Int’l Conf. on Pattern Recognition

    (2008)
  • R. Datta et al.

    Studying aesthetics in photographic images using a computational approach

    Proc. European Conf. Computer Vision

    (2006)
  • Y. Ke et al.

    The design of high-level features for photo quality assessment

    Proc. IEEE Conf. Computer Vision and Pattern Recognition

    (2006)
  • X. Tang et al.

    Content-based photo quality assessment

    IEEE Trans. Multimed.

    (2013)
  • K.-Y. Lo et al.

    Assessment of photo aesthetics with efficiency

    Proc. Int’l Conf. on Pattern Recognition

    (2012)
  • G. Peters

    Aesthetic primitives of images for visualization

    Proc. Int’l Conf. Information Visualization

    (2007)
  • H.-H. Yeh et al.

    Video aesthetic quality assessment by temporal integration of photo- and motion-based features

    IEEE Trans. Multimed.

    (2013)
  • C.-Y. Yang et al.

    Video aesthetics quality assessment by combining semantically independent and dependent features

    Proc. IEEE Conf. Acoustics, Speech, and Signal Processing

    (2011)
  • H. Tong et al.

    Classification of digital photos taken by photographers or home users

    Proc. Pacific Rim Conference on Multimedia

    (2004)
  • Y. Luo et al.

    Photo and video quality evaluation: focusing on the subject

    Proc. European Conf. Computer Vision

    (2008)
  • W. Luo et al.

    Content-based photo quality assessment

    Proc. IEEE Int’l Conf. Computer Vision

    (2011)
  • S. Dhar et al.

    High level describable attributes for predicting aesthetics and interestingness

    Proc. IEEE Conf. Comput. Vis. Pattern Recognit.

    (2011)
  • L. Marchesotti et al.

    Assessing the aesthetic quality of photographs using generic image descriptors

    Proc. IEEE Int’l Conf. Computer Vision

    (2011)
  • H. Su et al.

    Scenic photo quality assessment with bag of aesthetics-preserving feature

    Proc. ACM Int’l Conf. on Multimedia

    (2011)
  • B. Cheng et al.

    Learning to photograph

    Proc. Int’l Conf. on Multimedia

    (2010)
  • R. Datta et al.

    ACQUINE: aesthetic quality inference engine - real-time automatic rating of photo aesthetics

    Proc. Int’l Conf. Multimedia Information Retrieval

    (2010)
  • L. Yao et al.

    Oscar: on-site composition and aesthetics feedback through exemplars for photographers

    Int. J. Comput. Vis.

    (2011)
  • X. Lu et al.

    Rapid: rating pictorial aesthetics using deep learning

    Proceedings of the 22Nd ACM International Conference on Multimedia

    (2014)
  • X. Lu et al.

    Rating image aesthetics using deep learning

    IEEE Trans. Multimed.

    (2015)
  • S. Kong et al.

    Photo aesthetics ranking network with attributes and content adaptation

    Proc. European Conf. Computer Vision

    (2016)
  • K.-H. Lu et al.

    Image aesthetic assessment via deep semantic aggregation

    2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP)

    (2016)
  • Cited by (0)

    1

    This work was supported in part by Ministry of Science and Technology (MOST), Taiwan, under the grants MOST 107-2221-E-110 -067 and MOST 108-2634-F-001-004.

    View full text