Mobile photo recommendation system of continuous shots based on aesthetic ranking
Introduction
As the progress of imaging hardware, a new shooting mode called the continuous shooting (or burst shooting) becomes widespread in most modern mobile cameras. In the continuous shooting mode, numerous photos are taken by a single button-press. In the past, this mode is a luxurious function that is generally equipped with high-level digital cameras or DSLRs. Nowadays, it has grown into a common and popular functionality. Since multiple photos can be produced in quick succession by pressing a button or holding it down, the photographer is allowed to take a collection of photos for a scene by panning or moving the camera in a short period. He or she can then select the most visually appealing photos from the collection for sharing and preservation. However, selecting visually favorable shots from a set of collections is often tedious and time-consuming to end users. People have to browse the entire set of collections and then select/remove unwanted ones. In this paper, we develop a photo recommendation system that can help mobile users reduce the burden of photo selection process.
This system starts from finding the dominant photos with a hierarchical clustering algorithm in the compression domain. Then, we apply an efficient aesthetic quality (AQ) assessment algorithm on the dominant photos to calculate their AQ scores. Finally, the dominant photos are displayed on the screen in a descending order according to the AQ scores to enable the photo selection process. To evaluate the feasibility and effectiveness of the proposed system, we create a mobile APP and conduct a user-study experiment via crowd sourcing. We develop a system for automatic selection and recommendation of the photos taken by a mobile camera with the continuous mode. Our system is designed by leveraging the progress of AQ estimation of photos [1], [2], [3], [4] in computational aesthetics.
We summarize the main contributions as follows:
- 1.
We design a set of efficient aesthetic features for photo quality evaluation. The features can be implemented efficiently on portable devices of limited computation resources. The features are verified on public benchmarks to show their effectiveness.
- 2.
Leveraging the aesthetics scores evaluated by the proposed features, we develop a camera APP that can recommend the representative photos automatically from the continuous photos taken by the camera.
The paper is organized as follows: Section 2 surveys AQ assessment works and gives an overview of our system. Section 3 presents the AQ feature extraction method in our work. Section 4 presents our continuous-shot photo ranking and recommendation system, and the subjective experiments. The conclusion is drawn in Section 5.
Section snippets
AQ assessment
Six factors affecting the aesthetic impression of photos include color, form (shape), motion, spatial layout, depth, and human body, from the cognitive neural science [5]. Expert photographer can take good photos by judging these factors, and favorable photography rules include the rule-of-third, clarity contrast, color harmony, lightness, shape convexity, color saturation [6], [7]. In addition to coding these rules in an algorithm, training a classifier based on the aesthetic features via
Aesthetic feature extraction
The AQ features and the AQ classifier designed in this work are depicted as follows.
Continuous-shot photo recommendation and AQ ranking system
Our proposed photo ranking system for the continuous shooting mode includes two stages. Stage I selects the dominant images from the burst photo pool, and Stage II ranks the dominant images with the trained aesthetics classifier. They are described in the following.
Conclusion
Photo AQ assessment has been a popular research topic. Many previous works achieved high classification rates by designing new aesthetic features. However, most features are not describable or are very time-consuming to compute and thus not applicable for real-time applications. This paper developed a photo recommendation system of continuous photos based on aesthetic ranking, which can assist mobile users to select the representative photos with higher aesthetic quality from the continuous
References (42)
- et al.
A fast image retrieval system using index lookup table on mobile device
Proc. Int’l Conf. on Pattern Recognition
(2008) - et al.
Studying aesthetics in photographic images using a computational approach
Proc. European Conf. Computer Vision
(2006) - et al.
The design of high-level features for photo quality assessment
Proc. IEEE Conf. Computer Vision and Pattern Recognition
(2006) - et al.
Content-based photo quality assessment
IEEE Trans. Multimed.
(2013) - et al.
Assessment of photo aesthetics with efficiency
Proc. Int’l Conf. on Pattern Recognition
(2012) Aesthetic primitives of images for visualization
Proc. Int’l Conf. Information Visualization
(2007)- et al.
Video aesthetic quality assessment by temporal integration of photo- and motion-based features
IEEE Trans. Multimed.
(2013) - et al.
Video aesthetics quality assessment by combining semantically independent and dependent features
Proc. IEEE Conf. Acoustics, Speech, and Signal Processing
(2011) - et al.
Classification of digital photos taken by photographers or home users
Proc. Pacific Rim Conference on Multimedia
(2004) - et al.
Photo and video quality evaluation: focusing on the subject
Proc. European Conf. Computer Vision
(2008)
Content-based photo quality assessment
Proc. IEEE Int’l Conf. Computer Vision
High level describable attributes for predicting aesthetics and interestingness
Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
Assessing the aesthetic quality of photographs using generic image descriptors
Proc. IEEE Int’l Conf. Computer Vision
Scenic photo quality assessment with bag of aesthetics-preserving feature
Proc. ACM Int’l Conf. on Multimedia
Learning to photograph
Proc. Int’l Conf. on Multimedia
ACQUINE: aesthetic quality inference engine - real-time automatic rating of photo aesthetics
Proc. Int’l Conf. Multimedia Information Retrieval
Oscar: on-site composition and aesthetics feedback through exemplars for photographers
Int. J. Comput. Vis.
Rapid: rating pictorial aesthetics using deep learning
Proceedings of the 22Nd ACM International Conference on Multimedia
Rating image aesthetics using deep learning
IEEE Trans. Multimed.
Photo aesthetics ranking network with attributes and content adaptation
Proc. European Conf. Computer Vision
Image aesthetic assessment via deep semantic aggregation
2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP)
Cited by (0)
- 1
This work was supported in part by Ministry of Science and Technology (MOST), Taiwan, under the grants MOST 107-2221-E-110 -067 and MOST 108-2634-F-001-004.