Elsevier

Graphical Models

Volume 116, July 2021, 101104
Graphical Models

Geometry-Based Layout Generation with Hyper-Relations AMONG Objects

https://doi.org/10.1016/j.gmod.2021.101104Get rights and content

Abstract

Recent studies show increasing demands and interests in automatic layout generation, while there is still much room for improving the plausibility and robustness. In this paper, we present a data-driven layout generation framework without model formulation and loss term optimization. We achieve and organize priors directly based on samples from datasets instead of sampling probabilistic distributions. Therefore, our method enables expressing relations among three or more objects that are hard to be mathematically modeled. Subsequently, a non-learning geometric algorithm is proposed to arrange objects considering constraints such as positions of walls and windows. Experiments show that the proposed method outperforms the state-of-the-art and our generated layouts are competitive to those designed by professionals.1

Introduction

3D scenes are becoming fundamental in many areas of computer graphics, e.g., photo-realistic rendering, virtual reality (VR), providing datasets for computer vision [1], etc. However, the progressive development of computer graphics requires better modeling of 3D scenes and generation of layouts. Therefore, we have been investigating techniques of automatically generating scene layouts.

Generating scene layouts benefits various applications. First, it saves the effort of manually placing objects in video games or industrial designs2. Li et al. [2] generate various layouts for better simulations of wheelchair training. Handa et al. [1] generate multi-view images from much fewer 3D scenes.

Existing works already show the progress of scene synthesis [3], where scene layouts focus on their plausibility and aesthetic, i.e., visual identifications given generated layouts. Existing works are divided into neural network based techniques and others. The former trains several neural networks for different steps such as placing objects, rotating objects, deciding termination of arrangements [4]. The latter formulates a set of mathematical models including graphs, and typically optimize a shuffled area based on e.g., Markov Chain Monte Carlo (MCMC) [5], [6], since the models are too complicated to be solved. Nevertheless, algorithmic methods have not been investigated as far as we reviewed, because similarly we have to embed layout rules into an algorithm so that it operates properly. However, layout rules are innumerable. A qualitative comparison of existing techniques is beyond the scope of this paper. Despite underlying technical details, this paper focuses on the final results, i.e., improving the plausibility and aesthetic of generated layouts.

In this paper, we propose an algorithmic framework for generating room layouts as shown in Fig. 1. Our framework is split into a data-driven phase: coherent grouping and a non-data-driven phase: geometric arranging. In the coherent grouping phase, objects are clustered into several coherent groups (Section 3), where priors are learnt for suggesting layouts within each coherent group. We directly use correct and denoised samples extracted from datasets as priors. This brings two benefits. First, we no longer hypothesize distributions of layout rules between/among objects, especially the mathematically inexpressible relations. Second, we could easily formulate and represent relations among three or more objects since we only have to load samples of real distributions. Similar to “hyper-graphs” where an edge connects to more than two vertices, we name our learnt relations among objects “hyper-relations” (Section 4). Thus, several objects of the same coherent group are arranged in O(1) time by sampling from their hyper-relations. In the geometric arranging phase, given independent coherent groups where objects of the same group are already properly arranged with respect to each other, a geometric algorithm is proposed to assign positions and orientations of each group. Since layout rules among objects are applied during the coherent grouping phase, geometry phase concentrates on much fewer rules related to walls, windows, etc (Section 5).

Current technologies of synthesizing 3D scenes include selecting a set of appropriate objects and generating plausible layouts for the objects. We do “layout” while techniques for selecting objects are easily incorporated such as [7]. Note that in this paper, we prefer instance-based priors to category-based priors, e.g., we consider a spatial relation between a specific coffee table to a specific chair, where both of them have unique textures and geometries. If categories are being based, distinct features of objects are lost. As shown in Fig. 2, different shapes of several armchairs have their own priors to the same coffee table.

In this paper, we make the following contributions:

  • 1.

    We first introduce and learn hyper-relations among three or more objects, which increases the aesthetics and plausibility of arranging objects in same coherent groups and only requires O(1) time to sample layouts of each group, e.g., a coffee table surrounded with several distinct sofas and a TV stand, thus increasing the overall performance.

  • 2.

    We propose a new scalable geometry-based framework for layout generation, which considers detailed aspects of room layout, e.g., doors, windows, wall decorations, small objects, etc. In coordination with hyper-relations, more plausible and robust layouts are generated.

  • 3.

    We develop an open-source platform for manipulating 3D scenes, where operations such as rendering, exploring and modifying scenes are supported, thus allowing researches to focus thoroughly on algorithms.

Section snippets

Related works

3D scene synthesis is to select a set of appropriate objects and transform them plausibly [3]. Earlier works of synthesizing 3D scenes are mainly based on hand-crafted design rules, e.g., [8], [9] or data-driven priors. For the former, designing rules are mathematically formulated as a set of constraints followed by optimizations [10], [11]. For the latter, since learnt distributions are too complicated to be differentiated, MCMC is assembled to solve such situation by attempting proposals [5],

Definitions

Given a list of objects with the positions of doors and windows and a room shape, we formulate its corresponding graph G=<V,E> where each object oV. E is the set of edges which are also simply relations between/among objects. Note that in this paper, we assume that a relation may involve more than two instances i.e., a hyper-relation among objects (Section 4).

A coherent group g is a list containing objects where one object connects to at least one another object in the same group. In other

Priors

In this section we show how we extract relations among objects. We start by extracting traditional pairwise relations, e.g,. a desk with respect to a chair. Then, we present pre-computed pattern chains which generalize one-to-one relations to one-to-many relations, e.g., a dining table surrounded by several identical chairs. Finally, based on pairwise relations and pattern chains, we further generalize and formulate hyper-relations among objects, i.e, relations “among” more than two instances.

Coherent grouping

We show how we arrange objects in this section. First, objects are decomposed into several coherent groups giG based on finding maximal connected subgraphs using pairwise relations between objects as shown in Fig. 7, where whether or not two objects are connected depends on existence of pairwise relations between objects.

One secondary object can have at most one dominant object. If multiple available dominant objects exist with respect to a secondary object osec, we randomly select a dominant

Setup

We utilize a recent 3D scene dataset “3D-Front3” [30] with 70,000+ layouts and 9992 3D models. To roam and render 3D scenes, we develop an open-source 3D scene platform as shown in Fig. 8, where we can add, delete, modify and search for objects. We can orbitally control the perspective camera for selecting better views. By clicking “layout”, a configuration of the current room is generated by our proposed framework. We render 3D scenes

Conclusions and future works

In this paper, we present a new framework of generating room layouts and we experimentally verify the plausibility and robustness of the proposed method. The code of this framework and a toolbox platform is publicly available. We hope this could benefit the community. However, this work still suffers from the following weaknesses.

The biggest difficulty we encountered is arranging “chains” of objects around walls. For independent objects such as wardrobes, transformations of them have high

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the National Key Technology R&D Program (Project Number 2017YFB1002604), the National Natural Science Foundation of China (Project Numbers 61772298, 61832016), Research Grant of Beijing Higher Institution Engineering Research Center, and Tsinghua Tencent Joint Laboratory for Internet Innovation Technology. We also thank all reviewers for their constructive comments and Yuan-Chen Guo for a thorough proof-reading.

References (31)

  • Q. Fu et al.

    Human-centric metrics for indoor scene assessment and synthesis

    Graph. Models

    (2020)
  • A. Handa et al.

    Understanding real world indoor scenes with synthetic data

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2016)
  • W. Li et al.

    Automatic synthesis of virtual wheelchair training scenarios

    2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)

    (2020)
  • S.-H. Zhang et al.

    A survey of 3d indoor scene synthesis

    J. Comput. Sci. Technol.

    (2019)
  • K. Wang et al.

    Deep convolutional priors for indoor scene synthesis

    ACM Transactions on Graphics (TOG)

    (2018)
  • L.-F. Yu et al.

    Make it home: automatic optimization of furniture arrangement

    ACM Trans. Graph.

    (2011)
  • S. Qi et al.

    Human-centric indoor scene synthesis using stochastic grammar

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2018)
  • Y. He et al.

    Style-compatible object recommendation for multi-room indoor scene synthesis

    arXiv preprint arXiv:2003.04187

    (2020)
  • T. Germer et al.

    Procedural arrangement of furniture for real-time walkthroughs

    Computer Graphics Forum

    (2009)
  • G.H. Lyons

    Ten common home decorating mistakes & how to avoid them

    (2008)
  • P. Merrell et al.

    Interactive furniture layout using interior design guidelines

    ACM transactions on graphics (TOG)

    (2011)
  • T. Weiss et al.

    Fast and scalable position-based layout synthesis

    arXiv preprint arXiv:1809.10526

    (2018)
  • Y. Liang et al.

    Automatic data-driven room design generation

    International Workshop on Next Generation Computer Animation Techniques

    (2017)
  • Y. Liang et al.

    Knowledge graph construction with structure and parameter learning for indoor scene design

    Computational Visual Media

    (2018)
  • Y.-T. Yeh et al.

    Synthesizing open worlds with constraints using locally annealed reversible jump mcmc

    ACM Transactions on Graphics (TOG)

    (2012)
  • Cited by (11)

    View all citing articles on Scopus
    1

    Code is publicly available at https://github.com/Shao-Kui/3DScenePlatform, including the proposed framework (algorithm) and a 3D scene platform (toolbox).

    View full text