Abstract Modular construction is an attractive building method due to its advantages over traditional stick-built methods in terms of reduced waste and construction time, more control over resources and environment, and easier implementation of novel techniques and technologies in a controlled factory setting. However, efficient and timely decision-making in modular factories requires spatiotemporal information about the resources regarding their locations and activities which motivates the necessity for an automated activity identification framework. Thus, this paper utilizes sound, a ubiquitous data source present in every modular construction factory, for the automatic identification of commonly performed manual activities such as hammering, nailing, sawing, etc. To develop a robust activity identification model, it is imperative to engineer the appropriate features of the data source (i.e., traits of the signal) that provides a compact yet descriptive representation of the parameterized audio signal based on the nature of the sound, which is very dependent on the application domain. In-depth analysis regarding appropriate features selection and engineering for audio-based activity identification in construction is missing from current research. Thus, this research extensively investigates the effects of various features extracted from four different domains related to audio signals (time-, time-frequency-, cepstral-, and wavelet-domains), in the overall performance of the activity identification model. The effect of these features on activity identification performance was tested by collecting and analyzing audio data generated from manual activities at a modular construction factory. The collected audio signals were first balanced using time-series data augmentation techniques and then used to extract a 318-dimensional feature vector containing 18 different feature sets from the abovementioned four domains. Several sensitivity analyses were performed to optimize the feature space using a feature ranking technique (i.e., Relief algorithm), and the contribution of features in the top feature sets using a support vector machine (SVM). Eventually, a final feature space was designed containing a 130-dimensional feature vector and 0.5-second window size yielding about 97% F-1 score for identifying different activities. The contributions of this study are two-fold: 1. A novel means of automated manual construction activity identification using audio signal is presented; and 2. Foundational knowledge on the selection and optimization of the feature space from four domains is provided for future work in this research field. The result of this study demonstrates the potential of the proposed system to be applied for automated monitoring and data collection in modular construction factory in conjunction with other activity recognition frameworks based on computer vision (CV) and/or inertial measurement units (IMU).

中文翻译：

---

使用音频信号和机器学习的模块化构造中的活动识别

摘要 模块化建筑是一种有吸引力的建筑方法，因为它比传统的棒状建筑方法在减少浪费和施工时间、更好地控制资源和环境以及更容易在受控工厂环境中实施新技术和技术方面具有优势。然而，模块化工厂中高效及时的决策需要有关资源的位置和活动的时空信息，这激发了自动化活动识别框架的必要性。因此，本文利用声音，这是每个模块化建筑工厂中无处不在的数据源，用于自动识别常见的手动活动，如锤击、钉钉、锯切等。为了开发一个强大的活动识别模型，必须设计数据源的适当特征（即信号的特征），以根据声音的性质提供参数化音频信号的紧凑但描述性的表示，这非常依赖于应用领域。当前的研究缺少关于建筑中基于音频的活动识别的适当特征选择和工程的深入分析。因此，本研究广泛研究了从与音频信号相关的四个不同域（时间域、时频域、倒谱域和小波域）中提取的各种特征对活动识别模型整体性能的影响。通过收集和分析模块化建筑工厂手动活动产生的音频数据，测试了这些特征对活动识别性能的影响。收集的音频信号首先使用时间序列数据增强技术进行平衡，然后用于从上述四个域中提取包含 18 个不同特征集的 318 维特征向量。使用特征排序技术（即 Relief 算法）优化特征空间，并使用支持向量机（SVM）优化特征在顶级特征集中的贡献。最终，设计了一个包含 130 维特征向量和 0.5 秒窗口大小的最终特征空间，产生了大约 97% 的 F-1 分数，用于识别不同的活动。本研究的贡献有两个方面：1. 提出了一种使用音频信号自动识别人工施工活动的新方法；2. 为该研究领域的未来工作提供了从四个领域选择和优化特征空间的基础知识。这项研究的结果表明，所提出的系统与基于计算机视觉 (CV) 和/或惯性测量单元 (IMU) 的其他活动识别框架相结合，具有应用于模块化建筑工厂自动化监控和数据收集的潜力。为该研究领域的未来工作提供了从四个域中选择和优化特征空间的基础知识。这项研究的结果证明了所提出的系统与基于计算机视觉 (CV) 和/或惯性测量单元 (IMU) 的其他活动识别框架相结合，应用于模块化建筑工厂的自动化监控和数据收集的潜力。为该研究领域的未来工作提供了从四个域中选择和优化特征空间的基础知识。这项研究的结果表明，所提出的系统与基于计算机视觉 (CV) 和/或惯性测量单元 (IMU) 的其他活动识别框架相结合，具有应用于模块化建筑工厂自动化监控和数据收集的潜力。