Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this article, we present TagIt, a scalable data management service framework aimed at scientific datasets, which can be integrated into prevalent distributed file system architectures. A key feature of TagIt is a scalable, distributed metadata indexing framework, which facilitates a flexible tagging capability to support data discovery. Furthermore, the tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. We have integrated TagIt into two popular distributed file systems, i.e., GlusterFS and CephFS. Our evaluation demonstrates that TagIt can expedite data search operation by up to 10× over the extant decoupled approach.

中文翻译：

---

分布式文件系统的集成索引和搜索服务

可扩展分布式环境中的搜索、发现和管理等数据服务传统上与底层文件系统解耦，通常使用外部数据库和索引服务进行部署。然而，现代数据生产率、迫在眉睫的数据移动成本和元数据的缺乏，需要重新审视分离的文件系统-数据服务设计理念。在本文中，我们介绍了 TagIt，这是一种针对科学数据集的可扩展数据管理服务框架，可以集成到流行的分布式文件系统架构中。TagIt 的一个关键特性是一个可扩展的分布式元数据索引框架，它促进了灵活的标记功能以支持数据发现。此外，标签还可以与活动运算符相关联，用于预处理、过滤、或自动元数据提取，我们以负载感知方式无缝卸载到文件服务器。我们已经将 TagIt 集成到两个流行的分布式文件系统中，即 GlusterFS 和 CephFS。我们的评估表明，与现有的解耦方法相比，TagIt 可以将数据搜索操作加快多达 10 倍。