Large scale cloud services use Key Performance Indicators (KPIs) for tracking and monitoring performance. They usually have Service Level Objectives (SLOs) baked into the customer agreements which are tied to these KPIs. Dependency failures, code bugs, infrastructure failures, and other problems can cause performance regressions. It is critical to minimize the time and manual effort in diagnosing and triaging such issues to reduce customer impact. Large volume of logs and mixed type of attributes (categorical, continuous) in the logs makes diagnosis of regressions non-trivial. In this paper, we present the design, implementation and experience from building and deploying DeCaf, a system for automated diagnosis and triaging of KPI issues using service logs. It uses machine learning along with pattern mining to help service owners automatically root cause and triage performance issues. We present the learnings and results from case studies on two large scale cloud services in Microsoft where DeCaf successfully diagnosed 10 known and 31 unknown issues. DeCaf also automatically triages the identified issues by leveraging historical data. Our key insights are that for any such diagnosis tool to be effective in practice, it should a) scale to large volumes of service logs and attributes, b) support different types of KPIs and ranking functions, c) be integrated into the DevOps processes.

中文翻译：

---

DeCaf：诊断和分类大规模云服务中的性能问题

大型云服务使用关键绩效指标 (KPI) 来跟踪和监控绩效。他们通常将服务水平目标 (SLO) 纳入与这些 KPI 相关的客户协议中。依赖故障、代码错误、基础设施故障和其他问题可能导致性能回归。最大限度地减少诊断和分类此类问题的时间和手动工作以减少对客户的影响至关重要。日志中的大量日志和混合类型的属性（分类的、连续的）使得回归诊断变得非常重要。在本文中，我们介绍了构建和部署 DeCaf 的设计、实施和经验，DeCaf 是一个使用服务日志自动诊断和分类 KPI 问题的系统。它使用机器学习和模式挖掘来帮助服务所有者自动找出根本原因并分类性能问题。我们介绍了 Microsoft 的两个大型云服务案例研究的学习和结果，DeCaf 成功诊断了 10 个已知问题和 31 个未知问题。DeCaf 还通过利用历史数据自动对已识别的问题进行分类。我们的主要见解是，要使任何此类诊断工具在实践中有效，它应该 a) 扩展到大量服务日志和属性，b) 支持不同类型的 KPI 和排名功能，c) 集成到 DevOps 流程中。DeCaf 还通过利用历史数据自动对已识别的问题进行分类。我们的主要见解是，要使任何此类诊断工具在实践中有效，它应该 a) 扩展到大量服务日志和属性，b) 支持不同类型的 KPI 和排名功能，c) 集成到 DevOps 流程中。DeCaf 还通过利用历史数据自动对已识别的问题进行分类。我们的主要见解是，要使任何此类诊断工具在实践中有效，它应该 a) 扩展到大量服务日志和属性，b) 支持不同类型的 KPI 和排名功能，c) 集成到 DevOps 流程中。