Today, industry practitioners (e.g., data scientists, developers, product managers) rely on formal privacy reviews (a combination of user interviews, privacy risk assessments, etc.) in identifying potential customer acceptance issues with their organization’s data practices. However, this process is slow and expensive, and practitioners often have to make ad-hoc privacy-related decisions with little actual feedback from users. We introduce Lean Privacy Review (LPR), a fast, cheap, and easy-to-access method to help practitioners collect direct feedback from users through the proxy of crowd workers in the early stages of design. LPR takes a proposed data practice, quickly breaks it down into smaller parts, generates a set of questionnaire surveys, solicits users’ opinions, and summarizes those opinions in a compact form for practitioners to use. By doing so, LPR can help uncover the range and magnitude of different privacy concerns actual people have at a small fraction of the cost and wait-time for a formal review. We evaluated LPR using 12 real-world data practices with 240 crowd users and 24 data practitioners. Our results show that (1) the discovery of privacy concerns saturates as the number of evaluators exceeds 14 participants, which takes around 5.5 hours to complete (i.e., latency) and costs 3.7 hours of total crowd work ( $80 in our experiments); and (2) LPR finds 89% of privacy concerns identified by data practitioners as well as 139% additional privacy concerns that practitioners are not aware of, at a 6% estimated false alarm rate.

中文翻译：

---

精益隐私审查：以低成本收集用户对数据实践的隐私担忧

今天，行业从业者（例如，数据科学家、开发人员、产品经理）依靠正式的隐私审查（用户访谈、隐私风险评估等的组合）来识别其组织的数据实践中潜在的客户接受问题。然而，这个过程缓慢且昂贵，从业者通常不得不在几乎没有用户实际反馈的情况下做出与隐私相关的临时决定。我们介绍了精益隐私审查 (LPR)，这是一种快速、廉价且易于访问的方法，可帮助从业者在设计的早期阶段通过众包工作者的代理收集用户的直接反馈。LPR 采用提议的数据实践，快速将其分解为更小的部分，生成一组问卷调查，征求用户的意见，并以紧凑的形式总结这些意见供从业者使用。通过这样做，LPR 可以帮助揭示实际人们所面临的不同隐私问题的范围和程度，而成本和等待时间只是正式审查的一小部分。我们使用 240 名人群用户和 24 名数据从业者的 12 种真实数据实践来评估 LPR。我们的结果表明（1）当评估者的数量超过 14 名参与者时，隐私问题的发现饱和，这需要大约 5.5 小时才能完成（即延迟），并花费 3.7 小时的总人群工作（ 在我们的实验中 80 美元）；(2) LPR 发现 89% 的数据从业者发现的隐私问题以及 139% 的从业者不知道的额外隐私问题，估计误报率为 6%。