In this paper, we are concerned with differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization (SCO). Most of the existing work requires the loss to be Lipschitz continuous and strongly smooth, and the model parameter to be uniformly bounded. However, these assumptions are restrictive as many popular losses violate these conditions including the hinge loss for SVM, the absolute loss in robust regression, and even the least square loss in an unbounded domain. We significantly relax these restrictive assumptions and establish privacy and generalization (utility) guarantees for private SGD algorithms using output and gradient perturbations associated with non-smooth convex losses. Specifically, the loss function is relaxed to have an α-Hölder continuous gradient (referred to as α-Hölder smoothness) which instantiates the Lipschitz continuity ($\alpha =0$) and the strong smoothness ($\alpha =1$). We prove that noisy SGD with α-Hölder smooth losses using gradient perturbation can guarantee $(ϵ,\delta )$-differential privacy (DP) and attain optimal excess population risk $\mathcal{O}(\frac{\sqrt{d\log (1/\delta )}}{nϵ}+\frac{1}{\sqrt{n}})$, up to logarithmic terms, with the gradient complexity $\mathcal{O}(n^{\frac{2-\alpha }{1+\alpha }}+n)$. This shows an important trade-off between α-Hölder smoothness of the loss and the computational complexity for private SGD with statistically optimal performance. In particular, our results indicate that α-Hölder smoothness with $\alpha \ge 1/2$ is sufficient to guarantee $(ϵ,\delta )$-DP of noisy SGD algorithms while achieving optimal excess risk with a linear gradient complexity $\mathcal{O}(n)$.

在本文中，我们关注的是随机凸优化 (SCO) 设置中的差分私有随机梯度下降 (SGD) 算法。现有的大部分工作都要求损失是 Lipschitz 连续的和强平滑的，并且模型参数是均匀有界的。然而，这些假设是有限制的，因为许多流行的损失都违反了这些条件，包括 SVM 的铰链损失、稳健回归中的绝对损失，甚至是无界域中的最小二乘损失。我们显着放宽了这些限制性假设，并使用与非平滑凸损失相关的输出和梯度扰动为私有 SGD 算法建立隐私和泛化（效用）保证。具体来说，损失函数被放宽到有一个α-Hölder 连续梯度（称为α-Hölder 平滑度），它实例化了 Lipschitz 连续性（$\alpha =0$) 和极强的平滑度 ($\alpha =1$）。我们证明了使用梯度扰动具有α- Hölder 平滑损失的嘈杂 SGD可以保证$(\epsilon ,\delta )$-差分隐私（DP）并获得最佳的人口过剩风险 $\mathcal{哦}(\frac{\sqrt{d\mathrm{日志}(1/\delta )}}{n\epsilon }+\frac{1}{\sqrt{n}})$，最多对数项，具有梯度复杂度 $\mathcal{哦}(n^{\frac{2-\alpha }{1+\alpha }}+n)$. 这显示了损失的α - Hölder 平滑度与具有统计上最佳性能的私有 SGD 的计算复杂度之间的重要权衡。特别是，我们的结果表明α- Hölder 平滑度与$\alpha \ge 1/2$ 足以保证 $(\epsilon ,\delta )$- DP有噪声的SGD算法，同时以线性梯度复杂度实现最优的超额风险 $\mathcal{哦}(n)$.