220322

最新推荐文章于 2026-01-01 10:51:21 发布

原创最新推荐文章于 2026-01-01 10:51:21 发布 · 251 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#学习

日常工作总结同时被 2 个专栏收录

13 篇文章

订阅专栏

English

2 篇文章

订阅专栏

本文提出了一种有效扩展差分隐私到大型神经网络的方法，通过在公共数据上预训练模型来减少私人数据微调期间的可训练参数数量。在保持大型模型表达力的同时，仅对一小部分参数进行私人数据的微调，实现了领域适应过程中的最小训练干扰。研究还指出，选择精确的参数子集以优化差分隐私与准确性之间的权衡是未来研究的重要方向。

Batch Normalization (BN) and its variants have have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods.
We propose a simple yet effective method to scale up differential privacy to large neural networks at reasonable privacy budgets. Our key insight was to minimize the number of trainable parameters during private dataset finetuning, by leveraging additional public data. By pre-training large models on public data, we obtain a strong representation at no privacy cost. Next, finetuning a small subset of
parameters on private data, we maintain the expressiveness of large models while introducing minimal training disruption during the process of domain adaptation.
We note that our two proposed approaches – normalization transfer and convolution parameter transfer – albeit outperforming previous methods, are naive stabs-in-the-dark; an exploration into the precise parameter subset to choose in order to optimize over the fundamental privacy-accuracy tradeoff in differential privacy is likely to be a fruitful area of research.