【论文记录】Adaptive Clipping for Private SGD

• Differential privacy for machine learning models can be obtained in four ways: input perturbation, output perturbation, objective perturbation, and change in optimization algorithm. The fourth method modifies the optimization algorithm for training machine learning models. This includes noisy SGD methods, which we discuss in the next section.
  
  
  
  

• The analysis of such algorithms can be broken into two parts:

  1. Obtain (${\epsilon }^{\prime }$, ${\delta }^{\prime }$) differential privacy for each round of SGD, by ensuring that any information from the dataset that is used to update the model parameters is differentially-private.
  2. Compute the total privacy cost of all SGD iterations to obtain overall ($\epsilon$,$\delta$) parameters.
     
     
     

• keep track of accumulated privacy loss over multiple iterations of SGD 的几种方法 :

  1. Privacy accountant   \, [32]
  2. Moments accountant (for the Gaussian mechanism)   \, [14]
  3. Adaptive strategies to select privacy parameters ${\epsilon }^{\prime }$ and ${\delta }^{\prime }$   \, [37,38]
     
     
     

• bound the sensitivity of the gradients at each round of SGD 的几种方法 :

  1. bound the gradient norm by the Lipschitz bound   \, [16]
  2. clip each coordinate of the stochastic gradient vector to the range [−C,C]   \, [13]
  3. bounds the $l_2$-norm of the stochastic gradient by clipping the gradient $l_2$-norm to a threshold C   \, [14]
  4. adaptive strategies to select the $l_2$-norm threshold C   \, [40,41]
     
     
     

• AdaCliP
  
  
  
  

• 本文的主要创新点

  • 以前的算法：added noise to the gradients themselves.
  • 本算法：transform the gradient by a function, add noise, and apply the inverse of the function back.
    
    
    
    
    Note that choices of $a^t=(0,0,...,0)$ and $b^t=(C,C,...C)$ result in the algorithm of [14].
  • 改进后的优点：reduce the variance and bias of differentially private gradients and by Lemma 2 yield a better solution.
  • What are the optimal choices of $a^t$ and $b^t$?
  • Convergence analysis

Ref

Pichapati, V., Suresh, A. T., Yu, F. X., Reddi, S. J., & Kumar, S. (2019). AdaCliP: Adaptive clipping for private SGD. arXiv preprint arXiv:1908.07643.