Stochastic Gradient Descent

本文介绍了随机梯度下降(SGD)算法的工作原理及其特点。SGD通过每次迭代使用一个样本来更新权重,使得计算成本相对较低。文章还讨论了选择合适步长的重要性,并提供了使算法表现更佳的一些技巧。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Stochastic Gradient Descent

1. What is Stochastic Gradient Descent

Stochastic Gradient Descent(SGD) is similiar with Batch Gradient Desent, but it used only 1 example for each iteration. So that it makes some different as well. However, the stochastic gradient descent will not exactly converge into the minimum point. It will bounds around some ratio of the minimum point. Also the cost function will not decrease all the time. It oscillates and tends to converge/expand account for the step size which you have choose.

2. Stochastic Gradient Descent Algorithm

Since we use only one example for each iteration, so the weights would be optimize with a random gradient, as a result the direction is unsure. But after loop all examples, the trend of the algorithm will lead to converge. So the this algorithm can never converge exactly to the minimum point. Choose an appropriate step size is of significant importance.

Another tips to make this algorithm performs better is loop the whole procedure for some times, say 1 to 10 times. This should depend on the dataset, since loop a large data set for 10 times is also compute intensive.

The algorithm procedure is shown below.

Sthochastic Gradient Descent Algorithm

Sthochastic Gradient Descent Algorithm

3. Compute Effort

Since the stochastic use only 1 example each iteration, the compute effort of this algorithm is O(N).

Batch Gradient DescentStochastic Gradient Descent
use 1 example in each iterationuse all example in each iteration
relative compute looserelative compute intensive

4. Visualize Algorithm

The images below shown the stochastic gradient descent in 1 features and 2 features. It shows that the cost is not alway converge and it eventually converge.

Visualize Algorithm

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值