Gradient descent is an algorithm for finding values of parameters w and b that minimize the cost function J.

\; w &= w - \alpha \frac{\partial J(w,b)}{\partial w} \; \newline b &= b - \alpha \frac{\partial J(w,b)}{\partial b} \newline \rbrace \end{align*}$$ where, parameters $w$, $b$ are updated simultaneously. $\alpha$ is the learning rate. - Gradient:

\frac{\partial J(w,b)}{\partial w} &= \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})x^{(i)} \tag{4}\
\frac{\partial J(w,b)}{\partial b} &= \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)}) \tag{5}\

- Learning Rate: How small changes are. - ![image.png]( - Convex Function: Bowl shape ## Learning Rate - Draw a graph of the cost function to see - if it converge (**done**) - if it has some up and down (**bad learning rate**) ## Stochastic Gradient Descent Stochastic Gradient Descent (SGD) is an optimization algorithm commonly used in machine learning and deep learning for minimizing the loss function. It is a variation of the standard Gradient Descent algorithm that uses only a single or a small random subset of the data (mini-batch) to compute the gradient at each step.