5,350
edits
Deep Learning: Difference between revisions
→Strongly local min-max
| Line 1,316: | Line 1,316: | ||
;Lemma | ;Lemma | ||
If H has eigenvalue with negative real parts then eigenvalues of J lie in a unit ball iff <math>\eta < \frac{1}{|Re(\lambda)} \frac{2}{1+ (Im(\lambda)/Re(\lambda))^2}</math> for all <math>\lambda</math>. | If H has eigenvalue with negative real parts then eigenvalues of J lie in a unit ball iff <math>\eta < \frac{1}{|Re(\lambda)} \frac{2}{1+ (Im(\lambda)/Re(\lambda))^2}</math> for all <math>\lambda</math>. | ||
The convergence rate of Sim GDA is proportional to the spectral radius <math>O(\rho(J)^t)</math>. | |||
Thus we want the spectral radius to be small (e.g. 1/2) for a fast convergence rate. | |||
Suppose we have a really large negative real part in the eigenvalue. | |||
Then <math>\eta</math> will need to be really small which we don't want. | |||
Similarly if <math>Im(\lambda)/Re(\lambda)</math> is large then we will have slow convergence. | |||
{{ hidden | Proof | | |||
Suppose <math>\lambda</math> is an eigenvalue of H. | |||
<math>\lambda = -a + ib</math> with <math>a > 0</math>. | |||
<math>J = I + \lambda H</math> | |||
<math> | |||
\begin{aligned} | |||
|1 + \eta \lambda|^2 &= | 1 + \eta (-a + ib)| ^2\\ | |||
&= |(1 - \eta a) + i(\eta b)|^2\\ | |||
&= 1 + \eta^2 a^2 - 2 a \eta + \eta^2 b^2 < 1\\ | |||
\implies & \eta(a^2+b^2) - 2a < 0\\ | |||
\implies & \eta < \frac{2a}{a^2 + b^2} = \frac{1}{a} \frac{2}{1+(b/a)^2} | |||
\end{aligned} | |||
</math> | |||
==Misc== | ==Misc== | ||