Detection and Estimation (Estimation part)
GIST 황의석 교수님 [EC7204-01] Detection and Estimation 강의 정리 내용입니다.
An estimator \(\hat{\theta}\) is called unbiased, if \(E(\hat{\theta}) = \theta\) for all possible \(\theta\).
\(\hat{\theta} = g(x) \Rightarrow E(\hat{\theta}) = \int g(x)p(x;\theta)dx = \theta\)
If \(E(\hat{\theta} \neq \theta)\), the bias is \(b(\theta) = E(\hat{\theta}) - \theta\)
(Expectation is taken with respect to \(x\) or \(p(x;\theta)\))
Unbiased estimator is not necessarily a good estimator;
but a biased estimator is a poor estimator.
MSE Criterion
\(\text{mse}(\hat{\theta}) = E\left[(\hat{\theta} - \theta)^2\right]\)
\(\quad\quad\quad = E\left[\left((\hat{\theta} - E(\hat{\theta})) + (E(\hat{\theta}) - \theta)\right)^2\right]\)
\(\quad\quad\quad = \text{var}(\hat{\theta}) + \left[E(\hat{\theta}) - \theta\right]^2\)
\(\quad\quad\quad = \text{var}(\hat{\theta}) + b^2(\theta)\)
Minimum Variance Unbiased(MVU) Estimator
Any criterion that depends on the bias is likely to be unrealizable → Practically minimum MSE estimator needs to be abandoned Minimum variance unbiased(MVU) estimator
Alternatively, constrain the bias to be zero
Find the estimator which minimizes the variance (minimizing the MSE as well for unbiased case)
\(\text{mse}(\hat{\theta}) = \text{var}(\hat{\theta}) + b^2(\theta)\)
\(\quad\quad\quad= \text{var}(\hat{\theta})\) (MVUE에서는 bias가 없음)
→ Minimum variance unbiased (MVU) estimator
Lec 3. Cramer-Rao Lower Bound (CRLB)
Cramer-Rao Lower Bound (CRLB)
The CRLB give a lower bound on the variance of any unbiased estimator. (biased의 경우는 알 수 없다!)
Does not guarantee bound can be obtained.
만약 unbiased estimator의 variance가 CRLB라면 그 estimator는 MVUE.
likelihood function이 다음과 같다고 하면,
\[p(x[0];A) = \frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}[-\frac{1}{2\sigma^2}(x[0]-A)^2]\]
log-likelihood function은 다음과 같고
\[\text{ln}\ p(x[0];A) = -\text{ln}\sqrt{2\pi\sigma^2}-\frac{1}{2\sigma^2}(x[0]-A)^2\]
A에 대해 두번 미분해주고 마이너스를 붙여주면
\[-\frac{\partial ^2 \text{ln}\ p(x[0];A)}{\partial A^2} = \frac{1}{\sigma^2}, \quad \quad \text{var}(\hat{A}) = \sigma^2 = \frac{1}{-\frac{\partial ^2 \text{ln}\ p(x[0];A)}{\partial A^2}}\]
More appropriate measure is average curvature(곡률, (2차미분 값)), \(-E[\frac{\partial ^2 \text{ln}\ p(x[0];A)}{\partial A^2}]\)
(In general, the \(2^{\text{nd}}\) derivative will depend on \(x[0] \rightarrow\) the likelihood function is a R.V.)
Theorem: CRLB - Scalar Parameter
Let \(p(\mathbf{x}; \theta)\) satisfy the “regularity” condition
\[E_\mathbf{x}[\frac{\partial\text{ln}\ p(\mathbf{x}; \theta)}{\partial \theta}] = 0\quad \text{for all}\ \theta\]
Then, the variance of any unbiased estimator \(\hat{\theta}\) must satisfy
\[\text{var}(\hat{\theta}) \ge \frac{1}{-E_\mathbf{x}[\frac{\partial^2\text{ln}\ p(\mathbf{x}; \theta)}{\partial \theta^2}]} = \frac{1}{E_\mathbf{x}[(\frac{\partial\text{ln}\ p(\mathbf{x}; \theta)}{\partial \theta})^2]} = \frac{1}{I(\theta)}\]
where the derivative is evaluated at the true value \(\theta\) and the expectation is taken w.r.t. \(p(\mathbf{x};\theta).\)
여기서 세번째 텀에서 - 가 사라지는 이유는, 두번째 텀에서 - 가 붙는 이유를 생각해보면 된다.
두번째 텀에서 마이너스가 붙는 이유는 값을 양수로 만들어주기 위함이고, 세번째 텀은 일차 미분의 제곱이므로 자연스럽게 양수이다. 따라서 - 가 사라지게 된다.
Furthermore, an unbiased estimator may be found that attains the bound for all \(\theta\) if and only if
\[\frac{\partial\ \text{ln}\ p(\mathbf{x}; \theta)}{\partial \theta} = I(\theta)(g(\mathbf{x})-\theta)\]
for some functions \(g()\) and \(I\). That estimator, which is the MVUE, is \(\hat{\theta} = g(\mathbf{x})\), and the minimum variance is \(\frac{1}{I(\theta)}\)
이렇게 표현이 되면 g() 는 MVUE 이다.
Fisher Information: 추정 정확도를 측정하는 방법. 우도 함수의 곡률이 크면 추정량의 분산이 작고, 더 정확한 추정을 할 수 있음을 의미
정칙성 조건(Regularity Conditions): CRLB가 성립하기 위한 조건으로, 특정 미분 가능성 조건을 만족해야 한다.
CRLB Proof (Appendix 3A)
\[- E [\frac{\partial^2 \text{ln} p(\mathbf{x}; \theta)}{\partial \theta^2}] = \frac{1}{c(\theta)} = I(\theta)\]