Define \(Y=g(X)/f(X)\), so that \[ \int\log\left(\frac{f(x)}{g(x)}\right)f(x)dx = \int\log(1/Y)f(x)dx \]
Then \[ \mathbb{E}Y = \int\frac{g(x)}{f(x)}f(x)dx = \int g(x) = 1 \]
Now the Kullback-Leibler information is \[ \int\log\left(\frac{f(x)}{g(x)}\right)f(x)dx = \int\log\left(\frac{1}{Y}\right)f(x)dx = \int -\log Yf(x)dx = \mathbb{E}_f[-\log Y] \]
Note that \(h(y) = -\log y\) is strictly convex (second derivative is \(1/y^2\)). So by Jensen’s inequality \[ \mathbb{E}_f[-\log Y] \geq -\log\mathbb{E}_f Y = -\log 1 = 0 \]
This problem is vastly helped by noticing that the exponential distribution is a special case of the Gamma distribution: \(\text{Exponential}(\lambda) = \Gamma(1,\lambda)\) (in the shape/rate setting of the Gamma distribution).
The Gamma distribution is additive in the shape part, so that if \(X_1,\dots,X_m\sim\Gamma(1,\lambda)\) then \(\sum X_i\sim\Gamma(m,\lambda)\).
The joint densities has \(T_x=\sum X_i\) and \(T_y\sum Y_j\) as a complete sufficient statistic – we can see this by using that the exponential distribution is in the exponential family.
From the properties of the Gamma distribution, \(\mathbb{E}T_y = m/\lambda_y\). This suggests to us that something useful may be found in \(1/T_x\) as a way to find a value with an expectation that helps us with the numerator of the quotient.
Using Wikipedia (or your favorite information source) we find the density function of the Gamma distribution to allow us to integrate (here I use that \(\Gamma(n)=(n-1)!\) for integers \(n\)) \[ \mathbb ET_x^{-1} = \int\frac{\lambda_x^m}{m!}x^{m-2}e^{-\lambda_x x}dx = \frac{\lambda_x}{m-1} \]
Hence, since \(X\) and \(Y\) are independent, \[ \mathbb{E}\left[\frac{m-1}{T_x}\cdot\frac{T_y}{n}\right] = \lambda_x/\lambda_y \] producing an unbiased estimator. Since it is a function of a complete sufficient statistic, it is an UMVU.
To calculate expected square error losses, let us first calculate some expected squares of things: \[ \mathbb ET_x^{-2} = \int_0^\infty \frac{\lambda_x^m}{(m-1)!}x^{m-3}e^{-\lambda_xx}dx = \frac{\lambda_x^2}{(m-1)(m-2)} \]
We also know that \(\mathbb{E} T_y^2 = (\mathbb ET_y)^2+\mathbb VT_y = n(n+1)/\lambda_y^2\).
The expected square loss using \(dT_y/T_x\) as an estimator is \[ \mathbb E\left[ d\frac{T_y}{T_x}-\frac{\lambda_x}{\lambda_y} \right]^2 = d^2\mathbb{E}\frac{T_y^2}{T_x^2} -2d\frac{\lambda_x}{\lambda_y}\mathbb{E}\frac{T_y}{T_x} + \frac{\lambda_x^2}{\lambda_y^2} \]
Dropping the common factor of \(\lambda_x^2/\lambda_y^2\) and inserting the known expectations this amounts to minimizing \[ \min_d \left( d^2\frac{n(n+1)}{(m-1)(m-2)} -2d\frac{n}{m-1}+1 \right) \]
The minimum is achieved at \(d=(m-2)/(n+1)\).
The joint density of the \(X_i\) is \[ p(x_1,\dots,x_n|\alpha,\beta) = \exp\left[ -\frac12\left( \log(2\pi)+ \sum(x_i-\alpha t_i-\beta t_i^2)^2 \right) \right] = \\ \exp\left[ -\frac12\left( \log(2\pi)+ \sum x_i^2 - 2\alpha\sum x_i t_i- 2\beta\sum x_i t_i^2 + \left(\sum(\alpha t_i+\beta t_i^2)\right)^2 \right) \right] \]
The vector \(T=(\sum x_it_i, \sum x_it_i^2)\) is a complete sufficient statistic.
Now, \[ \mathbb ET_1 = \sum t_i\mathbb EX_i = \sum t_i(\alpha t_i+\beta t_i^2) = \alpha\sum t_i^2 + \beta \sum t_i^3 \\ \mathbb ET_2 = \sum t_i^2x_i = \sum t_i^2(\alpha t_i+\beta t_i^2) = \alpha\sum t_i^3 + \beta \sum t_i^4 \\ \]
Using these two equations and solving for \(\alpha\) and \(\beta\) we get \[ \alpha = \frac{\mathbb{E}T_1-\beta\sum t_i^3} {\sum t_i^2} \\ \beta = \frac {\mathbb ET_2-\alpha\sum t_i^3} {\sum t_i^4} = \\ \frac {\mathbb ET_2\cdot\sum t_i^2-(\mathbb{E}T_1-\beta\sum t_i^3)\sum t_i^3} {\sum t_i^4\cdot\sum t_i^2} \text{ so} \\ \beta\left( \frac{\sum t_i^4\cdot\sum t_i^2 - \sum t_i^3\cdot\sum t_i^3} {\sum t_i^4\cdot\sum t_i^2} \right) = \frac{\mathbb ET_2\cdot\sum t_i^2-\mathbb ET_1\cdot\sum t_i^3} {\sum t_i^4\cdot\sum t_i^2} \text{ so} \\ \beta = \frac {\mathbb ET_2\cdot\sum t_i^2-\mathbb ET_1\cdot\sum t_i^3} {\sum t_i^2\cdot\sum t_i^4 - \left(\sum t_i^3\right)^2} \\ \alpha = \frac {\mathbb ET_1\left( {\sum t_i^2\cdot\sum t_i^4 - \left(\sum t_i^3\right)^2} \right)-\left( {\mathbb ET_2\cdot\sum t_i^2-\mathbb ET_1\cdot\sum t_i^3} \right)\sum t_i^3} {\sum t_i^2\left( {\sum t_i^2\cdot\sum t_i^4 - \left(\sum t_i^3\right)^2}\right)} = \\ \frac {\color{green}{\sum t_i^2}\left( \mathbb ET_1\sum t_i^4-\mathbb ET_2\sum t_i^2 \right) \color{orange}{-\mathbb ET_1\left(\sum t_i^3\right)^2 +\mathbb ET_1\left(\sum t_i^3\right)^2} } {\color{green}{\sum t_i^2}\left( {\sum t_i^2\cdot\sum t_i^4 - \left(\sum t_i^3\right)^2}\right)} \]
Hence, unbiased estimators – also functions of \(T\), and therefore UMVU – are: \[ \hat\alpha = \frac {T_1\sum t_i^4-T_2\sum t_i^2} {\sum t_i^2\cdot\sum t_i^4 - \left(\sum t_i^3\right)^2} \qquad\qquad \hat\beta = \frac {T_2\sum t_i^2-T_1\sum t_i^3} {\sum t_i^2\cdot\sum t_i^4-\left(\sum t_i^3\right)^2} \]
Start with the sample variance (or its sum of squares) \[ (n-1)S^2 = \sum(X_i-\overline X)^2 = \sum\left(X_i-\frac1n\sum X_i\right)^2 = \sum_i\left( X_i^2 -\frac{2X_i}{n}\sum_j X_j +\frac{1}{n^2}\sum_j X_j\cdot\sum_k X_k \right) = \\ \sum_i\left( X_i^2 -\frac{2X_i}{n}\sum_j X_j +\frac{1}{n^2}\sum_j\sum_{k\neq j} X_jX_k +\frac{1}{n^2}\sum_j X_j^2 \right) \]
Take expectations \[ \mathbb E(n-1)S^2 = \sum_i\left( \frac{n-2}{n}\mathbb E[X_i^2] -\frac{2}{n}\sum_{j\neq i} \mathbb E[X_iX_j] +\frac{1}{n^2}\sum_j\sum_{k\neq j} \mathbb E[X_jX_k] +\frac{1}{n^2}\sum_j \mathbb E[X_j^2] \right) \]
Recall that samples are independent, so the covariance vanishes and \(\mathbb E[X_iX_j]=\mathbb EX_i\cdot\mathbb EX_j\). Furthermore, \(\mathbb E[X_i^2] = \mathbb VX+(\mathbb EX)^2\). This yields \[ \mathbb E(n-1)S^2 = \sum_i\left( \color{green}{\frac{n-2}{n}\left(\mathbb VX+(\mathbb EX)^2\right)} \color{orange}{-\frac{2}{n}(n-1)(\mathbb EX)^2 +\frac{1}{n^2}n(n-1)(\mathbb EX)^2} \color{green}{+\frac{1}{n^2}n\left(\mathbb VX+(\mathbb EX)^2\right)} \right) = \\ \sum_i\left( \color{green}{\frac{n-1}{n}\mathbb VX +\frac{n-1}{n}(\mathbb EX)^2} +\color{orange}{\frac{-2(n-1)+(n-1)}{n}(\mathbb EX)^2} \right) = (n-1)\mathbb VX \]
It follows that \(\mathbb ES^2 = \mathbb VX\) and \(S^2\) is an unbiased estimator.
If \(X_i\) is Bernoulli, they only take the values \(0\) or \(1\), so \(X_i^2=X_i\). So \[ (n-1)S^2 = \sum X_i^2-n\overline X^2 = \sum X_i-n\overline X^2 = n\overline X(1-\overline X) \]
Since, from this, we see that \(S^2\) is a function only of \(T=\sum X_i\), and \(T\) is a complete sufficient statistic, it follows that \(S^2\) is UMVU.
The exponential distribution has likelihood \[ \mathcal L(\theta|X) = \prod_i\frac{1}{\theta}\exp[-X_i/\theta] = \exp\left[ \color{blue}{-n\log\theta} \color{green}{-\frac{1}{\theta}} \color{orange}{\sum X_i} \right] = \\ \exp\left[ \color{blue}{-n\log\theta} \color{green}{-\frac{n}{\theta}} \color{orange}{\overline X} \right] \] so it is a full rank exponential family, with \(\overline X\) a complete sufficient statistic.
From \(\mathbb EX_i=\theta^{-1}\) follows that \(\mathbb E\overline X=\theta^{-1}\). From \(\mathbb VX_i=\theta^{-2}\) follows that \(\mathbb V\overline X = \sum\frac{1}{n^2}\mathbb VX_i + 2\sum_{i\neq j}\frac{1}{n^2}Cov(X_i,j) = \frac1n\mathbb VX=\frac1n\theta^{-2}\) since the observations are independent.
This means \[\mathbb E\overline X^2 = \mathbb V\overline X+(\mathbb E\overline X)^2 = \frac1n\theta^{-2}+\theta^{-2} = \frac{n+1}{n}\theta^{-2} = \frac{n+1}{n}\sigma^2 \]
Hence, \(\mathbb E[n\overline X^2/(n+1)] = \sigma^2\) and \(T=n\overline X^2/(n+1)\) is an unbiased estimator of \(\sigma^2\). Since \(\overline X\) is a complete sufficient statistic, \(T\) is UMVU.
By symmetry of the sample variables, \(\mathbb E[X_i^2|\overline X=c] = \mathbb E[X_j^2|\overline X=c]\) for any \(i,j\). The UMVU estimator is unique, so any \(\mathbb E[\delta|\overline X=c]=nc^2/(n+1)\).
We showed in a) that the sample variance is unbiased for \(\sigma^2\), so this means that \[ \frac{nc^2}{n+1} = \mathbb E[\delta|\overline X=c] = \mathbb E\left[\frac{\sum X_i^2-n\overline X^2}{n-1}\middle|\overline X=c\right] = \frac {n\mathbb E[X^2_i|\overline X=c]-nc^2} {n-1} \]
so, solving for \(\mathbb E[X^2_i|\overline X=c]\) we get \[ \mathbb E[X^2_i|\overline X=c] = \frac{2nc^2}{n+1} \]
The joint density function is \[ \frac{1}{\sqrt{(2\pi\theta^2)^nn!}} \exp\left[ -\frac{1}{2\theta^2}\sum_j\frac{x_j^2}{j} \right] \]
This forms a full rank exponential family density with \(T(x)=\sum_j x_j^2/j\). Hence \(T\) is complete and minimally sufficient.
The variables \(Y_1=X_1, Y_2=X_2/\sqrt{2}, \dots, Y_n=X_n/\sqrt{n}\) are all \(\sim\mathcal{N}(0,\theta^2)\). The sample variance \(S^2=\frac{\sum Y_j^2}{n-1}\) is an unbiased estimator for \(\theta^2\).
This sample variance is \[ S^2 = \frac{1}{n-1}\sum Y_j^2 = \frac{1}{n-1}\sum (X_j/\sqrt{j})^2 = \frac{1}{n-1}\sum X_j^2/j = \frac{T}{n-1} \]
Since \(S^2\) is a function of a complete sufficient statistic and has expected value \(\theta^2\), it is an UMVU.