Maximum Likelihood Estimation

General

• Obtain an estimate for an unknown parameter theta using the data that we obtained from our sample.
• Choose a value of theta that maximizes the likelihood of getting the data we observed.
• Joint probability mass function: If the observations are independent you can just multiply the PDFs of the individual observations.
  • $L(\theta )=\prod _{i=1}^{n}f(x_{i};\theta )$ (General formulation)

Bernoulli Distribution

• $f(x_{i};p)=p^{x_{i}}(1-p)^{1-x_{i}}$ for xi = 0 or 1 and 0 < p < 1.
• If the Xi are independent Bernoulli random variables with unknown parameter p, replace the general notation with the bernoulli notation:
• $L(p)=p^{\sum x_{i}}(1-p)^{n-\sum x_{i}}$
• $logL(p)=(\sum x_{i})log(p)+(n-\sum x_{i})log(1-p)$

Exponential Distribution

• Suppose we have samples from an exponential distribution with parameter lambda:
  • $X_{i}\sim {\textrm {Exp}}(\lambda )$, assuming i.i.d.
• Recall that the density is the product of $f(x_{\textrm {undertilde}}|\lambda )=\prod _{i=1}^{n}\lambda e^{-\lambda x_{i}}=\lambda ^{n}e^{-\lambda \sum x_{i}}$
• $L(\lambda |x_{\textrm {undertilde}})=\lambda ^{n}e^{-\lambda \sum x_{i}}$