Skip to main content

Parameter estimation

parameter-estimation

Parameter Estimation

Fundamentals

Problem Statement

Suppose that the population distribution follows a parameteric model $f(x|\theta)$ and given a random sample $X_1,X_2, ..., X_n$ from the population $X_i\tilde{} f(x|\theta)$, estimate the parameter of interest $\theta$

Basic assumption in parametric estimation is that the population distribution follows some parameteric model. Here, parametric models are those of the form:

$$\mathcal{F}=f(x,\theta), \theta\in\Theta$$

where $\Theta\subset R^k$ is the parameter space, and $\theta$ is the parameter.

Example

  1. Normal distribution has two parameters $\mu$ and $\sigma$

Terminologies

  1. Estimator $\hat{\theta}$ is a rule to calculate an estimate of a given quantity (model parameter) based on observed data.
  2. Estimate is a fixed value of that estimator for a particular observed sample.
  3. Statistic is a function of the data, e.g. sample mean
  4. Population distribution
  5. Sampling distribution of a statistic

Example

  • A pool seeks to estimate the proportion $p$ of adult residents of a city that support building a new sport stadium. Suppose that n is the sample size and $\hat{p}$ is the sample proportion, the rule that calculates the sample proportion is called the estimator of the population proportion. The actual value of sample proportion on the observed sample is called the estimate. The sample proportion is a statistic of the sample, note that the statistic itself doesn't need to associate with any parameter of interest.

Point Estimation

Point estimation involves the use of sample data to calculate a simple value which is the best estimate of an unknown population parameter.

Method of Moments

Let $X_1, X_2, ..., X_n$ are iid random variables from a parametric model $f(X,\theta)$ where $\theta=(\theta_1,\theta_2, ..., \theta_k)$ is a vector of $k$ parameters. We are interested in estimating $\theta$.

Moments

  • $\mu_k=E[(X-c)^k]$ is the k-th (theoretical) moment of the distribution around $c$, for k=1, 2, etc.

  • $A_k=1/n\sum_{i=1}^n (X-c)^k_i$ is the k-th sample moment around $c$, for k=1,2,etc.

Moments are often used to indicate moments around zero ($c=0$).
For $k>1$, we also use $c=\mu$, central moments. The second order central moment is the variance.

Suppose that the first K order moments of population exists, equating K theoritical moments to K sample moments gives us K equations with K unknowns.

$$E(X^k)=\frac{1}{n}\sum_{i=1}^nX^k$$

Solving these equations gives us the Method-of-moment estimators for K parameters of interest.

Examples 1 Method of moments estimator for uniform distribution

Assume that $X\tilde{}U(a,b)$ where a,b are unknown. We obtain a sample (1,2,3,4,5) from the uniform population, find the method-of-moments estimator for a,b.

The density function is

\begin{equation} f(x)= \begin{cases} \frac{1}{b-a} & a \leq x \leq b\\ 0 & \mbox {otherwise} \end{cases} \nonumber \end{equation}

The first theoretical moment:

$$E(X)=\int_a^bxf(x)dx=\frac{x^2}{2(b-a)}\biggr|_a^b=\frac{a+b}{2}$$

The second theoretical central moment:

$$E(X^2) = Var(X)+E(X)^2$$

$$Var(X)=\int_a^b(X-\frac{a+b}{2})^2*\frac{1}{b-a}dx=\frac{(b-a)^2}{12}$$

Comments

Post a Comment

Popular posts from this blog

Word embeddings

In this post, we are going to talk about word embedding (or word vector), which is how we represent words in NLP. Word embedding is used in many higher-level applications such as sentiment analysis, Q&A, etc. Let's have a look at the most currently widely used models. One-hot vector is a vector of size V, with V is the vocabulary size. It has value 1 in one position (represents the value of this word "appears") and 0 in all other positions. [0, 0, ... 1, .., 0] This is usually used as the input of a word2vec model. It is just operating as a lookup table. So this one-hot encoding treats words as independent units. In fact, we want to find the "similarity" between words for many other higher-level tasks such as document classification, Q&A, etc. The idea is: To capture the meaning of a word, we look at the words that frequently appear close-by this word. Let's have a look at some state-of-the-art architectures that give us the results of word ve...
Post a Comment
Read more

Quick text files merging, data preparation

It's very often that in natural language processing, you will have to re-format your data to take as inputs to different systems. In this case, these simple linux commands will help you do it much quicker without having to write a script. 1. Merging two files to one file with two column Input f1 looks like this: 1 2 3 4 Input f2 looks like this: a b c d Output f3 will look like this: 1  a 2  b 3  c 4  d Command: paste f1 f2 > f3  The delimiter by default is a tab. You can also define it (for example, separated by a comma) as follows: paste -d ',' f1 f2 > f3 2.  Create a line number to each line of a text file Assume that you want to create an index to each line in a text file, i.e. inserting a line number and then a tab before the content of each line: Input f1: a b c d Output f2: 1  a 2  b 3  c 4  d Command: nl f1 > f2 3. Joining two files with a common field Input f1: 1   aaa...
1 comment
Read more