Introduction

The Bernoulli distribution is a fundamental probability distribution used to model binary or dichotomous outcomes. It is widely used in various fields, such as statistics, machine learning, and psychology. In this blog, we will explore the concept of the likelihood function for the Bernoulli distribution, explain the relevant formulas and equations, and provide a step-by-step guide to calculating the likelihood for a given set of observations.

The Bernoulli Distribution:

The Bernoulli distribution represents a random variable that takes on two possible outcomes: success (typically denoted as 1) with probability p and failure (typically denoted as 0) with probability q = 1 - p. The probability mass function (pmf) of the Bernoulli distribution is given by:

f(x; p) = p^x * (1-p)^(1-x)

Where:

• f(x; p) represents the probability of observing the outcome x (either 0 or 1) in a single trial of the Bernoulli distribution.
• p is the probability of success (i.e., the parameter of the distribution).

The Likelihood Function:

The likelihood function measures the probability of observing a given set of outcomes (data) for a fixed value of the parameter. For the Bernoulli distribution, the likelihood function is calculated by multiplying the probabilities of the observed outcomes.

Let's consider a set of n independent and identically distributed (i.i.d) observations, denoted as x₁, x₂, ..., xₙ. The likelihood function L(p) for the Bernoulli distribution is defined as:

L(p) = f(x₁; p) * f(x₂; p) * ... * f(xₙ; p)

Where:

• L(p) represents the likelihood function.
• f(xᵢ; p) denotes the probability of observing the outcome xᵢ (either 0 or 1) for the i-th observation.
• p is the parameter of the Bernoulli distribution (i.e., the probability of success).

Calculating the Likelihood:

To calculate the likelihood for a given set of observations, we substitute the observed values of xᵢ into the likelihood function and multiply them together.

For example, let's assume we have a set of n observations: x₁, x₂, ..., xₙ. The likelihood function for these observations is:

L(p) = f(x₁; p) * f(x₂; p) * ... * f(xₙ; p)

To simplify the calculation, we can rewrite the likelihood function as:

L(p) = p^Σxᵢ * (1-p)^(n-Σxᵢ)

Where:

• Σxᵢ represents the sum of all observed values (the count of successes).

Maximum Likelihood Estimation:

The goal of Maximum Likelihood Estimation (MLE) is to find the value of the parameter p that maximizes the likelihood function. This is achieved by taking the derivative of the log-likelihood function with respect to p and setting it to zero. Solving this equation provides the Maximum Likelihood Estimator (MLE) for the parameter.

In the case of the Bernoulli distribution, the log-likelihood function is:

log L(p) = Σxᵢ * log(p) + (n-Σxᵢ) * log(1-p)

To find the MLE for p, we take the derivative of the log-likelihood function with respect to p:

d/dp (log L(p)) = (Σxᵢ/p) - ((n- Σxᵢ)/(1-p)

Setting this derivative to zero and solving for p, we obtain:

(Σxᵢ/p) - ((n-Σxᵢ)/(1-p)) = 0

Simplifying this equation leads to:

Σxᵢ = np

Therefore, the Maximum Likelihood Estimator (MLE) for p in the Bernoulli distribution is:

p̂ = Σxᵢ / n

Conclusion

The likelihood function plays a crucial role in estimating the parameter of a probability distribution based on observed data. In the case of the Bernoulli distribution, the likelihood function is calculated by multiplying the probabilities of the observed outcomes. The Maximum Likelihood Estimation (MLE) method allows us to find the parameter value that maximizes the likelihood function, providing the most likely estimate based on the observed data. Understanding the likelihood function and its formulas is fundamental for performing statistical inference and parameter estimation in various applications, ranging from hypothesis testing to machine learning algorithms.