Define Markov Chain. Explain with suitable example.

Definition

A Markov Chain is a stochastic process that satisfies the Markov property: the probability of transitioning to any particular state depends only on the current state and not on the sequence of events that preceded it. In other words, the future is independent of the past given the present.

Formally, for a discrete-time Markov chain with states $S=\{s_1,s_2,\dots ,s_n\}$: $$P(X_{t+1}=j\mid X_t=i,X_{t-1}=k,\dots ,X_0=s_0)=P(X_{t+1}=j\mid X_t=i)$$

The system is fully described by its Transition Probability Matrix $P$, where element $p_{ij}$ represents the probability of moving from state $i$ to state $j$.

Key Characteristics

• States: A finite or countable set of possible conditions.
• Transition Probabilities: ${\sum }_j{p}_{ij}=1$ for all $i$.
• Stationary Distribution: A probability vector $\pi$ such that $\pi P=\pi$, representing the long-term behavior of the chain.

Example: Weather Prediction

Consider a simple weather model with two states: Sunny (S) and Rainy (R).

Transition Rules:

• If today is Sunny, there is a 0.8 probability it will be Sunny tomorrow and 0.2 probability it will be Rainy.
• If today is Rainy, there is a 0.3 probability it will be Sunny tomorrow and 0.7 probability it will be Rainy.

Transition Matrix $P$: $$P=\left[\begin{array}{cc}0.8& 0.2\\ 0.3& 0.7\end{array}\right]$$ Rows represent current state (S, R); Columns represent next state (S, R).

Calculation: If today is Sunny ($X_0=S$), the probability that it is Sunny two days from now ($X_2=S$) is calculated using $P^2$: $$P^2=\left[\begin{array}{cc}0.8& 0.2\\ 0.3& 0.7\end{array}\right]\left[\begin{array}{cc}0.8& 0.2\\ 0.3& 0.7\end{array}\right]=\left[\begin{array}{cc}0.7& 0.3\\ 0.39& 0.61\end{array}\right]$$ The probability $P(X_2=S\mid X_0=S)$ is the element at row 1, column 1 of $P^2$.

Result: The probability that it will be Sunny two days after a Sunny day is 0.7.

Application in Simulation

In simulation modeling, Markov chains are used to model systems where the next state is probabilistically determined by the current state, such as:

• Customer queue lengths in service systems.
• Reliability of components (Working/Failure states).
• Inventory levels in supply chain management.