Continuous-time Markov chain

This article includes a list of references, related reading, or external links, but its sources remain unclear because it lacks inline citations. Please help improve this article by introducing more precise citations. (August 2020) (Learn how and when to remove this message)

A continuous-time Markov chain (CTMC) is a continuous stochastic process in which, for each state, the process will change state according to an exponential random variable and then move to a different state as specified by the probabilities of a stochastic matrix. An equivalent formulation describes the process as changing state according to the least value of a set of exponential random variables, one for each possible state it can move to, with the parameters determined by the current state.

An example of a CTMC with three states ${\displaystyle \{0,1,2\}}$ is as follows: the process makes a transition after the amount of time specified by the holding time—an exponential random variable ${\displaystyle E_{i}}$, where i is its current state. Each random variable is independent and such that ${\displaystyle E_{0}\sim {\text{Exp}}(6)}$, ${\displaystyle E_{1}\sim {\text{Exp}}(12)}$ and ${\displaystyle E_{2}\sim {\text{Exp}}(18)}$. When a transition is to be made, the process moves according to the jump chain, a discrete-time Markov chain with stochastic matrix:

${\displaystyle {\begin{bmatrix}0&{\frac {1}{2}}&{\frac {1}{2}}\\{\frac {1}{3}}&0&{\frac {2}{3}}\\{\frac {5}{6}}&{\frac {1}{6}}&0\end{bmatrix}}.}$

Equivalently, by the property of competing exponentials, this CTMC changes state from state i according to the minimum of two random variables, which are independent and such that ${\displaystyle E_{i,j}\sim {\text{Exp}}(q_{i,j})}$ for ${\displaystyle i\neq j}$ where the parameters are given by the Q-matrix ${\displaystyle Q=(q_{i,j})}$

${\displaystyle {\begin{bmatrix}-6&3&3\\4&-12&8\\15&3&-18\end{bmatrix}}.}$

Each non-diagonal entry ${\displaystyle q_{i,j}}$ can be computed as the probability that the jump chain moves from state i to state j, divided by the expected holding time of state i. The diagonal entries are chosen so that each row sums to 0.

A CTMC satisfies the Markov property, that its behavior depends only on its current state and not on its past behavior, due to the memorylessness of the exponential distribution and of discrete-time Markov chains.