Open main menu
An example of a Kaplan–Meier plot for two conditions associated with patient survival.

The Kaplan–Meier estimator,[1][2] also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data. In medical research, it is often used to measure the fraction of patients living for a certain amount of time after treatment. In other fields, Kaplan–Meier estimators may be used to measure the length of time people remain unemployed after a job loss,[3] the time-to-failure of machine parts, or how long fleshy fruits remain on plants before they are removed by frugivores. The estimator is named after Edward L. Kaplan and Paul Meier, who each submitted similar manuscripts to the Journal of the American Statistical Association. The journal editor, John Tukey, convinced them to combine their work into one paper, which has been cited about 55,000 times since its publication.[4][5]

The estimator of the survival function (the probability that life is longer than ) is given by:

with a time when at least one event happened, di the number of events (e.g., deaths) that happened at time and the individuals known to have survived (have not yet had an event or been censored) up to time .

Basic conceptsEdit

A plot of the Kaplan–Meier estimator is a series of declining horizontal steps which, with a large enough sample size, approaches the true survival function for that population. The value of the survival function between successive distinct sampled observations ("clicks") is assumed to be constant.

An important advantage of the Kaplan–Meier curve is that the method can take into account some types of censored data, particularly right-censoring, which occurs if a patient withdraws from a study, is lost to follow-up, or is alive without event occurrence at last follow-up. On the plot, small vertical tick-marks indicate individual patients whose survival times have been right-censored. When no truncation or censoring occurs, the Kaplan–Meier curve is the complement of the empirical distribution function.

In medical statistics, a typical application might involve grouping patients into categories, for instance, those with Gene A profile and those with Gene B profile. In the graph, patients with Gene B die much more quickly than those with Gene A. After two years, about 80% of the Gene A patients survive, but less than half of patients with Gene B.

In order to generate a Kaplan–Meier estimator, at least two pieces of data are required for each patient (or each subject): the status at last observation (event occurrence or right-censored) and the time to event (or time to censoring). If the survival functions between two or more groups are to be compared, then a third piece of data is required: the group assignment of each subject.[6]

Problem definitionEdit

Let   be a random variable, which we think of as the time until an event of interest takes place. As indicated above, the goal is to estimate the survival function   underlying  . Recall that this function is defined as

 , where   is the time.

Let   be independent, identically distributed random variables, whose common distribution is that of  :   is the random time when some event   happened. The data available for estimating   is not  , but the list of pairs   where for  ,   is a fixed, deterministic integer, the censoring time of event   and  . In particular, the information available about the timing of event   is whether the event happened before the fixed time   and if so, then the actual time of the event is also available. The challenge is to estimate   given this data.

Derivation of the Kaplan–Meier estimatorEdit

Here, we show two derivations of the Kaplan–Meier estimator. Both are based on rewriting the survival function in terms of what is sometimes called hazard, or mortality rates. However, before doing this it is worthwhile to consider a naive estimator.

A naive estimatorEdit

To understand the power of the Kaplan–Meier estimator, it is worthwhile to first describe a naive estimator of the survival function.

Fix   and let  . A basic argument shows that the following proposition holds:

Proposition 1: If the censoring time   of event   exceeds   ( ), then   if and only if  .

Let   be such that  . It follows from the above proposition that

 

Let   and consider only those  , i.e. the events for which the outcome was not censored before time  . Let   be the number of elements in  . Note that the set   is not random and so neither is  . Furthermore,   is a sequence of independent, identically distributed Bernoulli random variables with common parameter  . Assuming that  , this suggests to estimate   using

 

where the last equality follows because   implies  .

The quality of this estimate is governed by the size of  . This can be problematic when   is small, which happens, by definition, when a lot of the events are censored. A particularly unpleasant property of this estimator, that suggests that perhaps it is not the "best" estimator, is that it ignores all the observations whose censoring time precedes  . Intuitively, these observations still contain information about  : For example, when for many events with  ,   also holds, we can infer that events often happen early, which implies that   is large, which, through   means that   must be small. However, this information is ignored by this naive estimator. The question is then whether there exists an estimator that makes a better use of all the data. This is what the Kaplan–Meier estimator accomplishes. Note that the naive estimator cannot be improved when censoring does not take place; so whether an improvement is possible critically hinges upon whether censoring is in place.

The plug-in approachEdit

By elementary calculations,

 

where the one but last equality used that   is integer valued and for the last line we introduced

 

By a recursive expansion of the equality  , we get

 

Note that here  .

The Kaplan–Meier estimator can be seen as a "plug-in estimator" where each   is estimated based on the data and the estimator of   is obtained as a product of these estimates.

It remains to specify how   is to be estimated. By Proposition 1, for any   such that  ,   and   both hold. Hence, for any   such that  ,

 

By a similar reasoning that lead to the construction of the naive estimator above, we arrive at the estimator

 

(think of estimating the numerator and denominator separately in the definition of the "hazard rate"  ). The Kaplan–Meier estimator is then given by

 

The form of the estimator stated at the beginning of the article can be obtained by some further algebra. For this, write   where, using the actuarial science terminology,   is the number of known deaths at time  , while   is the number of those persons who are alive at time  .

Note that if  ,  . This implies that we can leave out from the product defining   all those terms where  . Then, letting   be the times   when  ,   and  , we arrive at the form of the Kaplan–Meier estimator given at the beginning of the article:

 

As opposed to the naive estimator, this estimator can be seen to use the available information more effectively: In the special case mentioned beforehand, when there are many early events recorded, the estimator will multiply many terms with a value below one and will thus take into account that the survival probability cannot be large.

Derivation as a maximum likelihood estimatorEdit

Kaplan–Meier estimator can be derived from maximum likelihood estimation of hazard function.[7] More specifically given   as the number of events and   the total individuals at risk at time  , discrete hazard rate   can be defined as the probability of an individual with an event at time  . Then survival rate can be defined as:

 

and the likelihood function for the hazard function up to time   is:

 

therefore the log likelihood will be:

 

finding the maximum of log likelihood with respect to   yields:

 

where hat is used to denote maximum likelihood estimation. Given this result, we can write:

 

Benefits and limitationsEdit

The Kaplan–Meier estimator is one of the most frequently used methods of survival analysis. The estimate may be useful to examine recovery rates, the probability of death, and the effectiveness of treatment. It is limited in its ability to estimate survival adjusted for covariates; parametric survival models and the Cox proportional hazards model may be useful to estimate covariate-adjusted survival.

Statistical considerationsEdit

The Kaplan–Meier estimator is a statistic, and several estimators are used to approximate its variance. One of the most common estimators is Greenwood's formula:[8]

 

where   is the number of cases and   is the total number of observations, for  .

For a mathematical derivation of the equation above, click on "show" to reveal

Greenwood formula is derived[9] by noting that probability of getting   failures out of   cases follows a binomial distribution with failure probability  . As a result for maximum likelihood hazard rate   we have   and  . To avoid dealing with multiplicative probabilities we compute variance of logarithm of   and will use the delta method to convert it back to the original variance:

 

using martingale central limit theorem, it can be shown that the variance of the sum in the following equation is equal to the sum of variances:[9]

 

as a result we can write:

 

using the delta method once more:

 

as desired.


In some cases, one may wish to compare different Kaplan–Meier curves. This can be done by the log rank test, and the Cox proportional hazards test.

Other statistics that may be of use with this estimator are the Hall-Wellner band[10] and the equal-precision band.[11]

SoftwareEdit

  • Mathematica: the built-in function SurvivalModelFit creates survival models.[12]
  • SAS: The Kaplan–Meier estimator is implemented in the proc lifetest procedure.[13]
  • R: the Kaplan–Meier estimator is available as part of the survival package.[14][15][16]
  • Stata: the command sts returns the Kaplan–Meier estimator.[17][18]
  • Python: the lifelines package includes the Kaplan–Meier estimator.[19]
  • MATLAB: the ecdf function with the 'function','survivor' arguments can calculate or plot the Kaplan–Meier estimator.[20]
  • StatsDirect: The Kaplan–Meier estimator is implemented in the Survival Analysis menu.[21]

See alsoEdit

ReferencesEdit

  1. ^ Kaplan, E. L.; Meier, P. (1958). "Nonparametric estimation from incomplete observations". J. Amer. Statist. Assoc. 53 (282): 457–481. doi:10.2307/2281868. JSTOR 2281868.
  2. ^ Kaplan, E.L. in a retrospective on the seminal paper in "This week's citation classic". Current Contents 24, 14 (1983). Available from UPenn as PDF.
  3. ^ Meyer, Bruce D. (1990). "Unemployment Insurance and Unemployment Spells" (PDF). Econometrica. 58 (4): 757–782. doi:10.2307/2938349. JSTOR 2938349.
  4. ^ "- Google Scholar". scholar.google.com. Retrieved 2017-03-04.
  5. ^ "Paul Meier, 1924–2011". Chicago Tribune. August 18, 2011.
  6. ^ Rich JT, Neely JG, Paniello RC, Voelker CC, Nussenbaum B, Wang EW (2010). "A practical guide to understanding Kaplan–Meier curves". Otolaryngol Head Neck Surg. 143 (3): 331–6. doi:10.1016/j.otohns.2010.05.007. PMC 3932959. PMID 20723767.
  7. ^ (PDF) https://web.stanford.edu/~lutian/coursepdf/STAT331unit3.pdf. Missing or empty |title= (help)
  8. ^ Greenwood, M. (1926). "The natural duration of cancer". Reports on Public Health and Medical Subjects. London: Her Majesty's Stationery Office. 33: 1–26.
  9. ^ a b (PDF) https://www.math.wustl.edu/%7Esawyer/handouts/greenwood.pdf. Missing or empty |title= (help)
  10. ^ Hall WJ and Wellner JA (1980) Confidence bands for a survival curve for censored data. Biometrika 69
  11. ^ Nair VN (1984) Confidence bands for survival functions with censored data: A comparative study. Technometrics 26: 265–275
  12. ^ "Survival Analysis – Mathematica SurvivalModelFit". wolfram.com. Retrieved 2017-08-14.
  13. ^ The LIFETEST Procedure
  14. ^ "survival: Survival Analysis". R Project. April 2019.
  15. ^ Willekens, Frans (2014). "The Survival Package". Multistate Analysis of Life Histories with R. Springer. pp. 135–153. doi:10.1007/978-3-319-08383-4_6. ISBN 978-3-319-08383-4.
  16. ^ Chen, Ding-Geng; Peace, Karl E. (2014). Clinical Trial Data Analysis Using R. CRC Press. pp. 99–108. ISBN 9781439840214.
  17. ^ "sts — Generate, graph, list, and test the survivor and cumulative hazard functions" (PDF). Stata Manual.
  18. ^ Cleves, Mario (2008). An Introduction to Survival Analysis Using Stata (Second ed.). College Station: Stata Press. pp. 93–107. ISBN 978-1-59718-041-2.
  19. ^ lifelines docs
  20. ^ "Empirical cumulative distribution function – MATLAB ecdf". mathworks.com. Retrieved 2016-06-16.
  21. ^ [1]

Further readingEdit

External linksEdit