{ "category": "Statistical functions", "display_name": "PEARSON", "entry_kind": "built_in_function", "eval_api": { "method": "POST", "url": "https://handbook.dnacalc.com/api/eval" }, "function_id": "FUNC.PEARSON", "implementation_state": { "answer_bank": null, "detail": "Recorded: 6 workbook cells.", "dnacalc_commit": "9e1884369a678158dd82d3d74188daf986995e87", "in_catalogue": true, "state": "matches", "state_label": "Matches every recorded Excel answer", "summary": "Matches all 6 recorded answers", "workbook_corpus": { "cells": 6, "matched": 6 } }, "implementations": [ { "commit": "9e1884369a678158dd82d3d74188daf986995e87", "flavour": "excel-compatibility-candidate", "language": "Rust", "name": "CalcFunc", "own_code": [ "https://github.com/DnaCalc/DnaCalc/blob/9e1884369a678158dd82d3d74188daf986995e87/crates/calc-func/src/catalogue/statistics.rs#L365-L367" ], "repository": "https://github.com/DnaCalc/DnaCalc", "shared_code": [ "ported/paired_stats_common::means", "ported/paired_stats_common::correlation_from_pairs" ], "source": "https://github.com/DnaCalc/DnaCalc/blob/9e1884369a678158dd82d3d74188daf986995e87/crates/calc-func/src/catalogue/statistics.rs#L365-L367", "state": "matches", "suite": null, "summary": "Matches all 6 recorded answers" } ], "json_url": "https://handbook.dnacalc.com/api/functions/FUNC.PEARSON.json", "name": "PEARSON", "notes": [], "schema": "efh.function-reference/v1", "section_order": [ "what-it-does", "try-it", "quirks", "how-to-implement", "version-differences", "alternatives" ], "sections": { "alternatives": "\n### Relationships\n\n\n- **`CORREL`** computes the same mathematical quantity. Two Excel surfaces, one statistic —\n and this pair carries a specific place in the published literature. **McCullough and Wilson**,\n in their series on Excel's statistical accuracy in *Computational Statistics & Data Analysis*,\n report that in Excel versions through the early 2000s `CORREL` and `PEARSON` — algebraically\n identical — returned **different** answers on ill-conditioned data, with `PEARSON` implemented\n through the numerically inferior formula, and that this was among the defects addressed in\n later versions. The Handbook names that finding as published literature. It has not re-run\n those tests, restates none of their figures, and makes no claim about any current Excel build.\n What the reader should take from it is the shape of the risk: **two names for one statistic in\n one product is exactly the configuration in which two different formulas hide**, and it is why\n comparing `PEARSON` against `CORREL` is the first probe on this page.\n- **`RSQ`** is `r²`: `RSQ(y, x) = PEARSON(x, y)^2` mathematically. Squaring destroys the sign\n and, numerically, halves the significant digits of a near-`1` correlation — so `RSQ` computed\n as `PEARSON^2` and `RSQ` computed directly need not agree.\n- **`SLOPE`**, **`INTERCEPT`**, **`STEYX`** and **`LINEST`** are the regression family built on\n the same three sums of squares and cross-products. `SLOPE(y, x) = r · σ_y/σ_x`. Unlike\n `PEARSON`, these are **not** symmetric in their arguments.\n- **`COVARIANCE.P`** and **`COVARIANCE.S`** are the unstandardized numerator.\n- **`FISHER`** is the usual consumer: `FISHER(r) = atanh(r)` is the variance-stabilizing\n transform used to build confidence intervals for `r`, and it is `#NUM!` at `r = ±1` — which is\n precisely why an implementation that returns `1.0000000000000002` for perfectly collinear data\n breaks a downstream formula that has nothing wrong with it.\n- **Confused with**: Spearman's rank correlation (which Excel has no direct function for —\n `PEARSON` over `RANK` values is the usual construction), and with the regression slope, which\n `r` is not unless both samples are standardized.\n\n\n", "how-to-implement": "\n### Numerical notes\n\n\nThe correlation coefficient is the standard classroom example of a formula that is correct in\nexact arithmetic and disastrous in floating point, and it deserves to be spelled out.\n\n**1. The textbook \"computational formula\" is the trap.** Algebra gives\n\n Σ(xᵢ − x̄)(yᵢ − ȳ) = Σxᵢyᵢ − n·x̄·ȳ\n\nand the right-hand side is attractive because it needs one pass and no stored means. It is also\n**catastrophically cancelling**: both terms are of order `n·x̄·ȳ`, their difference is of order\n`n·σ_xσ_y`, and the relative error is amplified by roughly\n\n (x̄ · ȳ) / (σ_x · σ_y)\n\nwhich is the product of the two coefficients of variation, inverted. For data with a large mean\nand a small spread — measurements around 1000 varying by 0.1, dates, prices, sensor readings —\nthat factor is `10⁸` or more and **the answer loses eight digits before anything else happens**.\nThe same one-pass form applied to `Σxᵢ² − n x̄²` can even produce a negative sum of squares,\nafter which the square root in the denominator is a domain error on data that is perfectly fine.\n\nThis is the exact defect the McCullough–Wilson series reports having found in Excel's\nstatistical procedures, and the exact defect that the recommended alternatives fix.\n\n**2. The two-pass centred formula is the honest default.** Compute `x̄` and `ȳ` first, then\naccumulate `Σ(xᵢ−x̄)(yᵢ−ȳ)`, `Σ(xᵢ−x̄)²` and `Σ(yᵢ−ȳ)²` in a second pass. Cost: one extra pass\nand one array traversal. Benefit: the cancellation is gone, because the quantities being summed\nare the quantities the answer is made of. Chan, Golub and LeVeque (*The American Statistician*,\n1983) analyse this comparison for the variance and give the error bounds; the covariance case is\nidentical in structure. A **corrected two-pass** variant, which adds back the (theoretically\nzero) residual `(Σ(xᵢ−x̄))²/n`, recovers another digit or two and costs nothing.\n\n**3. Updating formulas when one pass is required.** Welford's recurrence for the variance and\nits bivariate extension (the Youngs–Cramer form) maintain the centred sums incrementally:\n\n dx = xᵢ − x̄ₙ₋₁\n x̄ₙ = x̄ₙ₋₁ + dx/n\n C ← C + dx · (yᵢ − ȳₙ)\n\nThese are one-pass, streaming, and numerically comparable to the two-pass method. There is no\naccuracy argument left for the naive computational formula; only an ignorance-of-the-literature\nargument.\n\n**4. The denominator and the bound.** `√(Sxx · Syy)` should be formed as\n`√Sxx · √Syy` rather than `√(Sxx·Syy)` when the two sums differ wildly in magnitude, to avoid\noverflow or underflow of the product. And the result should be clamped to `[−1, 1]`: exact\ncollinearity can produce a computed `|r|` a few ULP above `1`, which is mathematically\nimpossible and downstream-fatal.\n\n**5. Summation order.** For long arrays the three sums are themselves a summation-accuracy\nproblem. Pairwise or Kahan summation costs little and removes the `O(n)` error growth of naive\naccumulation. Where a function is expected to reproduce another system's bits, the summation\norder becomes part of the specification rather than an implementation detail.\n\n**What a careful independent implementation does**: two passes (or Welford), corrected means,\npairwise summation, separated square roots, and an explicit clamp — with a stated bound. See\n[About implementation options](../model/07-implementation-options.md).\n\n\n", "quirks": "\n### Result and edge cases\n\n\nReturns `Number` in `[−1, +1]`.\n\n- **A constant sample.** Undefined mathematically. Microsoft's `PEARSON` page documents no error\n for this case at all — see Errors below, where the Handbook records that as a gap.\n- **Mismatched lengths** are a documented `#N/A`.\n- **Perfect correlation.** `r = ±1` should be returned exactly for exactly-collinear data, and\n whether it is depends entirely on the formula used — the classical failure is a computed `r`\n slightly exceeding `1` in magnitude, which then produces a `#NUM!` in any downstream\n `ATANH`, `FISHER` or confidence-interval calculation. Clamping to `[−1, 1]` is a defensible\n and commonly omitted implementation step.\n- **Arrays.** `PEARSON` is an aggregate over two grids, not a lift kernel; it returns a scalar.\n\n\n\n### Errors\n\n\nAs documented by Microsoft on the `PEARSON` page (retrieved for this page):\n\n| Error | Documented condition |\n|---|---|\n| `#N/A` | `array1` and `array2` are empty, or have a different number of data points |\n\n**That is the whole documented error surface — one row — and it does not cover the case that\nmatters most.** The `PEARSON` page states nothing about what happens when a sample is constant\nand the denominator is zero. That is the natural `#DIV/0!` condition, it is documented on the\n`CORREL` page for the mathematically identical function, and it is absent here. The reference\nengine's battery renders degenerate inputs and returns a division error for them; its outcomes\nshow beside this page.\n\nThe Handbook records this as a **documentation gap** rather than as a behavioural claim: the\ndivide-by-zero behaviour is not in dispute among implementations, but Microsoft's `PEARSON`\npage does not document it, while its `CORREL` page documents the analogous condition for the\nsame computation. Two pages describing one function, with different error coverage.\n\nError values arriving in either array propagate under the ordinary coercion rules.\n\n\n", "sources": "- Microsoft, \"PEARSON function\" —\n \n (syntax; the two required arguments; the rule that text, logicals and empty cells in an array\n or reference argument are ignored while zeros are included; and the single `#N/A` condition\n for empty or unequal-length arrays). Retrieved for this page; the equation is an image and was\n not read.\n- B. D. McCullough and B. Wilson, \"On the accuracy of statistical procedures in Microsoft Excel\",\n *Computational Statistics & Data Analysis* (1999, 2002, 2005) — the published record of the\n `CORREL`/`PEARSON` discrepancy and of the one-pass formula's failure on ill-conditioned data.\n Named as literature; no figure restated, and no claim made about any current build.\n- T. F. Chan, G. H. Golub and R. J. LeVeque, \"Algorithms for computing the sample variance:\n analysis and recommendations\", *The American Statistician* 37 (1983) 242–247 — the comparative\n error analysis of the one-pass, two-pass, corrected two-pass and updating formulas.\n- B. P. Welford, \"Note on a method for calculating corrected sums of squares and products\",\n *Technometrics* 4 (1962) 419–420 — the streaming update used for the bivariate case.\n- M. Welinder's work on Gnumeric's statistical functions — the standard practical account of\n implementing this family in a spreadsheet.\n- Handbook, [Coercion and lifting](../model/02-coercion-and-lifting.md) and\n [About implementation options](../model/07-implementation-options.md).\n\n\n", "version-differences": "No comparison between specific Excel builds is recorded here yet. This does not establish that all builds behave alike.", "what-it-does": "\n`PEARSON(array1, array2)` returns the **Pearson product-moment correlation coefficient** `r`\nbetween two paired samples — the standardized measure of linear association.\n\nFor paired observations `(x₁,y₁) … (xₙ,yₙ)` with means `x̄` and `ȳ`:\n\n r = Σ (xᵢ − x̄)(yᵢ − ȳ)\n ────────────────────────────────────────\n √( Σ (xᵢ − x̄)² · Σ (yᵢ − ȳ)² )\n\nMicrosoft's page gives this equation as an image and states that `x̄` and `ȳ` are\n`AVERAGE(array1)` and `AVERAGE(array2)`.\n\nEquivalently, in the language of moments,\n\n r = Cov(x, y) / ( σ_x · σ_y )\n\nwith the same denominator convention (population or sample) used top and bottom, so the choice\ncancels and `r` is unambiguous where variance and covariance are not.\n\n**Domain and range.** Defined for `n ≥ 2` paired numeric observations with **both** samples\nnon-constant. The range is the closed interval `[−1, +1]`:\n\n r = +1 exactly iff y is an increasing affine function of x\n r = −1 exactly iff y is a decreasing affine function of x\n r = 0 no linear association (which is not the same as independence)\n\nThe bound is the Cauchy–Schwarz inequality applied to the centred vectors, and equality holds\nexactly when they are parallel. Geometrically `r` is the cosine of the angle between the two\ncentred data vectors in `ℝⁿ` — which is the single most useful way to think about it, because it\nexplains at once why `r` is invariant under any positive affine rescaling of either variable\nand why it is undefined when either centred vector is zero.\n\n**Invariances.** For `a, c > 0`:\n\n r(ax + b, cy + d) = r(x, y)\n r(x, y) = r(y, x) symmetric\n r(−x, y) = −r(x, y) sign flips with orientation\n\nThe affine invariance is why `r` is dimensionless and comparable across units, and it is also\nthe source of the numerical hazard described below: a computation that is not itself\nshift-invariant will return different answers for data that differ only by a large offset.\n\n**Degeneracy.** If either sample is constant, the corresponding centred vector is zero, the\ndenominator vanishes, and `r` is undefined — genuinely undefined, not zero. This is not an edge\ncase to be papered over: constant columns arise constantly in real spreadsheets.\n\n\n\n### Arguments\n\n\nMicrosoft's page gives two required arguments:\n\n| Argument | Meaning | Required |\n|---|---|---|\n| `array1` | \"A set of independent values\" | yes |\n| `array2` | \"A set of dependent values\" | yes |\n\nThe independent/dependent labelling is a convention only — `r` is symmetric in its arguments,\nunlike `SLOPE`, `INTERCEPT` and `FORECAST`, which are not. Swapping the arguments of `PEARSON`\nchanges nothing; swapping the arguments of `SLOPE` changes the answer.\n\n> \"The arguments must be either numbers or names, array constants, or references that contain\n> numbers.\"\n>\n> \"If an array or reference argument contains text, logical values, or empty cells, those values\n> are ignored; however, cells with the value zero are included.\"\n\nNote what that rule does **not** say: it does not say how the pairing survives the omission. If\n`array1` has text in row 7 and `array2` has a number there, is the pair dropped, or is the text\ndropped and the numbers re-aligned? The published wording is compatible with both readings and\nthey give different answers. See the probe list.\n\n" }, "signature": { "display": "PEARSON(array1, array2)", "parameters": [ { "description": null, "name": "array1", "optional": false, "repeats": false }, { "description": null, "name": "array2", "optional": false, "repeats": false } ], "trailing_repeats": false }, "sources": { "metadata": { "admission": { "label": "implemented", "reason": null }, "arity": { "max": 2, "min": 2 }, "category": "Statistical functions", "classification": { "arg_preparation_profile": "ValuesOnlyPreAdapter", "coercion_lift_profile": "Custom", "determinism_class": "Deterministic", "error_collapse_profile": "None", "fec_dependency_profile": "None", "host_interaction_class": "None", "kernel_signature_class": "Custom", "rich_value_usage": "RichBlind", "surface_fec_dependency_profile": "RefOnly", "thread_safety_class": "SafePure", "volatility_class": "NonVolatile" }, "descriptions": { "long": null, "short": "Pearson correlation coefficient of two data sets", "short_source": "dna-calc-own" }, "docs": { "english_description": "Returns the Pearson product moment correlation coefficient", "microsoft_url": "https://support.microsoft.com/en-us/office/pearson-function-0c3e30fc-e5af-49c4-808a-3ef66e034c18" }, "entry_kind": "built_in_function", "localized_family_names": null, "localized_names": [ { "locale": "ar-SA", "name": "PEARSON" }, { "locale": "bg-BG", "name": "PEARSON" }, { "locale": "cs-CZ", "name": "PEARSON" }, { "locale": "da-DK", "name": "PEARSON" }, { "locale": "de-DE", "name": "PEARSON" }, { "locale": "el-GR", "name": "PEARSON" }, { "locale": "en-AU", "name": "PEARSON" }, { "locale": "en-GB", "name": "PEARSON" }, { "locale": "en-US", "name": "PEARSON" }, { "locale": "es-ES", "name": "PEARSON" }, { "locale": "et-EE", "name": "PEARSON" }, { "locale": "fi-FI", "name": "PEARSON" }, { "locale": "fr-FR", "name": "PEARSON*" }, { "locale": "he-IL", "name": "PEARSON" }, { "locale": "hr-HR", "name": "PEARSON" }, { "locale": "hu-HU", "name": "PEARSON" }, { "locale": "id-ID", "name": "PEARSON" }, { "locale": "it-IT", "name": "PEARSON" }, { "locale": "ja-JP", "name": "PEARSON" }, { "locale": "ko-KR", "name": "PEARSON" }, { "locale": "lt-LT", "name": "PEARSON" }, { "locale": "lv-LV", "name": "PEARSON" }, { "locale": "nb-NO", "name": "PEARSON" }, { "locale": "nl-NL", "name": "PEARSON" }, { "locale": "pl-PL", "name": "PEARSON" }, { "locale": "pt-BR", "name": "PEARSON" }, { "locale": "pt-PT", "name": "PEARSON" }, { "locale": "ro-RO", "name": "PEARSON" }, { "locale": "ru-RU", "name": "PEARSON" }, { "locale": "sk-SK", "name": "PEARSON" }, { "locale": "sl-SI", "name": "PEARSON" }, { "locale": "sr-Latn-RS", "name": "PEARSON" }, { "locale": "sv-SE", "name": "PEARSON" }, { "locale": "th-TH", "name": "PEARSON" }, { "locale": "tr-TR", "name": "PEARSON işlevi" }, { "locale": "uk-UA", "name": "PEARSON" }, { "locale": "vi-VN", "name": "PEARSON" }, { "locale": "zh-CN", "name": "PEARSON" }, { "locale": "zh-HK", "name": "PEARSON" }, { "locale": "zh-TW", "name": "PEARSON" } ], "signature": { "display": "PEARSON(array1, array2)", "parameters": [ { "description": null, "name": "array1", "optional": false, "repeats": false }, { "description": null, "name": "array2", "optional": false, "repeats": false } ], "trailing_repeats": false }, "surface_name": "PEARSON", "version_marker": null, "xlcall": { "code": "312", "symbol": "xlfPearson" } }, "microsoft": "https://support.microsoft.com/en-us/office/pearson-function-0c3e30fc-e5af-49c4-808a-3ef66e034c18" }, "try_it": { "formula": "=PEARSON({1;2;3},{1;2;3})" }, "url": "https://handbook.dnacalc.com/functions/PEARSON.html", "variants": [], "versions": [] }