Intraclass Correlation (ICC) Calculator

Use the intraclass correlation coefficient when you need to assess how consistent raters or measurements are. If multiple judges score the same essays, or a medical test is repeated on the same patients, ICC tells you whether ratings agree—not just correlate, but actually produce similar values. It's the standard measure for inter-rater reliability in clinical research, psychology, and quality assessment.

What Is Intraclass Correlation?

The Intraclass Correlation Coefficient (ICC) measures reliability and agreement for continuous measurements. It's used when multiple raters or measurement occasions are involved.

Question answered: "How consistent are measurements between raters or across occasions?"

Range: 0 to 1 (can be negative theoretically)

• ICC = 0: No reliability
• ICC = 1: Perfect reliability

When to Use ICC

Inter-Rater Reliability

Multiple raters score the same subjects:

• Judges scoring performances
• Clinicians rating symptom severity
• Coders categorizing responses

Test-Retest Reliability

Same measurement on multiple occasions:

• Repeated blood pressure readings
• Survey responses over time
• Performance assessments

Intra-Rater Reliability

Same rater on multiple occasions:

• Consistency of a single evaluator

ICC Forms and Models

There are multiple ICC formulations—choosing correctly is crucial.

Three Decisions

1. Model (Random vs. Fixed)

• One-way random: Each subject rated by different raters (rare)
• Two-way random: Raters are a random sample from a larger population
• Two-way mixed: Raters are the only raters of interest (fixed)

2. Type (Single vs. Average)

• Single measures: Reliability of a single rater
• Average measures: Reliability of the mean of k raters

3. Definition (Consistency vs. Agreement)

• Consistency: Relative ranking (ignores systematic differences)
• Absolute agreement: Exact scores (includes systematic differences)

ICC Notation

The standard notation is ICC(Model, Type):

The Formulas

Two-Way Random, Single Measure, Absolute Agreement: ICC(2,1)

$ICC(2,1) = \frac{MS_R - MS_E}{MS_R + (k-1)MS_E + \frac{k}{n}(MS_C - MS_E)}$

Where:

• $MS_R$ = mean square for rows (subjects)
• $MS_C$ = mean square for columns (raters)
• $MS_E$ = mean square error
• $k$ = number of raters
• $n$ = number of subjects

Two-Way Mixed, Single Measure, Consistency: ICC(3,1)

$ICC(3,1) = \frac{MS_R - MS_E}{MS_R + (k-1)MS_E}$

Example Calculation

3 raters score 10 subjects:

ANOVA decomposition:

• MS_R (subjects) = 245.6
• MS_C (raters) = 12.4
• MS_E (error) = 8.2

ICC(2,1):

$ICC(2,1) = \frac{245.6 - 8.2}{245.6 + 2(8.2) + \frac{3}{10}(12.4 - 8.2)} = \frac{237.4}{263.26} = 0.90$

Interpretation: Excellent reliability (ICC = 0.90).

Interpretation Guidelines

Context-Specific Standards

• Clinical decisions: ICC > 0.90 preferred
• Research: ICC > 0.70 often acceptable
• Screening: ICC > 0.80 recommended

Confidence Intervals

Use F-distribution to construct CIs:

$F_L = \frac{MS_R}{MS_E \times F_{1-\alpha/2}}$

Report 95% CIs to show precision of the ICC estimate.

Example: ICC = 0.85 [95% CI: 0.72, 0.93]

Choosing the Right ICC

Flow Chart

1. Are raters fixed or random?

   • Fixed (specific raters of interest) → Two-way mixed
   • Random (sample from population) → Two-way random
   • Different raters for each subject → One-way

2. Single rater or mean of raters?

   • Using one rater's score → Single
   • Averaging multiple raters → Average

3. Is systematic difference important?

   • Yes (need exact agreement) → Absolute agreement
   • No (relative ranking matters) → Consistency

Common Scenarios

ICC vs. Correlation

Pearson Correlation

• Measures linear relationship
• Ignores systematic bias
• Always between -1 and +1

ICC

• Measures agreement/reliability
• Can account for systematic bias (absolute agreement)
• Depends on population variability

Example: Two raters with perfect correlation (r = 1.0) but one always rates 10 points higher:

• Pearson r = 1.0
• ICC (consistency) = 1.0
• ICC (absolute agreement) < 1.0

Standard Error of Measurement

From ICC, calculate SEM:

$SEM = SD \times \sqrt{1 - ICC}$

This estimates measurement error in original units.

Minimum Detectable Change

$MDC = 1.96 \times \sqrt{2} \times SEM = 2.77 \times SEM$

Change must exceed MDC to be considered real (not measurement error).

Applications

Medical Imaging

Agreement between radiologists reading scans.

Physical Therapy

Reliability of range-of-motion measurements.

Psychology

Agreement between clinicians on diagnosis ratings.

Education

Consistency of essay grading across teachers.

Sports Science

Reliability of performance tests.

Reporting Results

Standard format:

"Inter-rater reliability was assessed using a two-way mixed effects model with absolute agreement. The ICC was .87 (95% CI: .79-.93), indicating good reliability."

Full report:

"Three trained raters independently scored 50 writing samples. Inter-rater reliability was evaluated using ICC(3,1) for absolute agreement. The single-measures ICC was .85 (95% CI: .76-.91), indicating good reliability. The average-measures ICC was .94 (95% CI: .90-.97)."

Common Mistakes

Using Wrong ICC Form

Choosing consistency when absolute agreement matters (or vice versa).

Not Reporting CI

ICC is estimated from sample—always include confidence interval.

Confusing with Kappa

ICC is for continuous data; Cohen's kappa is for categorical data.

Ignoring Variance Structure

ICC depends on between-subject variability. Low ICC may reflect homogeneous subjects rather than poor raters.

Factors Affecting ICC

Between-Subject Variability

More variability → higher ICC (if raters are consistent)

Within-Subject Variability

More measurement error → lower ICC

Number of Raters/Occasions

More raters → higher ICC (for average measures)

Range Restriction

Narrow range of scores → lower ICC

Negative ICC Values

Theoretically possible when:

• Within-subject variance > between-subject variance
• Often indicates problem with data or design
• Usually interpreted as 0

Historical Context

The ICC concept was developed by Ronald Fisher in the 1920s for analyzing hereditary traits. Shrout and Fleiss (1979) clarified the different forms and their appropriate uses, establishing the notation still used today.

Related Calculations

• Cronbach's Alpha — Internal consistency
• Pearson Correlation — Linear relationship
• Standard Error Calculator — Measurement precision
• Bland-Altman Analysis — Method comparison

Summary

The Intraclass Correlation Coefficient measures reliability and agreement for continuous measurements between raters or occasions. Choose the appropriate ICC form based on your design: one-way vs. two-way model, single vs. average measures, and consistency vs. absolute agreement. ICC > 0.75 indicates good reliability; > 0.90 is excellent. Always report confidence intervals and specify which ICC form was used. Calculate SEM and MDC to interpret reliability in measurement units.