When performing an Intraclass Correlation Coefficient (ICC) analysis, the decision between using single scorings versus average scorings relates to how you interpret the reliability of measurements and the practical context of your study. The choice affects the estimated reliability and how it applies to future measurements or decision-making.

1. Single Scorings

Single scorings ICC (often labeled as ICC(2,1) or ICC(3,1) depending on the model used) assesses the reliability of individual measurements made by a single rater or instrument. It evaluates how well one rater or measurement instance would perform in estimating a subject’s true score or value.

• What it measures: Single scorings ICC tells you the degree of agreement or consistency you can expect from just one rater or one measurement. It answers the question: How reliable will the measurement be if I use one rater or instrument?
• When to use single scorings:
  • Real-world scenarios involve single raters or measurements: If future assessments only involve one rater or one measurement per subject, you should use single scorings. For example, if a clinical study involves one doctor measuring a patient’s blood pressure or one radiologist interpreting an MRI, the single-scoring ICC gives the best reliability estimate.
  • Costs or logistical constraints limit the use of multiple raters: In some situations, using multiple raters or taking repeated measurements is impractical, so understanding the reliability of a single measurement is critical.
• Practical Example:
  • In a clinical setting, if only one doctor evaluates patient symptoms, the single ICC will tell you how reliable that doctor’s evaluations are. If the single scoring ICC is low, it suggests that a single rater’s assessment is unreliable, and additional strategies (e.g., training or multiple measurements) may be needed to improve reliability.

2. Average Scorings

Average scorings ICC (often labeled as ICC(2,k) or ICC(3,k)) assesses the reliability of the mean of multiple measurements or raters. This approach increases reliability by reducing the effect of random error or individual variability between raters or measurements. The more raters or measurements averaged, the higher the reliability, as random fluctuations tend to cancel out.

• What it measures: Average scorings ICC evaluates how reliable the average of multiple measurements is in estimating the true score. It answers the question: How reliable is this combined estimate if I use the average of multiple raters or measurements?
• When to use average scorings:
  • Multiple raters or repeated measurements are available: If the study or future assessments involve taking the average of multiple raters or measurements, this ICC is appropriate. For instance, if several doctors are assessing a patient and want to know how reliable their combined average score is, you would use average scorings ICC.
  • You want to improve reliability: Averaging multiple measurements naturally increases reliability because it smooths out inconsistencies between raters or measurement instances. Average scorings ICC is ideal when you aim to reduce the impact of individual variability.
• Practical Example:
  • In a clinical trial, if multiple doctors assess patient symptoms and their ratings are averaged to make a clinical decision, the average score ICC will indicate how reliable that score is. If the average ICC is high, it suggests that combining multiple raters provides a very reliable estimate of patient status.

Comparison of Single vs. Average Scorings

			Factor	Single Scorings ICC	Average Scorings ICC
			Purpose	Measures reliability of one rater or one measurement	Measures reliability of the average of multiple raters or measurements
			Use Case	When future assessments are expected to use single raters or single measurements	When future assessments will use the average of multiple raters or measurements
			Interpretation	Reflects the reliability of a single rating, typically lower	Reflects the reliability of averaged ratings, typically higher
			Practical Context	Useful when only one rater or measurement is available or practical (e.g., clinical practice)	Useful when multiple raters or repeated measurements are expected or feasible (e.g., research studies)
			Reliability	Generally lower due to more variability from a single rater	Generally higher due to averaging, which reduces random error

Why Does Averaging Increase Reliability?

Averaging multiple scores reduces the influence of random error or individual differences in scoring. Each rater or measurement might have some bias or random fluctuation, but when scores are averaged, those errors cancel each other out, leading to a more stable and accurate estimate of the true score. As a result, the average scorings ICC often yields higher reliability than single scorings ICC.

Practical Considerations for Choosing Between Single and Average Scorings

• Study Design and Future Application: Consider the design of your study and how measurements will be applied in the real world. If future users of the ICC will rely on a single measurement or rater, you need to know the reliability of that single rating (single scorings ICC). If multiple raters or repeated measures are practical, average scorings ICC is more appropriate.
• Feasibility of Multiple Measurements: If obtaining multiple raters or repeated measurements is costly or difficult (e.g., due to time, financial constraints, or availability of raters), you might have to rely on single scorings ICC, even though the reliability might be lower.

Summary

• Single scorings ICC is used when you are concerned about the reliability of one rater or one measurement and applies to real-world settings where only a single measurement is feasible.
• Average scorings ICC is used when multiple measurements or raters are combined, and it gives a higher reliability estimate because it averages out individual variability and error.

Your choice between single versus average scorings depends on the measurement process you are assessing and the practical context in which future measurements will occur.

 

1. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.
2. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163.
3. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.