Evidence-based assessment/Step 4: Revise probabilities based on intake assessments

Click Here for Landing Page

HGAPS New for Fall 2022: HGAPS and Psychology Conferences

Click Here for Landing Page

HGAPS is finding new ways to make psychological science conferences more accessible!

Here are examples from APA 2022 and the JCCAP Future Directions Forum. Coming soon... ABCT!
~ More at HGAPS.org ~

---

Medical disclaimer: This page is for educational and informational purposes only and may not be construed as medical advice. The information is not intended to replace medical advice offered by physicians. Please refer to the full text of the Wikiversity medical disclaimer.

← Step 3: Evaluate risk and protective factors and moderators

Evidence-based assessment

Step 5: Gather collateral, cross-informant perspectives →

Prediction: Revise Probabilities Based on Initial Assessments

Steps to put into practice

The idea of updating a probability based on new information is an old one.

Nomogram for updating probabilities during assessment

A probability nomogram is a visual way of updating probabilities. It turns Bayes' Theorem into a connect-the-dots exercise.

Psychometric Considerations

With regard to the concern that the DLRs are influenced by the reliability, convergent, and discriminant validity, and validity of the collateral informants, this is all true. It also is already “baked in” the DLR. The DLR is an alternate form of effect size; in fact, it is possible to convert r or Cohen’s d into an area under the curve (which is a plot of paired coordinates of sensitivity and specificity, meaning that the DLR is tied at a deep level to the effect size -- Swets, Dawes & Monahan talked about the slope of the line tangent to the AUC at a given threshold as a way of expressing this relationship). Why bother with a different effect size? Because DLRs make it possible to translate the assessment result into an updated probability for the individual client. Cohen’s d or Pearson’s r do not by themselves, and they would require even more arcane formula to get there (see Wiggins. 1973, Chapter 4, for example). The strength of the EBM approach to DLRs is that it establishes a common framework for translating all assessment results into a common metric that updates the probability for an individual client. If the reliability of the measure attenuates the observed effect size, then the DLR would be smaller (and the amount that it would change the posterior probability would shrink). If the convergent validity is higher, then the effect size would be larger (and the DLR would be more informative again). More subtly, if the discriminant validity is lower, then the scores would often be elevated for reasons besides the presence of the intended target. This would degrade the specificity, again reducing the AUC (or the Cohen’s d for the separation between the distribution of scores for the target condition versus the distribution for the comparison condition). Again, the net result would be a lower effect size, and necessarily a more modest DLR. At a deep level, the psychometric concerns are already summarized in the DLR -- with the same caveat that the DLR is a property of test scores or assessment results in the context of a particular sample and for a particular purpose… the same caveat that applies to all assessment per current standards (***).

Annotated bibliography

Using Diagnostic Likelihood Ratios

Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2011). Evidence-based medicine: How to practice and teach EBM (4th ed.). New York, NY: Churchill Livingstone.

  This is an excellent introduction to Evidence Based Medicine. The chapters about diagnosis and harm both describe and illustrate the use of diagnostic likelihood ratios with probability nomograms. 

References

← Step 3: Evaluate risk and protective factors and moderators

Evidence-based assessment

Step 5: Gather collateral, cross-informant perspectives →