Evaluation Theories/Week 5: Evaluation Theory Panel: Patton, Cronbach, Chen, Guba & Lincoln

yawnnotes Click here for the Google Doc with class notes - * These need to be imported and cleaned up for Wikiversity!

Q&A & Feedback

Today; while T1 hearts Cronbach, I heart Huey T. Chen;

Sophisticated Saying: “If it ain’t broke, don’t fix it.” - With Moderating; T1 did it so well that we’ll keep her moderating.

In Week 7: TODO: Turn in a 2-page Reflection paper.

You will have gone through the 12 Theorists; and we want this paper to be an opportunity to take you across theorists.

• 2. 1. Top three salient issues
• 3. Brief: To pages; double spaced; Times New Roman; 1” Margins - For Week 7.

Come prepared with a question (J: ???)

Patton

- Intended Use by Intended Users: Not for general stakeholders; just the primary stakeholders. 

Utility Feasibility Propriety and Accuracy;

Sarah: Used to be an evangelical preacher; said talking to people about whether their life was consistent with their values.

13:17:11 Lee Joseph Cronbach

Evaluation as a ____ for education; adress particular needs; create a forum where everyone can interact; educate; influence social policy.

Guba

PhD Stats. Qual Research Special Interest Group Eval Center Western Mich. Teas A&M

(Awards are meaningless unless they’ve got brand recognition…aka: Nobel or bust) 

Lincoln:

Construct knowledge for the specific context;

Huey Chen

• Born & Raised in Taiwan; U Alabama Birmingham Public Health
• (WHY THE NAME OF AWARDS?!!?! :))
• National Evaluation System for Evaluating a CDC-Funded HIV Education Program (Josh: WOOT! lol, I should go evaluate the Sudan one I produced!)

Pseudo-Patton: How do you think your academic training / background influenced how you think?

AS PSEUDO-PATTON, I THINK THE ORGANIZATIONAL PART OF MY DEGREE WAS IMPORTANT; - I KEEP UP WITH MY TEACHING BACKGROUNDWITH THE INTERNATIONAL DEVELOPMENT EVALUATION TRAINING; I DO A LOT OF TRAINING

CDC: TAXONomies for how to develop a useful evaluation. (J: I want ot make the taxonomies of EVALUATION - the use I can’ 13:30:52ake a big impact in; too many brilliant people are already working in it. The bigger picture is not together though. I love being in that position, because i’m usually

_____ completely reshaped his ideas of what evaluation is.

Cronbach

…. Unlike BLakc-box Evaluation;

GUBA

Valueing: Basic approach: construct knowledge that is context bound.

- Positivism; one basic reality; understood through generally quantitative means (J: EVERY INDIVIDUAL CONSTRUCTS THEIR REALITY THROUGH QUANTITATIVE MEANS. That’s why if you repeat something often enough people start to believe it.) 

(J: What is good about this system: it’s good for looking at who is well developed in what area. . .but mutually exclusive and collectively exhaustive (MECE) (Ethan Rasiel, 1999, 'The McKinsey Way', p. 6; and which I just ran across in MIT’s introduction to Probability course…)

Misc.

13:54:29 - ____ Most well developed is SOcial Programming;

Because…

Mentor: - Cronbach a bit abstract often

14:19:09 - Cronbach UTOS:

– . . .

External & Internal Validity: Campbell & Cronbach:

- Internal Validity: Generalizing from sample to population in same context: 
- External validity 

14:26:08 - Guber ;

Stakeholder APproach; Social Sciences Approach; ____ Approach.

Former Theory &

* Stakeholder THeory:  - Values of responsiveness to Stakeholders views;  
* Social sciences: Program theory; deriving based on how “program is actually operated and existing social science theory knowledge.” - (J: HOw good is SS Theory knowledge?) 

Balance between Stakeholder and Scientific Credibility.

1. Type 1: False Positive
2. Type 2: You think there isn’t an effect when there really is.
3. Type 3:

• - the right answer to the wrong question. This is sometimes called a Type 0 error
• - Type III error occurs when you correctly conclude that the two groups are statistically different, but you are wrong about the direction of the difference. (http://graphpad.com/support/faqid/1080/)
• - Chen: Theor

y was good; implementation was flawed. (Tiffany Berry P315Z; 2014-02-19)

Hermeneutic exchange

Definition:

• 8-Step Process:
• 1. You don’t know everything
• 2. Look at Stakeholders separately (using Maximum variation sampling)
• 3.
• 4.

Q:What information do you gather from these stakeholders? -

Do you educate stakeholders to ensure they can keep on doing things after you leave?

- Not education. . . we on’t want to influence. . . more enlightenment to other stakeholder’s point of view. (J: better word than enlightenment? 

- What makes this different from “Including Everyone else’s point of view?

Cronbach - Policy-Making; Info for decision-makin:

I’d advise: 1. Multiple perspectives 2. Embrace people usually excluded (J: Rawls) but not while silencing those who are already in control 3. Explicate dynamics of power and privilege interculturalhe program. 4. ___ context; not idly serve ____

Multiple Stakeholder Groups: You’ve Identified some: Most important groups of stakeholders that you would involve: 1. Public Servants (those who make the program for the public) (J: Our favourite public servants: Google. Re: Google Books Project. Until you @#$@# them and they became evil when they realized their idealism wouldn’t fly in the real world) 2. Members of the Program (who it’s made for)

Mentioned Leverage: Previous Ambiguity; Adjustment at the Margins?

INTEGRATIVE VALIDITY MODEL

three components. Campbelian Typology of Validity

1. Viability
2. Effectuality
3. Transferability

Superior to Campbell because Campbell’s approach Lab -> Real world; mine goes: “Can it work in Real world?” - then “Actually works in real world?” then ____

Transferrable validity: Revision of Campbell that takes into account the stakeholder’s perspective (Lisa ____, 2014-02-19)

- very practice oriented; more expansive idea of validity than the traditional typology.

Viable Validity: Should we even bother; do we have the resources; buy-in from stakeholders? Is it cost effective?

- Traditionally this is last step; with Chen this is First Step. 

Typologies: External Validity; Focus is on how to rule out threats to validity. Becomes less and less relevant to stakeholders.

Viable Vlaidity: a world-class center doesn’t matter at all if you put it on a hill and there’s no escalator up: IT doesn’t matter at all if we can’t get to it; can’t access it.

Really good thing about Halcolm from ___ (girl in the pink jacket)

Student Generated Questions:

Gets continued throughout the process: at every stage you’re doing this exchange; ensuring that entire population is informed; contributing; feeding back information through entire profcess. I don’t think you write as clearly about the process of evaluation as ____. . . (Guba & Lincoln)

Q&A:

Constant implementation failure but people persist in pursuing the overarching causal or program theory; what advice would you give in that situation? -

A: We deal with that all the time. Implmeentation is always flawed: it’s just levels of flawedness. Sometimes its so severe. . . one project wer’e advising now, Dosage is a big part of the implmentation; but once they looked at the [dosage] - they were serving these kids 2 hours per year, and were supposed to improve their resilience and grit; . .. ____ - (Great personal story)

Implementation is always flawed: that’s why it’s absolutely critical to measure it, always.