Evaluation Theories/Week 10: Connecting Theory to Practice: The Bunche-Da Vinci Case

In-Class Notes 2014-03-26 PSYCH 315z – Comparative Evaluation Theory

Information about Midterm

Midterm: Mean 84%; SD 12.8; Median 87.5%

Issues;

• Program Social Science & Evaluation Theory
• Categorization Schemes;
• Approaches and Associated Theorists
• Compare & Contrast Theorists on 4 Dimensions (J: What are the four dimensions?)
• Compare & Contrast Theorists on Shadish, Cook, & Leviton “Shadish, Cook, & Leviton’s 5 Components of Good Evaluation Theory

(J: I did good on Categorization Schemes)

Issues - Truth; problem solving; use; integration.

Theory Scheme -

Compare & Contrast: Components of a Good Theory;

Points were taken off for incorrect information; for not comparing and contrasting;

We fed all the scores into a computer, so there should be no issues with adding. I was really happy with the results.

There are these “Google Docs” floating around - I have not seen them: but do you know what I’m talking about - there are apparently some documents that may have made their way into your inboxes; and some of it seems to have been taken from not this class: maybe from other classes in previous years: and made their way into documents; and I have found some incorrect information on those Google Docs!

There were a series of incorrect responses on a question; and it seemed to have been linked to potentially wrong information - if you don’t know what the document is, I would say that’s a really good thing: don’t maybe use them – 

The information that we talked about in class was correct;and then, in some of these documents there might have been incorrect information: I want you to be careful with memorizing documents that might not be topics we verified in class;

Q: Can you give an example of incorrect information:

1. The phrase “Double-blind” for Goal Free evaluation makes no sense; you can’t have a double-blind social science thing; that came up on tons of exams: A term that’s not in the course material and not in any of the course materials
2. The Classification Scheme: Tarek’s classification scheme about the rivers and . . .that wasn’t something we had talked about in class; I don’t know if that was brought in from
3. Some of them were larger things: like conflating Causality with RCT’s; we don’t want you to equate those two things in your mind: although some theorists do, that is not a standard that is directly linked to RCTs. I don’t know if that came from a document or not.

It seemed like there was enough specific language.

I would say: Take notes in class; focus on your class notes. Use original resources: don’t just go to this Google doc and modify away - because it could be wrong, and then you’ll be in trouble.

Other questions on the exam?

For the final, we’ll be translating theories into specific context, so you still need to be familiar with theories.

We want to ensure that the exam is also used as a teaching tools so that you’ll know the correct answers.

Q: Regarding information; in our notes we actually did have “Double Blind” for Scriven -

Q: What would we like to do for studying compared to the Google Docs which might be wrong? A: Take really good notes during class; Get with your buddies that did really well on the exam and see how they did it.

I would like to add that we are holding you accountable for much greater depth on a few theorists that you did independent research on; the introductory chapter in Alkin gives an overview. We want you to be familiar with these key features which are represented in your assigned reading.

Student: - I recorded the debates; when Silvana (the lead TA co-teaching the class) would clarify something I would note that. ; - and then with the list of who had each person; I would email them;

Reflection Papers

The top themes that came out of the reflection papers were: Values; Stakeholders; Context; Use

This practice week, we will be incorporating different stakeholders; you’ll be getting those papers back as well; and the way we do this grading is Check minus; check; check plus.

Check plus we read and went, “Wow! You go girl! (Or boy) Check - Check minus - not quite there; didn’t respond to the prompt; wrote something different; or didn’t respond to the prompt adequately.

One thing was feedback on the class that was part of the graded assignment. We really value your feedback!

Paper

The syllabus has detailed information about the paper:

We developed prompts for different types of programs: you should select the program description that resonates with you.

There are three, that cover different topics and areas, with roughly the same information, to figure out how you might want to integrate a theorist’s perspective into that

We can give you structured feedback; on things like “Is this the right direction?”

In general, we will be looking for

1. accuracy in representing the theorists
2. specificity that we can figure out the s
3. appropriate application - Rational for context of program; based on your assumptions. We want to see logical connections. We want to see that those things make sense together.

Concerns about inaccurate information:

There are 8 dimensions which we want you to consider

1. Outline the ____ of the dimensions
2. You can say, “This is what my understanding of Greene’s approach to stakeholders is, and we can respond, “Yes! That is her approach.” That would be, we would suggest, the best way to start.
3. You can make assumptions; make it clear, perhaps with pullout boxes; like Donaldson did; “Assumption 1: The players will be interested in decision making.”
4. Do something similar with social science theory, and body of research

(J: ToDO: _1 FIND PROMPTS; _ 2 Identify Resources to use)

(J: So concerned about Accuracy. . .)

Q: Social science theory or bodies of research: it hasn’t pointed to some sort of social science theory. . .if it doesn’t end up pointing to one particular theory, is that okay? A: Yes; there’s a body of research of what “children in poverty” means; there isn’t necessarily one single theory of it all; but there is a body of literature that addresses that group. For the social psych folks in this room, you have theories every where.

As we talked about last module; the disciplinary training has a lot to do with their approaches to evaluation.

This will be a paper that you will flunk if you start it three days before it’s due: within the constraints of all the other things that you are doing in your life; try to manage your time effectively so that you get feedback from us; write a beautiful paper in beautiful prose so that we’re very impressed with your integration of knowledge.

Any other questions?

Module 2: Theory to Practice

1. Practice
   1. Design & Method Choice
      1. What that looks like
   2. Politics & Interpersonal Issues
      1. A really important piece of the context that we often don’t see in the basic research literature; meaning “How does politics and different politics of the context. . . of the stakeholder . . . of the interaction; how does that influence the ways in which we can do evaluation; what’s feasible to do; how do you take it into account and leverage that vs. try to ignore, hide, or push it away. So; how do we embrace and leverage it.
   3. Interpersonal Issues
      1. A lot of evaluation is being socially confident; interact with people who might disagree with you quite a bit: how do you keep your cool.
   4. Context & Cultural Competency
      1. We were trying to figure out what you had talked about in Foundations re cultural competency so as not to duplicate. (We read the AEA statement on cultural competency)
   5. Research Evaluation & Theory Development & Professional Issues
      1. We’ll talk about the professional issues; have a couple of guest speakers come in here and there to talk about their research; and that basically caps what we’re going to look at.

Today we’ll look at how these theoretical approaches are being translated into practice; if you don’t have any questions about where we are headed, we can go ahead and start.

So; the Bunche-Da Vinci Case:

They had different features; but how they used those features and played out was different between all four of them.

Next week, we’ll talk about a contingency based approach to evaluation; it will be similar in a lot of ways: engaging stakeholders; connecting to different value systems.

Q: Can you elaborate on what “contingency-based theory” means? A: Can we answer it next week? That’s going to be a large part of the class next week.

C: I found interesting how certain practices can be done for different intentions.

I hope you enjoyed reading them and have them in good working memory, because we are going to do a case activity here: we’re going to talk as a group about the context for the Bunche-Da Vinci Case; divide into 8 groups;

2. 4 Theorist Groups;
3. 4 Stakeholder groups
4. Prep a 5-7 minute presentation (25 min);
5. Each group presents (40-60 min total)
6. Stakeholders choose their evaluators and provide a rationale for their choice
7. Class Debrief (30 min): strengths, weaknesses, and issues in translating theory to practice.

We want to see how you guys are conceptualizing the approaches that were presented in the case, and how those approaches connect or don’t connect with stakeholder needs and groups.

Q: Are we doing what the reading did, where they present their evaluation? A: Highlight the strengths of the plan; how you’re able to address specific concerns; the evaluation plans you have. . .

Q: Are the stakeholders going first saying, “This is what we need?” and then the evaluators go and the stakeholders choose.

WE’re going to count you off: (J: I is 5)

What are some of the attributes or characteristics of the context?

1. Funky partnership between org and district (non-charter school; but using different curriculum)
2. Assessments from state level and different corporations; Assessments were not in agreement
   1. Emphasized reading and english language (J: that was the school’s decision right, not Da Vinci)
3. Teachers had longer school years; and had to do training after the school
   1. Students were grouped by ability rather than grade
   2. Kids should maybe be moving more because teachers couldn’t keep up with the assessment tool that the Da Vinci people thought was a good idea.

Challenges that they talked about?

1. Teachers asked to focus outside of content area
2. Parents complained about high use of technology taking away from …
3. High turnover / attrition of students
4. Parent’s viewed the school positively; there’s not explanation for high turnover.

How realistic is this case? Do we feel it’s pretty realistic? Is it really set up to be an egregious, challenging case.

1. Positives the program can offer: Implicit standards; rough area;
   1. Where standardized state tests are going; weighing Da Vinci to state tests; whether that’s a fair comparison.

Mentor: We’re movign to common core (In the US) - trying to promote critical thinking; problem solving; communication, in a way that promotes their college and career readiness: beacuse we’re seeing kids graduating from highschool that know how to answer multiple choice tests but no critical thinking skills.

In my mind this is absolutely 100% typical; not just of hte educational context; but every context I’ve ever worked in.

You talk to different people; get different stories; different assessments tell discrepant stories. THis is an accurate case of this crazy world that we live in, and trying to bring evaluation to lend some support, and understand and de-mystify some of the chaos.

These contexts tend to be so challenging that it’s not just a simple answer.

Notes from the Role-Plays

School Staff

Principal Garcia: Concerns

1. School performance; literacy problems that I don’t think the Bunche Da-Vinchi partnership has not resolved
2. Concerned about the staff situation at my school;
   1. High turnover
   2. Low retention Rate
      1. I want a strong Team of teachers at a school; 30% of my team teaches outside of their expertise
3. Curriculum Design

Personally, I want to see how to find strong student performacne: as principle I started doing collaborative teaching.

Teacher Representative:

1. WE’re struggling with conflict between AFrican American and Latino students, and looking for a way that this conflict can be resolved; so it doesn’t take away from energer by studetns and teachers.
2. Teaching students that don’t really speak english: a lot of the teachers don’t speak spanish, so there is a gap in communication;
3. We want to know why there is a discrepency of test scores; why they dropped and wen tup again

School Guidance counselor:

1. Heavy load; 35 or more students per day;
2. Want ___ to be

SUperintendent:

Why was tehre an initial increase?

1. Partnership or In-service programs from the division?
2. Why is there a discrepancy between the

The staff can go through the motions; but really undermine the program.

I will find funds from an outside source to fund this project.

Parents

We are the parents of the studetns who are at the school; I am one of the very involved parents at the school:

I’m really concerned about how these test scores are going down: things we’d like to have answered:

1. Does this program work to begin with?
2. Will it work in our area?
3. If changes are made to the program, how soon will these be implemented and students getting results?
4. We want to know how this program compares to others: is it worth keeping our children at this school?
5. Is this school stable enough to have a program like this? It seems to have a lot of potential to be great, but with the faculty turnover, can this program help our students?

I’ve got 2 preteens and a 7 year old; we moved here from across the country; I’m struggling with the progress of my youngest; - As regards the evaluation proposals:

1. Given the info needs we’re looking for; we think your approach . . . how does it work in our situation; why doesn’t it work in our situation?
2. Concerns abotu the amount of time the approach would take. Improving school and finding answers is great; but I don’t want my kid to be ____

Jeff Cooper;

8-year old child; he’s eight years old; hasn’t progressed very much - he spends a lot of time in front of his computer because he can’t read;

YOu use focus group to gather information to get your info theory: how are you sure you’re gathering a diverse-enough group.

Just moved here; Imigrants; Fluent in Spanish but not English yet; 3 kids;

1. Is the school spending enough time teaching my children english, or recognizing that htey don’t know ENsligh as a first language and trying to educate them.
2. For Greene: your eval is context dependent; but there’s a lot of turnover; will it work in other

Maria - I have two children attending; not their first language;

1. Ms. King; could you - how can this school say that our school is effective if there is so much turnover; to prove that it’s the school and not me?

Final question:

1. How are you going to have us parents be involved in this?

Representatives from Federal and State Level

I think taht schools are incredibly important; want this money to be spent effectively; and I don’t have the information I need on which program to fund, and why.

I like the idea of privatizing things; but don’t have enough information about how these thigns work.

Advisors to President (of Country):

1. We are looking at how to encourage better standards and assessmetns; promote innovation & excellence in the schools.
2. Local programs that can be generalized nation-wide; what’s working in those programs
3. Do test scores improve? That’s our main concern

_____(State comptroller?)

1. Accountability
2. Decrease the achievement gap between student groups
   1. We issue Title 1 achievement awards to students that show progress.
3. Interested in effectiveness

Duncoln;

1. WE’re excited about the $40bln being dedicated; we’re passionate about the money they spend
2. We’re going to be tough holding schools accountable and figuring out what’s working in schools across teh country.

Funding is of huge importance for education; we’re excited to be funding systems and programs that work;

1. We’re concerend about how they work
2. About what programs work.
3. WHo shoudl and shouldn’t be getting funding.

OUr desire in this evaluation is test assessments and ___; goals; discrepencey between goals.

After we find out what works we’d like to know if they’re held accountable

District Representatives

(…)

Donaldson

My approach to evaluation is Program tehory evaluation science (J: he wouldn’t have used this term in this context …..)

1. Cost effectiveness
2. Accuracy
3. Utility

I’ll have a top-notch team with - content specialists ; I’ll do lit reviews (J: “lit reviews - another term he probably wouldn’t use)

We will engage representatives from stakeholders; all of hte groups that talked:

1. Give them a voice
2. take their values into consideration

This is the first stage of evaluation:

1. 3 months of discussion; talking about the context; how it compares to simliar programs
2. From there we’ll develop a big list of questions we can answer; we’ll develop the list. (J: you have some already, right here; are you going to address any of those? (not that they really had time to:))

For focus groups: we would have representative parents;

1. WE would have hte ability to use surveys
2. WE could do interviews;
3. It’s not just limited ot hte focus groups.

We’ll have:

1. English and Spanish speakers.
2. A lot of parent involvement
3. Get their views;
   1. What htey expect fro the program

While of course, we will have to think about all the other stakeholder views.

As regards to whehter this works; can work; by looking at implementation gives us a lot we can talk about:

1. Is it implememted correctly in this school?
2. Mediators and moderators (J: Another term he probably wouldn’t use) : does it work for differnet ability levels
3. Discrepency between test scores: misalignment; We would do a lot of construct vlaidity (J: Another term!) things to see if htey match and align; help to improve those tests (J: Really? )

Ultimate goal of:

1. Student improvement
2. Performance on the test

Let me tell you guys something else: our evaluation is realistic ‘’’and’’’ fair.

We do know that evaluation is going to take a long time; so we put it in stages; think about all the information you can get in three months, before the evaluation has even started.

(J: I would like a comparison of timelines between tehse evaluators. . . )

Henry

We are proposing a 4-stage process:

1. Values inquiry;
   1. Focus groups; selected randomly; (J: too much detail); different economic status and enrollment. INTERJECTION: can you just back up where the 25 schools came from? A: This is not just one schol; we are doing a quasi-experiment (J: whoa . . .that was not handled well. . .I’m going to have to work hard to influence for this evaluator; )
2. ____
3. Implementation Assessment; Implementation Fidelity
4. Outcome Analysis
   1. Net Effects of Da-Vinci Learning Model (J: they keep saying “We’re looking” rather than “we would like to look at” . . . - it’s not in character…)
   2. ___
   3. ___

As you can see this will be large and expensive: but we’re looking at social betterment, change for the greater good; there are very diverse needs; the outcomes we want to achieve are for these children; we want to look at everyone’s perspective; and hope that this affects other schools in california.

WE are looking at the diverse needs of differnet people; and how this works in comparison to other programs.

Our evaluation does NOT look just at test scores: there are other things:

1. WE hope to create a working hypothesis that can assess whether students are meeting college goals, for example.

Assessing teacher turnover rate:

1. we want to incorporate them in our logic-model processing; part of this program changing; assess their needs too

The way we plan to implement has a very strong methodological backing

1. Able to withstand any scrutiny from peer review and so forth.

I want to address how it affects the various stakeholders: BEcause we’re looking at a grander scale:

1. WE’ll help the Federal Government help make sure that things are actually effective
2. WE’re using comparitive data;
3. School staff will benefit because they’ll know if hte time required from them is effective; is it necessary? It is being used effectively
4. Parents: I feel for the parents of this program; I feel that before the implementation of this program it should have been tested: I would be lying if I said that it was going to affect your child right away; but i belivee its

The bottom line is that hte davinci model should have been tested before it was implemented (

We will test the Da Vinci Learning model for:

1. CHildren of minorities
2. Language learners
3. ___

First off: Strong evidence does not come cheap: you’re asking us whether the program works generally, and whehter it works for your program.

1. We’re helping you by looking at your diverse needs
2. Fiscally, we will rely heavily on grant money
3. Lastly; why we’re better than the other three proposals: Each of you asked “ how do you determine that the program is effective?”
   1. We’re finding whether this Da Vinci program works in a general sense; it has trickle-down effects for parents and staff for determining whether this model is a good fit for the staff at the school, and whether it can benefit the (parents).,

Jean King

First of all, I’m practical: our general aprpoach is to emphasize capacity-building.

IN the long run, it won’t cost as much, and you can direct money to the important;

By teaching Staff admin to use evaluations ;they’ll be able to utilize more effectively and taylor to student & learning needs.

Here’s how we’ll conduct these evalutions:

I’ll start off with some of the stakeholder groups that came and spoke:

1. THey seemed to be pretty positive and happy; not much conflict it seems; but conflict we know is something that is inherent in the evaluation process: we expect it and ___ it;
   1. By creating an advisory committee with can-do-attitudes; we want to use Garcia’s upbeat tempo; we have hte evaluation virus (J: super-power might be better term:) that we want to infect everybody with, so get ready to be infected!

It’s about:

1. Building COMmunity
2. Creating Infrastructure
3. Making sense of htese test scores

We’ll do this with:

1. Highly visible participatory ___ and
2. ___ Action ___

Since we are doing capacity building, we are focused on getting teachers and administrators at the local level involved in ___ and in teh future making sure that the changes work.

THe way we include other stakeholder groups, including parents, is high transparency.

1. All of our evaluation is very open so you can see what’s going on.
2. This applies to other districts as well; can easily see waht we do (J: how will you ensure information dissemination that is useful to us? Would it be like the Whole Foods store-by-store metrics that are released to all stores?)

___ PLanning; cycles; good teachers already do this; and our evaluation makes it explicit and ensures the processs is continuous.

Though the three stages are highly transparent; this one, Action Research is private, to maintain the integrity of individual people within the program.

Our approach is not for everybody: I began my career as a Teacher; have a Class A Licence in Eval; ran school departments for years:

WE want to ensure that the evaluand is qualified to be evaluated by our approach; - that you are able to put together - I want to do my homework; and make sure that I have everything put together and have an understanding of how everyone is perceived; capacity of leaders . . .

I’m going to demand a lot of active involvement: A lot of meetings; a lot of ____; and to address Maria’s specific question: over the long term of being able to really see whether it’s what you’re doing or what the school is doing that is creating differences for your child: we’ll be able to bake that into the system.

(J: What if we get a collaboration between King and Henry? King runs the individual thigns so they are self-sustaining in a subset of schools; and Henry runs the meta-analysis and comparison for the Da Vinci Curriculum? - Then that data is fed into the system for continual feedback. . .)

Greene

Greene: what you all care about most: four commitments

1. Contextuality: recognizing the uniqueness and hte context;
2. Inclusion of all stakeholders
3. learning in and thorughout the evaluation
4. engaging with difference and diversity.

How this will play out: we’ll begin by talking with individual stakeholders:

1. Teachers; Parents;
2. THe main questions: quality of hte program itself;
   1. In what ways are stuetns obtaining the meaningful, valuable outcomes you want?

To do this we propose an interpretive case study mixed methods; Different groups; Title 1; try to focus on all those differen tpeople

1. Interviews with teachers administartors parents
2. Curriculum review; Quality of instruction (depending on consent from Teachers union)
3. Provide you with feedback every 3-4 months about how the evaluation is going;

WE want to do this on a 2-year program;

1. hopefully short enough to capture some of the student experiences

We want to establish respectful, reciprocal partnership relationships; all the stakeholders know what’s going on; provide concerns / programs:

Ensure that you all come together and feel what you really want for your kids; what really matters to you.

Also, for holding ourselves accountable; we suggest a meta-evaluation at the very end, by two evaluators:

1. One with the same philosophy as us;
2. ONe with a differnet philosophy;

That way you can getfeedback not only on teh evaluation itself; but also other things.

Take 30 Seconds to pick:

(J: This whole exercise is actually a good model for stakeholder involvement. . . have hte different groups get together, present their interests . . .hear from evaluators!)

Teachers Choice

Donaldson:

1. Addressed concerns about why he’ll engage multiple stakeholder groups
2. MEasured accountability and scores; influence factors
3. Going to take into accoutn how much money it’s going to process.

Parents:

1. Donaldson: Our consern is with understanding
2. why our children are doing
3. How they’re doing

These other approaches seemed to, by including peopel to do diverse backgrounds;

Federal and State

We have funding to direct and resources we need to allocate wisely; we care a lot about strenght of evidence and effectiveness; what works and what doesnt: Henry was the only one taht talked about experimental;

Personal opinion getting context and long-term payoff; I would have chosen King; BEcause the value of educational systems increasing their capacity to self-evaluate;

The initial investment would have been .. .

Question: For STate and Federal level representatives endorse? - She’s prioritizing.

Comment: Da Vinci Corporation tshould have hired henry

- People more broadly involved went with henry; it’s a matter of applying what you need; being involved.

It’s difficult to have every approach meet everyone’s unique perspective.

Mentor:

If you think about henry’s approach, it’s completely inappropriate for those questions that the school wants an answer to. When you think about other worthy questions and worthy values that you could get answers to, that’s where social betterment comes in;

Question: In your experience who will ultimately make the decision of who to hire? A: It would be the Superintendent and Principal.

Again: there was no budget attached to this; so it’s a difficult exercise: again this is a very typical case: you go out; there is chaos raining across the land; and usually the budget to asnwer those questions: take a guess for a reasonable budget for a local school to engage?

The budget for a small local school like this would be about $15k: I wouldn’t endorse any of these . .. (J: jus tover 1,000 students in htis school . . . my 80k estimate was based on 5-7000 students, K-12 - which is about right; 5*15k = $80k).

__ (MISSED THIS PART: FIND IN AUDIO RECORDING)

Henry would not be interested in this evaluation at all: as soon as he modified it to something that he was interested in, he was able to go along. $15k on a daily rate; that’s not too many days of work. . .

Data collection; report writing; dealing with stakeholder groups; how you can do that, accurately, cost-effectively . . .

In those initial conversations it would be really important to figure out: is the client the local school, or is it the Da Vinci corporation?

Lets make this a bigger study; and hte funds are there: then you can adapt hte study; and it goes from the local context to this corporate model: does that model work.

The questions are different; they’re not necessarily alligned to ____; the Da Vinci corporation becomes the main primary client.

So many times, you approach all the problems according to your interests and training; this might not be a context that’s particularly relevant for capacity building (although, Silvana might disagree) - but just because your’e Jean King, I get approached by a new problem . .. yes, in teh long term maybe it will help. Social Betterment: . . . but ___ best for the students; and so on.

I want you to think, as you go to Starbucks, of some of hte critiques of these major approaches . . .

You did a great job presenting strenghts; what we asked you to do; so address critique as we move into design.