Evaluation Theories/Greene, Henry, Donaldson, King Comparison/Greene's "Value-Engaged Approach"

Article Metrics

• Name: Jennifer C. Greene, 'A value-engaged approach for evaluating the Bunche–Da Vinci Learning Academy' - New Directions for Evaluation Special Issue: Theorist's Models in Action, pp. 27–45.
• Original Work Metrics: 7555 words, 48515 characters, 19 pages; Skim Highlights and Comments: 2945 words, 19443 characters
• Notes to Original Work Ratio: 2.6 to 1

What follows is a raw dump of Skim comments and highlights.

A Value-Engaged Approach for Evaluating the Bunche–Da Vinci Learning Academy

Jennifer C. Greene

active engage- ment with values that are drawn principally from democratic and culturally responsive traditions in evaluation (Hood, 1999; House and Howe, 1999; MacDonald, 1977).

Because value-engaged evaluation is fundamentally responsive to the particular character of an evaluation context and the programmatic and policy issues that are present in that context, it is not possible to develop an a priori or preordinate (Stake, 1987) evaluation design using this approach.

concentrates on envisioning these front-end aspects of evaluation in this context

front-end

Assumptions will also be made along the way to enable a more complete story to be told

\ COVER LETTER\ 

Developing an Understanding of the Context and the Program to Be Evaluated

(J: Brilliant: Greene starts with an email. This is helpful because working with "real working data" (or as close as possible to it) can be better for people wanting to learn a process than working with descriptions of those things.)

Principal

District Superintendent

Dear Ms. García:

I am

keenly (This is a filler that adds spice to the email: if it wasn't there, some enthusiasm would be lacking, and it could be read to be something other. . . I think this is indicative that we expect higher-energy interaction with others on first meeting)

committed

eager

would like to learn more

This will enable me to develop an evaluation plan that can yield the kinds of informa- tion likely to be meaningful and useful to your school community (J: sets up deliverable)

meaningful and useful (J: Primary values?)

your school community (J: Primary stakeholder?)

For this purpose, I am hoping that I can visit Bunche–Da Vinci several times this spring and (J: Sets up activities)

review relevant materials (for example, annual reports and previous evaluation studies),

i

talk with various members of your commu- nity

l

simply spend some time at your school.

learn more about the kinds of evaluation priorities that various members of your school community have. (J: Speaks to her value of the democratic process / stakeholder involvement)

I will follow up this email with a phone call to pursue this idea further and hopefully identify particular days for my visits. (J: next-steps) /COVER LETTER/

The Bunche–Da Vinci Community

\REITERATION OF CASE\ (J: good communication practice for active listening: tell them what you just heard)

A very important first step in developing an evaluation plan for the Bunche–Da Vinci Learning Academy is to better understand the particular characteristics and diversities of this school community

especially important not to make assumptions about this particular context based on urban stereotypes.

Learning about this particular school community—its uniqueness, its complexities, and its continuing and dy- namic evolution—is fundamental to developing an evaluation plan

/REITERATION OF CASE/

\TEAM ENGAGEMENT\

Estella Marquez <emarquez@uiuc.edu>, Hashid Robinson <hjr3@uiuc.edu (J: STAKEHOLDERS: EVALUATION TEAM MEMBERS)

/TEAM ENGAGEMENT/

\REITERATION OF CASE\

The Bunche–Da Vinci Learning Academy

/REITERATON OF CASE/

Just What Is Being Evaluated?

\DESCRIPTION OF EVAL PHILOSOPHY\ 

a more complete understanding of the Da Vinci program and its implementa- tion in the Bunche–Da Vinci Learning Academy is a critical initial step in a value-engaged approach to evaluation. (J: "value-engaged" sounds nebulous here; like marketing speak. It would seem to be a critical initial step in ANY approach to evaluation...)

value-engaged

Identification of the evaluand, or just what is being evaluated, is not assumed in this evaluation approach, but rather is a matter of discussion among as many stakeholder groups as possible in this front-end part of the evaluation

The evaluator con- tributes to this discussion her or his expertise and perspectives, including ideas from relevant research literature.

/DESCRIPTION OF EVAL PHILOSOPHY/

\REITERATION OF CASE\

/REITERATION OF CASE/

value commitments of this value-engaged evaluation approach

understanding the character of the pro- gram (and/or institutional structure, like a partnership) to be evaluated in its particular and unique context

contextuality

inclusion of all legitimate stakeholder views and perspec- tives

with special efforts to include the more marginalized people in the context (House and Howe, 1999

(J: i.e.) discouraged teachers and tran- sient families

Identifying the Key Evaluation Questions

Principal

we brought to that meeting key evaluation priorities identified during our con- versations

discussion at

faculty meeting last Thursday

w

our review of relevant documents

over the past month

and

we invited further faculty comment on these priorities during the meeting

the most important clusters of evaluation priorities at this time con- cern the following issues:

the demands of the curriculum

Teachers

feel overloaded with

the long hours

the extra courses

Teachers also feel underappreciated as professionals

With constant teacher turnover, it is hard to develop and sustain a sense of community in the school.

\GOAL\ sustain a sense of community in the school /GOAL/

The Da Vinci curriculum is being questioned by teachers and parents alike for its relevance

power to enable meaning- ful learning

and

concerned about the rele- vance and effectiveness of the curriculum in meeting the needs of English lan- guage learners and students with special needs.

concerns about the integration of most special services into the regular classroom and curriculum

wonder why so many families don’t stay at Da Vinci, but rather transfer after just one or two years.

committed to raising the test scores of participating students

concerned that the school leadership does not have the authority to make changes in the Da Vinci pro- gram as might be needed,

We will use these evaluation priorities as the basis for developing our key evaluation questions and our evaluation plan, (J: What "PRIORITIES"? So far you've talked about concerns, one commitment, and one "wondering")

attached please find a draft of our description of the Bunche–Da Vinci Learning Academy and its surrounding communities. When

we would welcome your feedback on this draft, especially its accu- racy and completeness.

We also intend to share this draft with several teach- ers and community members who have been involved in this school for many years and who also volunteered to provide feedback.

teach- ers

community members

aug- ment our understanding of the quality and effectiveness of the evaluand

\DESCRIPTION OF EVAL PHILOSOPHY\ The major purpose of evaluation in a value-engaged approach is to

This is

educative

requires that the evaluation include assessments of the program

and experienced

as implemented

assessments of what important differences the pro- gram has made in the lives of its participants

evaluation questions attend specifically to the interests of people traditionally underserved in our country, including racial and ethnic minorities, low-income people, immigrants, and people with disabilities.

These frames for evaluation questions

reflect additional value commitments underlying this value-engaged approach to evaluation

A commitment to learning in and through evaluation

to contribute to bet- ter understanding of how well the program being evaluated meets impor- tant needs and interests of participants in the particular setting at hand

in addition to how well participants perform in the program)

A commitment to engaging with difference and diversity

as manifested in the context at hand

to assess how well the program being evaluated meets the particular needs and interests of people traditionally underserved

key eval- uation questions

Overall Question

In what ways and to what extent does the educational program offered at the Bunche–Da Vinci Learning Academy meet the important educational needs of the children and families served by this school, in particular the distinctive needs of English language learners, children from racial and ethnic minority groups, children from low-income families, and children with special needs? And in what ways and to what extent does the struc- ture of the partnership support this primary educational mission?

Subquestions

Quality

"1. What is the quality of the educational program offered at Bunche–Da Vinci Learning Academy for this particular community of children and families?" (p. 44)

Educational Outcomes for Children

"2. To what extent and in what ways are the children at the Bunche-Da Vinci Learning Academy attaining meaningful and valued educational outcomes?" (ibid.)

Establishing the Criteria for Making Judgments of Program Quality

Principal

Sandra Martin

lead teacher

We have completed our analysis of what various members of your school community said when we asked them, “What matters to you most in an educational program for children at this school? And how would you know that the program in this school is a good one?” (J: Double-barreled question)

share this draft among the Bunche–Da Vinci staff

Bunche–Da Vinci staff

Criteria for judging educational quality in this context are complicated by

the presence of the Da Vinci program, which likely has its own criteria of quality (which are not yet known),

the diversity of program stake- holders

draft set of general criteria envisioned as possible in this context includes

high quality, based on relevant the- ory and research (for example, research on effective teachers of minority children) and

curriculum and the pedagogy

perspectives and experiences of the Bunche–Da Vinci staff and families.

offer relevant and valuable approaches to learning

curriculum and pedagogy

for all of Bunche–Da Vinci’s particular population of students.

well aligned with state stan- dards in all subject areas,

curricular program and its assessments

especially language arts and mathematics

students at Bunche–Da Vinci

Bunche–Da Vinci Learning Academy community

public responsibility and authority

The Evaluation Design

rich description of the program and its context (including relevant history)

three parts

begin with

a clear statement of the primary eval- uation purposes and audiences (including the various stakeholder interests in the evaluation),

presentation of the key evaluation questions and critical criteria for judging quality.

Second, the more technical evaluation design section

would spec- ify the overall inquiry design to be used, along with

data-gathering meth- ods and samples,

analysis procedures,

technical quality considerations,

informed-consent procedures

third section of the plan would present information on

reporting plans.

staffing, and

evaluation management

budget

Overall Design.

an interpretive case study with multiple and mixed methods implemented for purposes of understanding selected phe- nomena more comprehensively and with greater insight and value con- sciousness (Greene, Benjamin, and Goodyear, 2001).

Methods and Samples.

emphasize

assessments of the quality of the program experience for students

and teachers

also include analyses of relevant achieve- ment test data

Curriculum Review.

reviewed by approximately five experts in curriculum and the teaching of children like those in the Bunche–Da Vinci school population

address

questions of

curricular quality

using care- fully identified criteria.

curricular fit to the Bunche–Da Vinci school population

elements representing the Da Vinci program’s underlying theoretical rationale

likely to include

principal developers of this program at the Da Vinci Learning Corporation

factors drawn from the research literature

interviewed (J: what about documents? Documents -> Interview)

Classroom and School Observations

One class at each grade level

iden- tified as “representative or typical”

Classes

require teacher and parent con- sent.

observations will be made about once a week for about half of the school day during the months of October, November, February, March, and April, scheduled so that all components of the curriculum are observed mul- tiple times.

observations will generate rich descriptions of the learning activities, interactions, and environments of these classrooms.

focuses

include

engagement and learning of different kinds of children in the various components of the curriculum

character of the integrated in-class services provided

how

time

is used

the evaluation team will record field notes of unstructured observations conducted at other sites in the school, including

lunchroom

playground

fif- teen teachers will be individually interviewed

Teacher and Administrator Interviews

selected to be collectively representative of the school’s faculty, along dimensions of

sample

education

classroom or specialist assignment

experience

longevity at the school

Very new teachers and teachers new to the school will be excluded

race and ethnicity

commitment to the school

(J: less likely to provide "rich" information)

With the consent of the teachers’ union and

Information on Teacher Quality.

the district and school administrators

group-level evaluative information about the Bunche–Da Vinci teaching staff over the past three years

profile of strengths and weaknesses

weaknesses

If access to this information is denied, the eval- uation team will consult with the administration and faculty at Bunche–Da Vinci regarding other avenues to secure data on teaching quality.

Parent Interviews.

Six group interviews of parents

five to ten parents

in each group

formed primarily around the demographics of the school

will include

1) one group of Latino parents of children who have good English fluency

2) one group of Latino parents of children in the Structured English Immersion classes

(3) one group of African American parents

5) one group of parents of children in special education (J: Short Essay Prompt 1: What percentage of the possible combination of named parent groups is this? Short Essay Prompt 2: What percentage of students does this combination reach?)

(4) one group of parents of children in Title I

For all groups, parents will be sampled for representativeness, along

longevity in the commu- nity

age

gender

work

income status

number of children in the school

longevity in the Bunche–Da Vinci school

(J: LOGISTICS:)

Child care, transportation, and food will be provided for all group interviews.

interviews will be conducted in Spanish if the Spanish-speaking parents prefer this

major focus of these interviews will be to

gather parent descriptions of and evaluative reflections on their children’s learning experiences in this school

sense of connectedness and commitment to the school.

their

Test and Test Score Analysis

full content analysis

comparison of

teacher- developed assessments

Da Vinci program tests

state tests and standards

with the consultation of

additional content experts as needed

two experts in psychometrics

stu- dent performance on all of these tests over the past four years, using the first year before the Da Vinci partnership as a baseline, will be analyzed by stu- dent subgroup and skill area.

analysis will endeavor to identify the spe- cific areas of relative strength and weakness displayed by various subgroups of students

for sub- groups and specific skill areas

patterns in performance over time

summary of methods as connected to evaluation questions

Table 3.1. (J: Communication note: Put links to summary tables (or the tables themselves) at the BEGINNING of documents)

= Reporting = ("June 5, 2004")

reporting section of our evaluation plan

periodic brief reports on the evaluation’s progress first in writing and then in person, including time for questions, comments, and dis- cussion

We believe that ongoing engagement in the evaluative issues being pursued, especially by your staff and interested parents, can enhance the potential power of the evaluation to be a meaningful and valuable learning activity.

for this first year of the evaluation, can we reserve one of your faculty meetings for evaluation reporting and discussion in mid- November, one in early March, and then one in late May?

thought that it could be useful to have a joint pub- lic forum for all interested members of the Bunche–Da Vinci community to discuss with one another the emerging findings and continuing issues being engaged in the evaluation.

we

\LOGISTICS\ We could organize a pasta and salad dinner for the occasion, and we would have interpreters there for the Spanish-speaking parents /LOGISTICS/

every three to four months, the evaluation team will present written progress reports to

leadership of the Bunche–Da Vinci Learning Academy (principal and lead teachers), the district leadership (superintendent and identified others), interested parent and community groups, and designated representatives of the Da Vinci Learning Corporation.

outline in brief, non- technical form the major evaluation activities accomplished since the last progress report

reports will

activ- ities upcoming

highlights of recent evaluation findings

third, the value-engaged evaluator will seek opportunities for meaningful dialogues with stakeholders and especially opportunities for stakeholders to engage A VALUE-ENGAGED APPROACH 43 44 THEORISTS’ MODELS IN ACTION in dialogue with one another about

Resources permitting, a Spanish version of all progress and annual highlights reports will be prepared for the Spanish- speaking members of the community.

The Evaluation Process

responsive to the issues and concerns of all legiti- mate stakeholder groups

requires an on-site presence and an ear keenly tuned to the multiple and diverse rhythms of lived experiences in this particular community

provide spaces and places for thoughtful, data-informed reflections on practice (J: How do you measure this??)

also fundamentally educative

attend

to the substantive and methodological dimensions of the evalua- tion, but also its relational dimensions

to how the evaluator is present in the context at hand

to evaluation itself as a moral and ethical practice (Schwandt, 2004).

(Schwandt, 2004).

enacted in three principal ways.

ry to establish respectful, reciprocal, partnering relationships with all stakeholders

Respect and reciprocity can be communicated, for example, by assuming that all stakeholders have something valuable to contribute to the evaluation and by listening well to each one (Greene, 2003).

Second, the evaluation pro- cess will be an open and transparent one that invites participation by mul- tiple stakeholders in all phases, but especially at the front end, when the evaluation agenda is being established, and at the back end, when inter- pretations of results and action implications are discussed.

inclusion of all legitimate stakeholder perspectives in the evaluation and anchoring of the evaluative work in the particularities of this context

Less important is actual stakeholder partici- pation in the nitty-gritty activities of data gathering and analysis, although stakeholders would be welcome in these processes as well

in dialogue with one another

Meta-Evaluation

near the end of the two years of this evaluation project, a modest meta-evaluation will be commissioned for this value-engaged evaluation of the Bunche–Da Vinci Learning Academy

Two well-regarded evaluators will be contracted to evaluate this evaluation.

evaluator who advocates responsiveness and learning in evaluation practice

evaluator with a different evaluative philosophy and practical approach

evaluate this evaluation on the basis of

1) criteria intrinsic to this approach, (2) the widely accepted evaluation standards (utility, feasibility, propriety, accuracy), and (3) her or his own ideas about what constitutes “good” eval- uation practice