Content Prioritization for Standards-based Education

In contrast to the other methods of content prioritization presented here, some designers are not given latitude to disregard or eliminate content. Most often this is because the content is directly related to externally or internally imposed standards, such as OSHA-compliant safety awareness programs, or state and district academic standards required of primary and secondary schools.

Schools in particular are required to achieve a considerable number of standards, and the general consensus among administrators and teachers alike is that there are too many standards to cover uniformly and fully. Noted education consultant Dr. Robert Marzano notes that in order to cover all standards “You would have to change K-12 to K-22. The sheer number of standards is the biggest impediment to implementing standards.”

Yet state and local standards must be met. Content prioritization is essential in this case as well, though rather than discarding content altogether, the prioritization is of time and energy devoted to accomplishing each standard.

There are two main difficulties users will encounter when attempting to execute instructional design using prioritized standards:

Evaluating standards against the specific criteria of your school or organization.
Achieving consensus among the stakeholders regarding high, medium, and low priority standards.

Thus, the method needed will need to be adaptable and democratic. An adaptable nature will allow it to be tailored to the organization’s unique situation, and being democratic will ensure that the results are well rounded and - just as importantly – fair, meaning that teachers, trainers, and/or designers are more likely to buy into the decisions made based on the consensus.

About this Method

The method that you will learn about in this section has two stages:

Develop additional criteria using Nominal Group Technique.
Prioritize standards using a Decision Matrix.

The method detailed below uses an example that takes place in a K-12 setting, but it can be used for other organizations that use externally-implemented standards and/or high-stakes standardized testing.

After reading the section about the Nominal Group Technique, you will understand how it allows decision-makers to develop criteria that are important and specific to their institution and how the group ranking process promotes fairness in the adoption of the criteria in the decision-making process.

After reading the section on processing the resultant data with a Decision Matrix, you will understand how the criteria are used to prioritize the standards.

Nominal Group Planning

Nominal Group Planning allows participants to anonymously submit their ideas and rank the group results. Group discussion allows participants to debate the pros and cons for each criterion, and then anonymously revise their rankings.

Advantages

Allows for participants to submit ideas anonymously, without fear of reprisal.

Encourages group discussion and debate.

Group collaboration encourages buy-in from all participants.

Disadvantages

Process takes a significant amount of time.

Consolidating ideas can prove tricky.

Timid participants may find themselves overruled during group discussion.

Steps for Conducting a Nominal Group Planning Session

Gather all decision makers into a group. If more than ten people are present, it may be advisable to break into subgroups, as participant attention and enthusiasm will wane if the proceedings take too long.
The facilitator (chosen beforehand) explains the purpose and process of prioritization to all participants. This should include a clear statement of the objective of the meeting.
Next, each member of the group should write down criteria which they feel will be useful in evaluating the standards. This should be strictly individual, as to preserve the anonymity of the author at a later stage. This is the most unstructured step in the whole prioritization process, and because it is done individually and early in the process, it is likely that the facilitator will need to supply plenty of encouragement for participants who are still uncertain in their purpose. It is likely that a few criteria will already be established externally, or may be particularly obvious, so the facilitator may want to use those criteria as an example.
The facilitator will take all lists of suggestions and compile them into a master list that will be manipulatable by the facilitator and visible to all participants (such as a dry erase board or transparency on an overhead projector). The master list should not have grouped entries from the same individual list, but should be random. If the number of decision makers and/or the number of criteria offered is substantial, offering the group a break will give the facilitator the time to compile the large list without boring the participants.
Through group discussion, the entries in the large list should be consolidated and simplified. Undoubtedly some specific criteria will be subsumed in more general criteria, but the facilitator should warn against over-consolidating into a few criteria that are too general or broad. For instance, ‘opportunity cost’ and ‘pecuniary cost’ should not be rolled into a general ‘cost’ criteria, as each is a very different dimension, and ‘cost’ will mean very different things to different people.
Having reduced the master list into a consolidated list, each participant should individually (again, to preserve anonymity) compile a ranked list of criteria, where #1 is what they consider the most important criteria of prioritization. It should not be necessary for participants to rank more than ten of the criteria if the total exceeds that number. The facilitator should remind the group that they will have the opportunity to re-rank their decisions at a later stage, so it is not imperative to take a lot of time and consideration when ranking their lists.
The facilitator will tally the results, one individual list at a time. One point is assigned to the top ranked criteria on each list, 2 points to the second, and so forth. The individual scores of each criterion are summed, and the criteria are ranked according to the group scores. For example, if the group consisted of three participants with three criteria, the individual lists might look like this:

The ranked tally would then look like this:

Criteria X: 1 + 1 + 2 = 4
Criteria Y: 2 + 3 + 1 = 6
Criteria Z: 3 + 2 + 3 = 8

According to the group, criteria X is the most important, Criteria Z the least, and Y in between.

Next, discuss the results as a group. Participants are encouraged to voice concerns about why a high-ranked criteria should not be held in such positive regard, or why a low-ranked criteria should be elevated. The facilitator should encourage discussion among all participants, not just a vocal few.
The penultimate step in the process is the same as step 6, only this time participants should be given more time to rank their lists.
Facilitator once again tallies and ranks the results. The top criteria are selected for use in the Decision Matrix. The number of criteria selected depend on the situation and if some criteria were established previously to the session. However, if there is a natural break in the ranking data (such as a 50% difference in the score of adjacently ranked criterion) it generally indicates a good cutoff point.

Narrative Walkthrough for Conducting a Nominal Group Planning Session

Context: Teachers in a large California school district have faced considerable difficulty in covering the ever-increasing number of state standards during the academic year, and the issue has been a source of friction between the educators and the demanding district school board. The board has responded by enacting a program of standards prioritization that will aid the teachers in refining their curriculum to be functional and to improve concurrency of content taught in each classroom. This is where you come in. Let’s say that you have been charged with acting as a facilitator for a group of five sixth grade social studies teachers who are prioritizing their content standards. You and the teachers have received a “prioritization packet” created under supervision of the district board that explains the general idea of prioritization and a few criteria that should be used when evaluating the standards. During a summer time Teachers’ Conference meeting your job is to guide the teachers through a Nominal Group Planning brainstorming session with the goal of discovering other criteria that can be used to prioritize the standards.

Beforehand: You reserve a conference room that is quiet and largely free from distractions. You make sure that there is adequate seating for you and the five teachers who will be participating in the process. Knowing that the process requires each participant to write, you place a stack of lined paper in the middle of the conference table along with more pencils than will be needed. In order to indicate their seats, you place an itinerary (complete with the steps used in the Nominal Group Planning session) in front of each participant’s chair. You are also aware that a group display is needed, and you have decided to use the conference room’s whiteboard. Erasers are accounted for and the markers are in working order, so you are confident that the meeting is ready to proceed.

Step 1: The participants arrive and are seated. Because there are relatively few participants, only a single group is required.

Step 2: Although the teachers are aware of the purpose of this meeting, it is still a good idea to have a discussion. You ask the teachers to volunteer reasons why prioritization will be valuable to their practice as well as any concerns they may have. The teachers respond immediately and are very vehement in their frustrations in covering all standards, and commiserate with one another over their mutual hardships. After a few minutes you sense that the discussion is slipping too far into negativity and attempt to turn the conversation constructive by asking the teachers how they think prioritization will help them to cover all standards. One response is “We will finally be able to downplay the topics that aren’t of much use to our students.” Another teacher adds “We will all be on the same page when teaching, rather than some classrooms concentrating on one topic for a long time when the others only spend a few days on it.” The other teachers voice their agreement, and you thank each for invoking the two central goals of the meeting. You ask the teachers to help you formulate a goal statement for this session, and after brief discussion it is decided. You write ‘What factors should we, acting and teaching as a group, consider when deciding what content students will need to succeed in school and life?’ at the top of the dry erase board. You ask the participants to keep this in mind during the session, and take a few minutes to review the steps of the process with the teachers, using the model on the itinerary as an aid.

Step 3: You ask everyone to take a piece of paper and a pencil for the next step. Even though you have just explained the process, you review the actions that need to be taken during this step. Keeping in mind that the teachers may require some additional guidance due to their relative inexperience using this decision making method, you use some of the criteria already established by the district board as an example. You write ‘Readiness for School’ on the whiteboard and, as if thinking aloud to yourself, explain that this is an important criterion to consider because the lessons taught today become the foundation of lessons taught later on. You write ‘State Test’ and think aloud about how students (and teachers) must cover some material if only because it has a significant presence in standardized testing. Finally you write “Usefulness in Life” and think aloud how some topics may have a small presence on tests or later on in school, but have substantial significance or use in one’s life, such as learning basic economic principles. You ask the participants to come up with at least four or five other criteria that they think should be considered during prioritization, reminding them to keep their list to themselves.

Step 4: After several quiet minutes you ask the participants to finish up, and then collect each list from each person yourself. You explain that you will combine these lists into a consolidated list of all suggestions and that the participants may take a short break while you accomplish this.

The five sheets you obtain read as below:

Participant 1	Participant 2	Participant 3
Student interest School’s past performance Time/effort needed to teach adequately Cost of ancillary materials Teacher interest Alignment with school’s mission	Is it easy to teach? Student weaknesses What materials are available? Are students interested?	Available internet resources District’s past performance Greatest room to improve on CAT Cohesiveness with other standards Interest level
Participant 4	Participant 5
School needs Interactivity Time	What do we need to raise social studies scores? What can we teach easily? How familiar are students with the topic already? Do we have any (more exciting) resources available besides the textbook?

You realize that the anonymity afforded by working individually will be worthless if you just wrote each list in total, one after another, on the board. Instead, you randomize the responses as best you can, crossing out each response as you write it on the board. The result looks like the below:

What can we teach easily?
Are students interested?
Cost of ancillary materials
Student weaknesses
District’s past performance
How familiar are students with the topic already?
Cohesiveness with other standards
Interactivity
Greatest room to improve on CAT
Do we have any (more exciting) resources available besides the textbook?
Is it easy to teach?
Interest level
Time/effort needed to teach adequately
School needs
Student interest
School’s past performance
Available internet resources
What materials are available?
What do we need to raise social studies scores?
Time
Teacher interest
Alignment with school’s mission

Step 5: Once the teachers have returned from their break, you remind them that in this step of the process group discussion will be used to combine criteria that are essentially the same idea. The first few are immediately called out and unanimously agreed upon. ‘What can we teach easily?’ and ‘Is it easy to teach’ are combined, as are ‘are students interested?’ and ‘student interest.’ Mindful of the difficulties that can arise later if the criteria are too general, you take the opportunity to ask the group exactly what “easy to teach” means. They are silent for a moment until one teacher responds, “Well, to me it means that I don’t have to fight with the kids about it. Something they like and will work at without me standing over them every second.” Another volunteers, “To me, easy means that I don’t have to do a lot of prep work. If it’s a pre-planned lesson that I can tweak on the fly, that makes my job much easier.” You point out that the criteria as it is currently phrased means different things to different people. The first teacher points out, “True, and what each of us got out of it is represented by criteria that are also already on the board – ‘student interest’ and ‘what materials are available.’” After some more discussion, the group elects to remove ‘easy to teach’ from the board, as it is too general, and can be represented by several other criteria listed.

After further consideration, consolidation, combination and elimination, the group agrees on the following consolidated list of criteria.

Availability of ancillary materials
School’s past performance
Cohesiveness with other standards
Teacher’s interest in topic
Students’ interest in topic
Time needed to teach thoroughly
Alignment with school’s mission
Students’ previous knowledge

Step 6: You ask the teachers to once again take a blank piece of paper. You ask them to rank the consolidated list according to what they believe are the most important criteria for prioritizing state standards. Remind them that they should keep their responses confidential and that they shouldn’t split hairs over their ranking, as they will have opportunity to change their minds later.

Step 7: After a few minutes you ask the participants to finish their ranking and gather the papers as before. You offer them another break as you pore over the results, which are below:

Participant 1	Participant 2	Participant 3
School’s past performance Students’ interest in topic Availability of ancillary materials Time needed to teach thoroughly Teacher’s interest in topic Students’ previous knowledge Alignment with school’s mission Cohesiveness with other standards	Students’ interest in topic Availability of ancillary materials School’s past performance Teacher’s interest in topic Time needed to teach thoroughly Students’ previous knowledge Cohesiveness with other standards Alignment with school’s mission	School’s past performance Students’ interest in topic Time needed to teach thoroughly Students’ previous knowledge Availability of ancillary materials Teacher’s interest in topic Alignment with school’s mission Cohesiveness with other standards
Participant 4	Participant 5
Students’ interest in topic Teacher’s interest in topic School’s past performance Time needed to teach thoroughly Students’ previous knowledge Availability of ancillary materials Alignment with school’s mission Cohesiveness with other standards	School’s past performance Students’ interest in topic Students’ previous knowledge Availability of ancillary materials Teacher’s interest in topic Time needed to teach thoroughly Cohesiveness with other standards Alignment with school’s mission

You assign one point to each top ranking criteria, two to the second, and so on. You then add up the points awarded to each criteria. Finally, you arrange the results into a group-ranked master list, which lists the lowest scoring criteria first and the highest last.

How many points were recieved by 'Availability of ancillary materials?'

How many points were recieved by 'School’s past performance?'

How many points were recieved by 'Cohesiveness with other standards?'

How many points were recieved by 'Teacher’s interest in topic?'

How many points were recieved by 'Students’ interest in topic?'

How many points were recieved by 'Time needed to teach thoroughly?'

How many points were recieved by 'Alignment with school’s mission?'

How many points were recieved by 'Students’ previous knowledge?'

Show me the math used in step 7 and the group-ranked list that is the result

Step 8: You transfer the group-ranked master list to the whiteboard for everyone to examine. You open the discussion by asking the teachers if they see a low-ranked criterion that should be higher. “I think the ancillary materials could be higher,” one participant ventures with some hesitation. “If a topic has a lot of quality materials available, then we can give the kids a wider choice in what learning tools work best for them. It would be a good way to reach out to more learning preferences and styles.” Heads nod in agreement and another participant quickly volunteers “I think ‘teacher’s interest’ is way too high. God knows some lessons are dull as dishwater for us, but its not really about us – it should be about the students.” A few other points are made, but after a few minutes the discussion peters out and you take it as your cue to move on.

Step 9: You ask the teachers to take a piece of paper yet again. You explain that they are going to rank the master list one more time, this time for keeps. Assure them that they will have plenty of time to think and consider this time.

Step 10: Once all participants are finished, you collect their papers for the final time and once again sum their scores and arrange the criteria into a final group-ranked master list. When finished, you notice that there is a strong break between the third ranked criterion and the fourth. Realizing that this indicates a strong preference for the first three criteria, you announce that they will be carried over to the standards prioritization session, while the others must be discarded.