Instructions

Set Up: Prepare for the Activity

Share the Collaborative Performance Measurement Worksheet as handouts or create copies of the file for each group to use.

Organize participants into small groups (4-6 ppl).

Begin by introducing the learning goals of this activity.

Step One: Introduce Three Levels of Performance Measurement (10 min)

Ask participants to reflect back on the algebra course they took in high school or college. Have them take the role of the teacher in that class trying to assess students' performance. Which of the measures listed below would be best for measuring performance:

• The number of assignments students were asked to do.

• The grade students received on a standardized test.

• The students’ ability to use core concepts later in life.

Then explain the three levels of measurement:

• Input - These measures represent the effort put into achieving a result.

• Output - These measures represent the actual result that was achieved.

• Outcome - These measures represent what happened after the output was achieved.

Ask participants to rank the importance of these measures. Which measure is the most desirable? Why? For example:

• Input measures tend to be the easiest to identify but these are not indicative of performance itself but just effort expended. They are however commonly used.

• Output measures are often reflective of surrogates of performance. They are often invented to give some sense of credibility. They are often commonly used.

• Outcome measures can be challenging because it takes time to assess whether the initial results have a lasting impact. These measures are the most desirable, but hard to develop.

Step Two: Introduce Scenario (5 min)

In small groups, ask participants to review the scenario on the worksheet.

Scenario: Artificial Intelligence Ethics Training

An organization is trying to decide if they want to invest in an Artificial Intelligence Ethics Training for their employees. The training is expensive but could be valuable for their workforce. In order to decide whether to make this investment, the organization needs to decide how the effectiveness of the training will be measured.

Share the following prompts:

• As a group, write a statement that describes what you think the training should achieve. What do you want trainees to be able to do or demonstrate after completing the training?

• List the specific outcomes related to this overarching goal for the training on the worksheet.

Step Three: Develop Measures for Outcomes (10 min)

Next, ask each group to discuss and record on the worksheet:

• How do you plan to measure the desired outcomes?

• What measurement challenges do you anticipate?

• What factors are likely to affect your measurements?

• How might your measurements be misleading?

Step Four: Develop Measures for Inputs and Outputs (10 min)

Ask each group to make a list of potential inputs and outputs of the proposed training program.

• How would you measure inputs and outputs of the training program?

• What are the strengths and weaknesses of each measure?

• How do these measures relate to our desired outcome?

Step Five: Arrive at Short-Term and Long-Term Measures (10 min)

Ask participants to think about the input and output measures and how they can be made as useful as possible in the short term. Challenge them to think of how these measures can be used in linked combinations (e.g., number of participants who attended the training and then passed a qualifying test).

Next, ask participants to think about what can be done now to capture the information for eventual use in outcome measures.

Step Six: Debrief as a Full Group (10 min)

• Why is it important to develop performance measures collaboratively?

• When we evaluate outcomes, what matters most? Numbers? Stories? What seems more credible? Why?

• If you were to use your understanding of input, output, and outcome measures, how would you answer the question: How are you better because of the CDP certificate program?