Documentary Impact Evaluation

One model for measuring the impact of documentaries is presented in a 2014 paper by Diesner, Pak, Kim, Soltani, & Aleyasen

Elements in the Model

• Dimensions: Content, Medium, Target, Impact,
  • Level: multiple levels exist within each dimension, Diesner et. al defines this as a "set of sub-categories of evaluation criteria"
    • Evaluation Factors (what Diesner et. al call "Index") the set of evaluation factors in a given level.
    • Analytics = analytic methods that can be used for each evaluation factor
• Items: sets of "specific features to be measured" within each evaluation factor.

Dimension 1: Content

Levels:

1. Message = This can be identified by interviews with the film-maker, document analysis of governing and planning documents for the film, and analysis of the content of the film itself. Diesner et. al define a message as "the main message that a film wants to convey." Reflection questions: 1) Do you think that this definition is necessary and sufficient for the "Message" of a documentary? 2) What would you add or subtract from this definition? 3) Why?
2. Expected Outcome = These can be found in the grant application, through interviews with the film-makers, etc.
3. Evaluation Priority = Create a list of priorities based on the outcomes identified in the step above, and rank them in order of importance.
4. Resource = The resources needed to produce the film. This level is necessary for any evaluation of cost-effectiveness.

Evaluation Factors:

1. "Guiding Factor"

Analytics:

1. Description
2. Ranking
3. Weighing

Items:

1. Reports by producers (Diesner et al, 2014, Table 2, p.6)
2. Reports by funding agencies (Diesner et al, 2014, Table 2, p.6)

Dimension 4: Impact

Levels:

Level 1

1. Individual
2. Communal
3. Societal
4. Global

Diesner et al identify four categories of measurement that For each of these levels:

1. Cognitive <- Evaluation factor "Awareness", analytical methods: "Statistics, Text mining, web analytics, network analysis", as well as surveys
2. Attitudinal <- Evaluation Factor "Sentiment", analytical method: "Sentiment Analysis"
3. Behavioral <- analytical methods: text mining, web analytics, network analysis, as well as observational protocols.
   1. Engagement <-
   2. Enactment
   3. Connectedness
   4. Capacity
   5. Expansiveness
   6. Centralization
4. Temporal <- Evaluation factor: "Impact Dynamics", analytical method: "Longitudinal analysis"

Diesner et all call these the "four stimulous dimensions" (p. 8)

Analytics:

1. Text Mining (mentioned twice, for Behavioral and Cognitive levels)
2. Web Analytics (mentioned twice, for Behavioral and Cognitive levels)
3. Network Analysis (mentioned twice, for Behavioral and Cognitive levels)
4. Longitudinal analysis (mentioned once, for Impact Dynamics)
5. Sentiment Analysis (mentioned once, for Attitudinal level)

Applicable to Temporal categories:

1. Longitudinal Analysis