Intended audience: DataKind Volunteers
While the evaluation plan should have already been discussed during the design stage, during hand-off it’s essential to confirm the details of the evaluation plan with the Project Champion so they are prepared to participate as needed in the project evaluation. This step can most easily be written out in a brief email, and should answer the following questions:
When will the evaluation take place?
Evaluation typically takes place about six months after project completion, but this can be adjusted based on the expected rollout timeline for the project and how long it will take for the project impact to be seen. Decide on the exact dates and set up a calendar invitation now for a time to conduct the evaluation.
Who will be involved in the evaluation?
Who else from the project partner organization does the Project Champion need to ensure is prepared to be involved? For example, will someone else be doing data collection? Who from DataKind is committed to doing the evaluation?
What will the evaluation consist of?
The Project Champion and DataKind team should align on expectations for the evaluation and what it will include. Evaluations at DataKind ideally include the following components, which are a great way to explain expectations to the project champion:
- Quantitative Evaluation: We collect quantitative data based on the measurement plan that was created during the Design Stage. This measurement plan was drafted based on how the partner organization indicated that they are measuring their impact already, in order to measure project success against intended impact for the partner organization within their existing impact measurement systems. This involves coordinating with the team collecting the impact data in accordance with the measurement plan already created, as the Project Champion will simply share the results with DataKind.
- Qualitative Evaluation: We survey the Project Champion, end users, and/or communities impacted by the product to understand qualitatively the results that came from the project by collecting data on implementation, retention, and impact. This enables us to understand how successful our project was in a more nuanced way.
- Additional Support: We identify if the organization needs additional support to continue to move the project and their work forward with data science. We offer to support if DataKind has capacity, or help point the project partner organization in the right direction if DataKind is not the right fit to provide the needed support.
- Ethical Evaluation: Additionally, we conduct ethical evaluation of how the project output is being used, carefully reconsidering the ethical implications of the project and looking at the product from the point of view of the communities impacted. For example, this step might include a data science fairness evaluation of whether the model is performing in a way that disadvantages certain groups, now that it is being implemented.
- Evaluation Report: We consolidate our findings in an evaluation report, used for internal reflection on the project at DataKind and for the partner organization to use to report on the project results internally.
- Data Publication or Destruction: We create and implement a data publication or destruction plan to ensure that all data security protocols are carefully followed and no data remains in DataKind’s hands without appropriate permissions. Note that any decisions on whether data will be published should have already been made at the point of data sharing.
- Technical Project Close: Once we have confirmed that the project will not be continued or supported additionally by anyone at DataKind, we close down DataKind team cloud environments. If additional support is being provided, the cloud environments are revisited after that support ends.
- Share Evaluation Learnings: We share evaluation learnings internally with the DataKind team. We also consider possibly presenting results again or in a new external format.
Contributer(s): Caitlin Augustin, Mitali Ayyangar, Caroline Charrow, Mallory Sheff, Rachel Wells
Contact us
If you would like to learn more about us, partner with us, or get in touch, visit our website or email community@datakind.org.
Subscribe