Intended audience: DataKind Volunteers
Qualitatively evaluating the project is all about understanding the context and gathering supplemental information about impact that isn’t captured quantitatively. This can be done a number of ways, so the guidelines are flexible to fit with your specific project’s needs, although some form of interview is the most common. Here are some best practices when conducting an evaluation interview:
- It’s best having two teammates attend the call, one who can facilitate the conversation and another who records and annotates what is discussed. Having a designated note-taker allows the facilitator to focus on drawing out insights, rather than taking notes.
- If you choose to record the call to ensure you accurately capture feedback, make sure to get the project champion’s consent to record and create an environment where they feel comfortable being honest about any challenges. Inform partners at the start of the call that the recording would be only used internally for report creation, and their responses will be anonymized to prevent identification.
- Use information from the Post-Project Surveys to prepare questions. For example, “In the survey, you mentioned that _____. Can you tell us a little more about that?” This will prevent redundancy while giving more clarity and insight to our partners’ thoughts.
- Try to be concise in order to respect our partners’ time. If some responses are long, try to target the most impactful questions/areas in order to get the most out of our partners. Remember that sometimes they may answer several questions in one response, so you may be able to cut some questions as you go.
One suggested format for the qualitative interview is an unstructured group interview, or structured focus group , with end users and people impacted by the product if deemed appropriate after discussing with the partner organization. But if you worked with a very small organization, you might just meet one-on-one with the Project Champion. At a very large organization where you have access to several people impacted by the product as well as multiple individuals maintaining the product, it might be valuable to split it up into two conversations for the different types of users. Structure the qualitative evaluation to fit the unique circumstances of your project.
The following sections provide different topics and types of content to plan to cover in your qualitative evaluation.
Usefulness
First, identify if the product is still being used and/or was successful in achieving its intended purpose. If the product is not in use, it is still prudent to ask about any uses or outcomes that may have come from the project, as many organizations still see benefits even when the intended purpose has not been realized. Ask the following questions from the Share Stage in the interview again, to see if the answers have changed: “How understandable do you feel the solution to be? For example, what can you tell me about how easy or hard it is to train others on the solution or explain to multiple audiences?”.
Behavior Change
Dig into if and how the organization changed from our work. The projects we do have the potential to influence organizational decision-making, operations, hiring practices, priorities, and other aspects of our partner organizations. Questions in this area can include:
- Were there any efficiency improvements from the project? If so, what were they?
- If applicable, how did this project change the way decisions are made at your organization?
- Are you engaging in any other new data science projects? If so, please explain the projects and any way in which they connect to the DataKind project.
- Have you hired any data or technical staff? If so, for what roles and why?
- How are you incorporating the lessons learned from this project into the collection of new data?
- Have you expanded on the project in any way? If so, how?
- Have you saved any financial resources or time from the project? If so, how much?
- Have you had any funding or branding improvements based on the learnings or results? If so, how much?
Scale
Ascertain whether the solution / learnings have scaled or could scale. Ask whether other organizations have used the results or findings. Listen for cases where the organization knows the work has been used across the sector. Other examples include situations where the organization has written about or talked about the work publicly. Sometimes, our partners are very cognizant of the opportunities and challenges associated with scaling the solution, and these are good to look out for. Now that they have been using the results for some time, re-ask the question from the Share Stage interview: “Can you envision this solution being used elsewhere in the [sector name] sector? If so, how? If not, why?” to see if the answer has changed.
Support
Understand what other data science support the organization may need. Is there something they need done quickly to better use the product from this project? What is it? How could this project be taken into a follow-up phase to expand on the product? Additionally, have them walk you through any unrelated needs for data science support in a different part of the organization.
Data Maturity
After a DataKind project is executed, it is highly recommended to assess maturity again to understand if the partner organization’s data maturity has grown. Our projects in the past have given organizations their foundation in data science and data literacy, influenced shifts to more efficient data collection, and even changed hiring practices, so the data maturation of our partners can look very different depending on the specific organization and their data maturity level before the project. Note that there is no need to do the whole assessment again if it is burdensome. Rather, this evaluation is an opportunity to ask about potential areas of growth. Prior to the interview, review the results from their original data maturity assessment and flag areas that were weak so that you can ask specifically about those to see if there’s been any change. Questions to address data maturity can include:
- In what ways have the staff at the organization increased in data maturity as a result of the project?
- Do you feel more comfortable with data? Explain how and in what ways, if so.
- Have any of your staff learned new tech or data skills? If so, walk me through how and why.
Open Data Outcomes
If the data has been shared openly already, ask about the potential impact that this shared data might have had on the broader community or other ecosystem actors. Consider reaching out to contacts in the industry to evaluate this or gather information on engagement with the data.
Contributer(s): Benjamin Kinsella, Caroline Charrow, Manojit Nandi, Mallory Sheff
Contact us
If you would like to learn more about us, partner with us, or get in touch, email us at community@datakind.org
Subscribe