For an enhanced digital experience, read this story in the ezine.
In recent months, the coronavirus (COVID-19) pandemic has laid bare the impact from decades of disinvestment in communities of color and low- income communities. Environmental justice researchers and advocates have long emphasized the need for equitable distribution of quality parks and programming to reduce adverse health effects and improve quality of life throughout our society. For many others, it took the collision of a global pandemic with the Black Lives Matter movement to truly understand why.
Evaluation provides park and recreation agencies with a powerful set of tools to understand how they serve members of their community, where they fall short, and how they can fill service gaps that have left some community members behind. For agencies without dedicated staff on hand to assist with program evaluation, it may be tempting to begin and end program evaluation with customer service surveys conducted at program events and in community centers. Extrapolating results to infer about the needs of the wider community, in this case, is problematic when you consider groups that may not be represented in the survey sample. When agencies draw conclusions about the wider community based on surveys conducted at agency events, results may be biased. Bias can unintentionally justify current levels of service and may exacerbate community inequities. In other words, an agency may not be asking the right questions of the right people, and this has consequences for the trustworthiness of the evaluation.
Relying on a single method for evaluation — such as customer satisfaction surveys — limits the types of questions agencies can ask, the ways that residents can express themselves, and the individuals who are included in the evaluation in the first place. To mitigate this threat, dedicated evaluators often work with complementary types of data, including qualitative, quantitative and spatial, which is frequently elicited through the use of mixed methods.
Qualitative vs. Quantitative Data
Qualitative data is subjective information conveyed through words and phrases. It can be gathered through surveys with questions that ask anything, such as someone’s preferences or level of satisfaction. When qualitative data is acquired through interviews, agencies can explore questions of “how” and “why” with key stakeholders, including those who do not participate in programming. Qualitative data gathered through interviews also is useful to guide the elaboration of valid survey instruments by ensuring that agencies ask relevant questions and capture important demographic information.
In contrast, quantitative data is information that comprises objective numbers and characteristics that can be measured. For example, evaluators might want to find out how many days a week someone uses a park, or how much more someone would be willing to pay in taxes for additional services. Other types of data include spatial and temporal information, which can tell agencies how resources or service gaps are distributed across space and time.
By building a mixed-method evaluation that relies on a combination of data types, evaluators also can ensure that they sample from, or speak with, many different stakeholders across the community. This includes people whose responses fall outside of the normal range — such as responses that are discounted as “outliers” that may be excluded from an analysis focused on averages. The truth is that evaluators can learn just as much from a handful of people who are considered outliers — often the most vulnerable or the most resilient members of a community — as they can from those who tell us what is happening on average.
The Art and Science of Equity
Evaluation for equity is as much art as it is science. While evaluation requires a basic understanding of the principles of study design, sampling and analysis, it also is an art with skills gained through practice employing a suite of complementary mixed methods. It may help to imagine that each method used in an evaluation is a color of paint on a canvas. An artist who paints with many colors creates a more realistic image. It is impossible to paint an image that replicates real life, but when the goal is to evaluate for equity, agencies should aim to paint an accurate and colorful picture of what is happening in their community. Agencies that succeed not only elevate the quality of life of all community members, but also are better able to justify to constituents and decision-makers alike the value they bring to their communities.
Lauren Redmore, Ph.D., is NRPA’s Evaluation Manager (lredmore@nrpa.org).