Measuring Impact: The First Step | Beth’s Blog

Measuring Impact: The First Step

Measurement

“You Know Your Film is Making a Difference When…” Guest Post by Ellen Schneider

Note from Beth: How often have those of worked in nonprofits asked the question or been asked the question by funders, “What’s the impact?” And that’s a difficult question to answer – whether it is your social or network strategy or a work of art. The first step is identify what impact might look like as this post explains.

 

 

 

“I cried after my first big film was released,” she admitted, in hushed tones over the phone.  This was no traditional case of post-partum blues. This was an award-winning filmmaker, who had spent years of her life, thousands of her own dollars, and probably a relationship or two to bring her poignant, hybrid, social issue documentary to the light of day (or in her case, a night on PBS following a limited theatrical release). “I wanted to change the world with that film. But even though the critics liked it, and people saw it, I had no proof that anything changed. Silence. I still don’t know if it had any impact at all.”

I had phoned my friend (let’s call her Maya) because I wanted to see how the indie crowd at the Sundance Film Festival would respond to my question: “You know your film is making a difference when…” I first attended Sundance in the late-80s when I was a script reader in Hollywood, and participated during the decade I was at the PBS series POV. But this year in my current role leading the Active Voice Lab for Story & Strategy I did more listening than screening. As a lifelong believer in the power of story — and other creative work – to advance social change, these days I’m spending a lot of time trying to prove it.

I can admit it now: early in my career when I wrote the “Evaluation” part of grant proposals I would wince, concoct a few bullets about the kind of change I wanted to see, and keep my fingers crossed that the funder would be so dazzled by what we were actually able to accomplish programmatically that they wouldn’t feel compelled to go back and reconcile it with our originally-stated objectives. During the first 10 years at Active Voice we commissioned outside firms and social scientists to give us qualitative feedback about our process and our partners’ satisfaction. We weren’t trying to impress funders; we simply couldn’t afford to repeat mistakes. But it wasn’t until Active Voice brought on a full time evaluator that were we able to identify and actually measure the kind of shifts we think films can contribute to.

Maya couldn’t have known if her film had made a difference because there wasn’t a solid strategy in place to connect the story strategically to the movement she was hoping to serve. There were no indicators she could monitor to find out if the film was contributing to specific outcomes; so she couldn’t have made mid-course corrections in distribution, communications, and partnerships that would have led to more “impact.” This is now the core work of Active Voice, designing outcome-oriented strategies and implementing story-fueled, measurable campaigns.

No filmmakers actually cried when I circulated my “making a difference” question at Sundance soon after speaking with Maya, but boy, did I get an earful of responses.  Predictably, some filmmakers reject the concept altogether because they draw a sharp line between “artist” and “advocate;” or as one said, “I tell stories;  what people take away is up to them”  But many talented filmmakers, like Maya, also define themselves as change agents. They are fully dedicated to their craft and want their artistic contributions to be authentic and tangible. They know that their vision and talent can help fuel social transformation in unique ways, and they would go to great lengths to try to make that happen if they had the right kind of map and markers along the way.

But they don’t necessarily think of this as evaluation. Like it or not, evaluation is a crusty kind of term. It has the ring of judgment – which does little for creativity.  If you ask a filmmaker about evaluation, they will often grimace, as visions of foundation bureaucracy float into view. Sometimes our language, traditions, and yes, those pesky power dynamics keep us apart. But if you ask a filmmaker, “How do you know if your film is making a difference?” the conversation will likely become purposeful: What can be achieved? Who are we trying to reach? What resources will be needed? Why this story, now? To me, these sound like the questions that all good evaluation starts with. In other words, when it comes to measuring the impact of arts and culture, we may be surprised by how much funders, creatives, advocates, and communities have in common.

Maya’s film may have indeed had impact. Who knows how many dinnertime conversations followed, and how many attitudes shifted? Maybe organizations got their hands on the film and used it to raise money, visibility, and awareness? Perhaps policymakers took a second look at piece of legislation based on having seen her story.  From my point of view, the connection between “arts and culture” and engaging people in solving urgent problems is too potent to guess about.

Ellen is the former executive producer of the PBS series, POV, as well as the founder the Television Race Initiative and Active Voice. She is a pioneer in using film to creatively tackle social issues.

12 Responses

  1. Charlene says:

    Hello Beth,

    You have a great blog!

    Our motto at ipeopletise is: Invest in your Community and they will invest in your Business; Invest in your Businesses and they will invest in your Community

    If you know of any groups that would like to get Free, group custom t-shirts (Sponsor Tees) for any of their upcoming events please spread the word and have them check out http://www.ipeopletise.com. Once an event is posted businesses from all over will get the chance to sponsor their t-shirts while learning about their cause at the same time = )

    Thank you for your time

  2. Really enjoyed reading this. Yes how do you measure social change? I have been working with a couple of social campaign groups working to prevent family violence, attitudes to family violence and other campaigns involving changes to how the public perceive issues.

    When there are so many factors involved in changes to attitude how do you map these back to a specific campaign?

    I guess creative types don’t like evaluation because they view film as a subjective personal, experience. But of course documentaries about issues are created on a need to challenge society and its perceptions.

  3. In response to Daniel’s question “How do you measure social change?”

    I think Ellen made an excellent point in this regard when she identified that no solid strategy was in place.

    Too often organizations execute campaigns/programs without having a strategy in place *before* they begin. Unfortunately they then accept the misconception that “well you can’t really measure social change.” and continue to execute repeatedly without ever trying. This is one of the reasons why so many orgs struggle with demonstrating impact.

    I work to instill in my clients a culture of continuous improvement based on a cycle of discovery, strategy, execution and analysis. They find that when they’ve identified ahead of time the things they wish to affect and how, and then build a strategy to accomplish that goal, they have a yardstick against which to measure their progress, and regardless of whether they were successful or not, then have valuable data on which to build future campaigns.

    It’s not easy, but it is possible and rewarding. Ellen’s post includes a great example of this process.

  4. I have a public affairs series on issues that affect pedestrians. It runs on public access cable, where there aren’t even any viewer numbers. In recent years it has also been on the Internet, where I have decent analytics, but still cannot measure how it has changed opinions or influenced actions. I also have analytics for my website, with similar limitations.

    When gauging impact, I rely almost entirely on personal feedback from viewers. So I take heart when someone tells me it contributed to the conversation about walkability in their community, or it was a factor in getting funding for new sidewalks near their school. And I assume I made a difference when someone tells me that their planning director watches it, or an advocate in another country tells me that they get ideas and inspiration from watching it. But those stories don’t give me any hard numbers.

    I’m not worried about compromising artistic integrity in a quest to have more impact. I’m just not sure how I would measure impact even with a lot of resources, for a program that is produced with extremely limited resources.

  5. Beth says:

    HI Lance,

    Have you written anything about your methodology? Would love to read more.

    Beth

  6. Lance says:

    Hi Nancy,

    I’ve written tons on it in bits and pieces for individual clients and the occasional proposal. The only formal detailed write-up is an early draft of an ebook that’s currently in the works.

    Contact me directly via email and I’ll send you a copy when it’s finished!

    Best,
    Lance

  7. Beth says:

    Lance, I will do that and perhaps you might like to write a guest post.

  8. Alec Leggat says:

    I suppose critics are the M&E specialists of broadcast media. That sounds a bit flip but when your media isn’t specifically targeted and you don’t have objectives for change in mind then it is really difficult to claim any cause and effect relationship. If you can measure a behaviour change as a result of your media increasing knowledge and changing attitude then you can argue that your work has an impact but if you’re just putting it out there then that’s great but don’t expect to be as to claim impact.

  9. Lance says:

    Beth, that would be great! Let me know the details and I can bounce a couple of ideas off of you. Thanks!

  10. Chari Smith says:

    I think the way we do evaluation is changing, and with it more acceptance. Lance, I appreciate your approach, as mine is similar. It’s so important to build a culture of using data within organizations. So often, it’s a culture of compliance – they collect the data because they feel they have to, not because they want to.

    A shift in mindset is critical, to wanting to collect data, to understand impact, what’s working, and what’s not.

    Beth, I agree, evaluation is still a ‘crusty’ term. I think that can be moved, through advocating building a culture of evaluation. Where organizations truly want to understand – are we making a difference? In my opinion, that’s exactly what doing an authentic evaluation answers.

  11. kuldeep says:

    Hi Lance,
    Can you share some of your ideas or your methodology on how to measure impact on social change, love to know more for proposals or individuals/communities

    Kuldeep

  12. Beth says:

    Lance, I’ll follow up with you

Leave a Reply