Big Data Without Defining Success First Is A Big Mistake | Beth's Blog

Big Data Without Defining Success First Is A Big Mistake


With the release of the “Measuring the Networked Nonprofit:  Using Data to Change the World, I’m looking forward to participating in the conversation about how nonprofits can use measurement, data, and learning to for social change.

We are living in a “Big Data” is a theme I’m seeing discussed this past month in both the for-profit and non-profit sectors.   It refers to large data sets that requires powerful software tools to capture, storage, search, sharing, analysis, visualization and sense-making – or what DJ Patel calls “Data Jutjitsu.”    I love this stuff, but it could become a shiny object or distraction for nonprofits that have yet learned how to “jutjitsu”  the small data sets they are collecting internally.

Here’s a terrific round up of posts about “big data for small nonprofits” from the good folks at Wild Apricot.   But I think jumping into a process:  “Gather, Analyze, and Act”  without defining success (or failure) on the front end might lead to wasted time.

  • Define results
  • Identify research to formulate a hypothesis
  • Identify KPIs
  • Gather data
  • Jutjitsu data
  • Make decisions based on data

How DoSomething.Org Does It

DoSomething Dashboard - Click for larger image has a big hairy social change goal:  To harnesses teenage energy and unleash it on causes teens care about by launching a national campaign.  The call to action is always something that has a real impact and does not require money, an adult, or a car.   Their measurable goal is to get 5 million active teen members engaged in social change campaigns by 2015.    Their use of social media, mobile, and data all strategically selected and use to reach that goal.

A recent example is their “Pregnancy Text” Campaign featured on their quarterly dashboard.    This clever sex education campaign is an updated version of the teen pregnancy education program where young people carried eggs around and pretend they are babies.   It was a text campaign where teens opted in to receive texts on their mobile phones from the “baby.”     Once they joined (and they could share it with their friends). they received regular annoying text messages at all hours from the “baby”  that poops, cries, and needs their immediate attention.

The team at uses data to base the program design, key performance indicators and a hypothesis to be tested.    They looked at  survey data from the National Campaign:  nearly 9 in 10 (87%) young people surveyed also say that it would be much easier for teens to delay sexual activity and avoid teen pregnancy if they were able to have more open, honest conversations about these topics with their parents and/or friends.    So, success of this campaign would be mean that participants talk with their family or friends about the issue and delay sexual activity.

The basic design had those who signed up challenge their friends to take care of a text baby either by (1) going to DoSomething website and selecting 5 friends to challenge or (2) after receiving a text from DoSomething (sent to DoSomething’s 300k mobile subscribers) would opt to challenge friends after reading a quick stat on US teen pregnancy.   Participants that accepted the challenge would then start receiving texts the following morning from the text-baby.  After completing the challenge user were prompted to send it to their own friends. also followed up with 5k of the users with a text-based survey to measure impact.

Once defining success and identifying the right data collect, here’s some of the insights they gleaned  according to Nancy Lublin, CEO of DoSomething and Jeffrey Bladt:

  • SMS as a platform:  They are monitoring engagement per communication channel and it has revealed SMS to be 30xs more powerful for getting their users to take action as compared to email
  • Challenging 5 friends: we’ve tested various group sizes for SMS experience and have found the a group of 6 (1 alpha inviting friends) leads to the highest overall engagement
  • Research Based Messaging:  The general messaging for the campaign was based on survey findings that found (1) big scare tactics (e.g. getting pregnant = not going to college) we not as effective as highlighting who being a teen parent changes daily life (e.g can’t go to the movies because baby sitter cancelled); (2) a CDC report that found: “The impact of strong pregnancy prevention messages directed to teenagers has been credited with the [recent] teen birth rates decline.
  • A/B Testing: They pre-tested different messages and frequency of sending the messages to smaller test groups of  teens to optimize the number of messages the baby would send during the day, as well as the content.   They ended up doubling the frequency and rewording several interactions as well as building in a response system (so the baby would respond if  teen texted an unsolicited response).  The insights from these tests pushed up engagement and likelihood of forwarding at the end.
  • Impact:  They did a survey to measure this.   1 in 2 teens said that taking the Pregnancy Text made it more likely that they would talk about the issue of teen pregnancy with their family and friends.

As you can see from the above insights,  DoSomething just not gather and analyze topline data:

  • 101,444 people took part in the campaign with 100,000 text-babies delivered
  • 171,000 unsolicited incoming messages, or 1 every 20 seconds for the duration of the campaign. During the initial launch period (first 2 weeks), a new text message was received every 10 seconds.
  • For every 1 direct sign-up, DoSomething gained 2.3 additional sign-ups from forward to a friend functionality.  The viral coefficient was between 0.60 and 0.70 for the campaign.
  • 1 in 4 (24%) of teens could not finish a day with their text-baby (texted a stop word to the baby)

I heard Todd Park, Chief Technology Officer at the White House,  speak about Big Data at the Mashable Social Good Summit say, “Data by itself is useless. I can’t feed my baby daughter data, as much as I’d love to because I love data. It’s only useful if you apply it to create an actual public benefit.” You can’t do that unless identify your results, collect the right data, and generate insights.

How is your nonprofit using data to change the world?

11 Responses

  1. Jay Geneske says:

    This is really incredible, and a great example of tangible impact using new media.

  2. Lee Zukor says:

    Excellent post. From my vantage point, this is an area where non-profits and for profits are in the same boat. It’s easy to be awed by the amount of data collection that is possible– much harder to define why we need it in the first place.

  3. Beth says:

    Lee and Jay: Thanks for your comments. I love data as much as the next person, but I worry about getting too side tracked with data for the sake of data without using it to generate insights.

  4. […] well worth digging into if you’re interested in this sort of thing, as I am. A recent post, Big Data Without Defining Success First Is A Big Mistake, is especially good. It describes a recent campaign by, focused on preventing teen […]

  5. Phil Nickel says:

    Are you connected with the American Evaluation Association? I have not yet seen a summary of your book, but it looks like what you are presenting is a process of evaluation–just what the AEA is all about. (And, btw, their annual conference is in Minneapolis in two weeks.)

  6. Thanks for this contribution! Very valuable indeed! We are a SMALL NGO; we think BIG; we implement BIGGER, please visit Global Climate Change Crisis & Sustainability Panorama (; we are appreciated a LITTLE BIT MORE (METIS Global Awareness Network NGO Accredited Observer to the UNFCCC; Memberships: CC:iNet; GCCA; World Environmental Education Network; Accredited Presenter to the Climate Reality Leadership Corps. However we desperately need BIG support…

  7. […] But Beth Kanter warns Big Data Without Defining Success First Is A Big Mistake. […]

  8. […] tangible handle on the process.  However, as Beth Kantner titled her September 28, 2012 blog post, Big Data Without Defining Success First Is A Big Mistake.  This is where the three O’s come […]

  9. […] Kanter in Big Data Without Defining Success First is a Big Mistake on September 28, 2012 clearly states, But I think jumping into a process: […]

  10. […] Big Data Without Defining Success First Is A Big Mistake […]

  11. Big data has a lot of capacity to profit organizations in any kind of industry, ubiquitously in world. it is a useful to decision-making and helpful to improve the financial position of any organization.

Leave a Reply