Example of A/B Testing Results
I’ve been reflecting on why some nonprofits do a better job of measurement and learning, while others do not. What is the difference? It comes down to organizational culture. The nonprofits that embrace measurement have a data-driven culture. That is they make decisions based on meaningful data, rather than solely by gut.
Not all nonprofits are born with the spreadsheet gene. And it isn’t simply a technical problem that can be solved through training or purchasing analytics software. The challenge has to do with organizational beliefs and work styles. Whether it be a widely held belief that measurement practice is not worth investing resources. Or a practice that swings the other away where there is an excessive investment in collecting gobbly gook data to appease a funder.
What is needed for nonprofit organizations to make this shift?
The Evolutionary Stages of A Data-Driven Culture
It is helpful to look at making the switch as an evolutionary process. In the end, it comes down to leadership.
Dormant: At this stage, the organization does not know where to start. Does data collection may occur from time-to-time, but not formal reporting. There are no systems in place, no dashboards or simple collection method. Staff is often overwhelmed by thought of measurement and the task falls to the bottom of the to do list. Or there is an emphasis on collecting lots and lots of data, but does not relate it to decision-making. There not is a reflection process for analyzing success or failure to take into next use or campaign.
Testing and Coordinating: At this stage, the organization is regularly collecting data but in a bunch of different spreadsheets and collected by different people or departments. Data is focused on the metrics that are specific to social media channel. It is used to improve content, messaging, and engagement on specific channels. Social media data is not linked to higher level organizational results or mission-driven goals across programs. Discussions on how to improve results are rarely part of staff meetings.
Scaling and Institutionalizing: Has an organization wide system and dashboard for collecting measurement data that is shared with different departments. Has different views or level of detail for senior leaders, implementors, and different departments. Holds weekly campaign check-ins to evaluate what’s working and what’s not across communications channels, as well as, any specific social media feedback received that would help shape our future campaigns or social media use. Monitors feedback from target audience in real time but balances with trend or survey data. Documents quantitative results against goals and value when asked by senior leadership. Works with measurement consultants or specialists to improve skills and capacity. Provides training and professional development for staff to learn how to use measurement tools.
Empowering: Sets organization wide key results areas and key performance indicators that are used across programs. Has a staff position responsible for stewarding organization’s data, but staff are empowered to check and apply their own data. In addition to weekly check-ins, the organizational dashboard includes key performance metrics related to goals as well as more detailed metrics. The organizational dashboard is shared across departments and there is a process for analyzing, discussing, and applying results. They use data visualization techniques to report the data analysis but also to reflect on best practices culled from the data.
There is a regular report to senior leadership which details high level successes, challenges, and recommendations for moving forward. Staff performance reviews incorporate how well the organization is doing on KPIs. Works with measurement consultants or specialists to improve skills and capacity or to commission independent study and provides training and professional development for staff. Celebrates successes by sharing measurement data across the organization.
DoSomething.Org: A Data-Driven Nonprofit In Action
DoSomething.Org is most definitely moving into the “Empowering Stage” and are leaders in the non-profit world for exhibiting the characteristics and work habits of a data-driven organization. Look at their approach to social media measurement in this terrific slide show by George Weiner, CTO, at DoSomething.Org called “What Does The Data Say.” Despite being a relatively small nonprofit, they have a “Data Analyst” on staff, Bob Filbin. What makes an organization to make this kind of investment in being data-driven?
It has to do with leadership. Their Board, which is dominated by leaders in the tech field including Reid Hoffman, co-founder of Linked-In, and Raj Kapoor, co-founder of Snapfish, are all staunchly behind the philosophy of ”The future of the web is data.”
The board supports the organization’s orientation towards using tech and data to realize its mission. CEO Nancy Lublin was the driving force for hiring a data analyst and leading the charge for DoSomthing.Org to become a poster child for a data-driven nonprofit.
So, what does a data analyst do at a nonprofit? It is more than hiring someone who knows how to program formulas in Excel spreadsheets. Bob’s job is to make sure that departmental and overall organizational goals are aligned, and that social media data are seamlessly integrated into achieving their organizational key results.
Bob’s responsibility is less to provide fish to staff, but more to teach them how to fish. “My goal is to make sure that every person on staff has access to the data they need in order to create actionable changes in the way they do their programs. Ideally, each person will receive the data they need with automated dashboards that have different levels of detail and ladder up to organizational results.”
One of the biggest barriers in nonprofits for staff is finding time to devote to rigor and discipline measurement. The time to collect data, the time to analyze, and the time to action on it. Bob concurs. ”DoSomething.Org understands the value of data-driven social change and has backed that up by creating a “data team” of three staff people. For the past month-and-a-half, I’ve been working organizing our data collection, storage, analysis, and dissemination efforts. Unless someone is put in charge of data, and it’s a key part of their job description, accelerating along the path towards flying is going to be hard, if not impossible.”
Bob points out the secret is to not to collect more data, but smarter data. He says, “Just in case data collection can get in the way of achieving goals because it is wasted energy and time. I am working with each department to make sure departmental all data collection supports decision-making. ”
Do.Something is integrates critical metrics from social media, e-mail, SMS, and Web. They don’t just count the data, they use it to improve their tactics. Says Bob, “DoSomething.Org uses A/B testing, where people can be randomly assigned to get different messages simultaneously. ” This fall, DoSomething.org will start a push to acquire members via mobile phone, and A/B testing will be a crucial part of figuring out how to keep those new members engaged. (The graph above is an example.)
Bob likes to quote Hal Varian, Chief Economist at Google, “I keep saying that the sexy job in the next 10 years will be statisticians,” to others on the DoSomething.Org staff. He thinks the business world is moving in the direction of more data analysis. “With the advent of social technology, we are facing an avalanche of data. The goal is to be able to sift through it, and find the diamonds in the rough on how to improve organizational effectiveness. That’s where statisticians, or data analysts, come in. Non-profits know this is true, but the problem is investing in the resources needed to become a data-driven non-profits.”
Bob also believes that part of the problem moving away from making decisions by “gut” feelings, or intuition. Bob says, “The data should tell us whether or not the program is effective.”
To make the shift, Bob suggests using small wins and share an example from an analysis on Facebook Ads for an event sign up. ”We discovered the conversion rate was very low because we directed people to an external site (our web site) versus a sign up on Facebook.” This insight will help use Facebook ads more effectively the next time around.”
Bob also talks about how to overcome resistance on staff to using data for decisions. ”Your reports should be presented in a way that seeks to avoid bruised egos. Rather than bringing a number to a meeting, people should be reviewing their own statistics and data. This is part of what I am doing at Do Something – closing the data loop. Making sure each department can access its data to answer their questions.”
Tips for Making the Switch
Culture is an organization’s operating patterns of behavior, activities, and attitudes, influenced by a shared set of values and beliefs that characterize the way people work together. Changing a nonprofit culture isn’t as simple as
identifying the new ways of working you want to see or articulating a new set of beliefs and values associated with them. Most people won’t change their behaviors until they observe the role models in their organization acting
differently as DoSomething.Org has done. Also, when new behavior is positively recognized and rewarded — job promotions or some praise from the top of the organization – change begins to happen.
1. Start at the top. Does your Executive Director know where the organization stands? Educate through examples – showing how adding a data-driven approach to your social media can avoid ineffective campaigns and increase audience satisfaction. The organization’s leadership needs to model and encourage a data-driven approach.
2. Make the case to improve your measurement practice. The only way to evolve is through implementing a series of social media measurement pilots and small data wins. Keep the end in mind when agreeing to how experiments will be structured, run, and measured.
3. Think big, but take baby steps. Start with looking at Key Result areas and key performance indicators, but since these may outcomes deal with long-term changes, you can’t get there overnight. Keep the steps in the plan small and manageable. Use measurement pilots.
4. Share stories: Celebrate every bar graph that leads to a program or campaign victory. Share it at staff meetings. Also circulate stories about other nonprofits that have become data-driven success stories.
Does your nonprofit have a data-driven culture? How are you making the shift? Where does social media measurement fall in that mix?
I’m working on a book with KD Paine about social media, networked nonprofits, and measurement. Have a story to share? Let me know in the comments. You could be in the book!
Recent Comments