Outcome Evaluation of Policy Research: Recommendations and Best Practices for Funders, Researchers, and Evaluators

The recommendations and best practices that follow have been excerpted from the last chapter of my Master’s Thesis ‘Who Used Our Findings: Framing Collaborations between Foundations and Think Tanks through Practices in the Evaluation of Policy Influence’, defended at Georgetown University on April 22nd, 2016. They were extracted from a total of 22 semi-structured interviews to experts in foundations, think tanks, and independent evaluation firms.

The study set out to examine the collaborations between foundations and think tanks in order to identify if the former are driving the evaluation of the latter, and to determine whether and how the influence of public policy research is being assessed in that intersection. The motivation for the study was grounded in a paradox: while many collaborations between foundations and think tanks are longstanding and well known, there has been to date no attempt at framing how they are affected by the increasing reliance of organized philanthropy on impact and outcome assessment.

In contrast with that general landscape, the main hypothesis of the study that think tanks supported by foundations are more likely to conduct outcome evaluation, has been supported with the identification of four case studies in which foundations can clearly be attributed as the driving force for the construction of monitoring and evaluation capacities and the conduction of evaluative research. When think tanks build such systems, they are significantly improving their ability to make a persuasive argument about their influence and contribution to the policymaking environment; and this argument is an important part of their fundraising strategy. At the same time, the Think Tank Initiative and other examples of think tank evaluation may have opened the field to studies on the impact of prominent think tanks, like the Brookings Institution, the Bipartisan Policy Center, and the Pew Research Center, that have recently been commissioned by the Hewlett Foundation.


1: Learn from the field of strategic communications

One of the most recurring pieces in attributing value to the contributions of think tanks was the consideration that evidence ought to be packaged strategically to stay influential. Some of the interviewees, like Justin Milner at Urban Institute, mentioned the production of visualizations, videos and blog posts.

Jackie Kaye highlighted the benefits of connecting policy research with the lessons of strategic messaging and communications from a more experimental perspective:

If you have policy research findings, wouldn’t it be great to do randomized control mini-experiments, where half of the people you want to influence get a two-page summary of the findings, and half of the people get a video and a two-page summary of the findings? I think that the work to evaluate policy research needs to get a little more creative.

(Personal interview with Jackie Kaye, Wellspring Advisors, February 22, 2016)

Similarly, Julia Coffman saw that the “wave of the future” may lie in the power of digital analytics for capturing “audience exposure to information and electronic consumption”. Even without advancing that far, some of the tools of market research that gauge behaviors, attitudes and beliefs of target audiences could be adopted more often by public policy research organizations.

2: Think about audiences

As a design principle, the focus on users can be adopted in the earliest phases of research design by addressing the information needs of specific audiences and concentrating dissemination on them.

A recurring recommendation linked to the above is to focus on the users of information. This recommendation dilates into multiple dimensions, as the information consumed can consist in evaluation findings, or in the policy research project; while intended users can be internal to the organization, members of a partnering organization, or decision makers. For that reason, the recommendation to understand the work culture and information needs of users can take many forms. Some organizations focus on understanding the needs of policymakers at the national and local level; others have focused on fostering the understanding of how policies at the federal level constrain the possible decisions of other policy makers; while still others have in mind what can be done to make evaluation findings more useful beyond the context that generated them.

As a design principle, the focus on users can be adopted in the earliest phases of research design by addressing the information needs of specific audiences and concentrating dissemination on them. Additionally, that intensive focus decreases the barriers for measuring uptake in terms of opinions, behaviors, attitudes and use.

3: Map the inclusion of public policy research in larger initiatives

Some of the respondents expressed the hope that the findings of public policy research will be integrated in a purposeful and intentional manner in larger initiatives including different tactics, like grassroots mobilizations, awareness campaigns, etcetera. In this context, evaluation is seen as a test that can help identify what is the most effective placement of new information in a planning sequence.

Best practices for funders

1: Build anchor institutions by aggregation

Offer financial incentives for collaborations between grantees working in the same area with the objective of working through their disagreements and finding a common message that can be disseminated through a collaborative platform.

2: include unnatural allies in the coalition

Including communities and voices that are usually on the other side of an issue, like a hunter and an environmentalist, can decrease the ideological perception of a message and strengthen its credibility.

Best practices for researchers

1: Package information differently

Push beyond the traditional publication of information in books and pdf reports towards formats that summarize information for easy consumption: videos, infographics, visualizations, stories or shorter texts.

2: Target specific audiences

Although federal legislation is usually targeted due to its high profile, less known policymakers at the state and local level, as well as mid-rank employees in government agencies have an important weight in policy implementation. Targeting them at the right time with information that is relevant and connected to their sphere of influence is hard, but can make a difference.

3: Connect the dots

Policymakers are rarely aware of the consequences of their actions for the work of other agencies. Relevant information about unintended consequences in other policymaking communities can promote more nuanced, self-aware and responsible decisions.

4: Create multidisciplinary teams

Varied skillsets can effectively triangulate the lifecycle of a project between the phases of planning, research, communication, and evaluation.

5: Collaborate with advocates

When there are clear legislative goals, collaborate with advocates that can make the message heard in an active way. The inclusion of advocates can also facilitate the assessment of which groups and communities have been informed by a project or are using evidence to support their argument.

Best practices for evaluators

1: Avoid superimposing strict models

Although logic models and theories of change can be useful tools for identifying and testing assumptions and assigning intended outcomes to a project, many funders avoid superimposing their own model, and focus instead on offering support to organizations with similar goals, including support for capacity building in evaluation and planning.

2: Establish intermediate outcomes

Interviewees indicated how intermediate outcomes signaling changes in attitudes, behaviors, discourse and opinions can be measured with tools like public polling or discourse analysis.

Outcome evaluation of policy research cannot be tracked with legislative change only. Other intermediate outcomes that signal changes in attitudes, behaviors, discourse and opinions can be measured. For example, public polling, discourse analysis of public political debates and bill proposals that were not passed, etc.

3: Use evaluation to better understand the world

In the realm of public policy research, the assessment of influence is usually not a summative look on whether a program delivered what it promised. Much rather, it is an examination of how the surrounding environment interacts with the objectives of a program to understand whether the message has to be repeated, translated or shifted.

4: Identify and address key decision points

Evaluation is more likely to be informative and useful when it has been designed to address program-related information needs and is delivered on time for a decision. This often implies designing the evaluation and the program as interdependent processes.

5: Make Social Network Analysis part of the influence assessment toolkit:

The Robert Wood Johnson Foundation has adopted the use of SNA as a way to measure the social outreach of academics, but also to provide insights to advocacy organizations into which groups and communities are underexposed to a certain message. The references to the utilization of SNA for assessing policy influence are augmenting: see Borgenschneider and Corbett (2010, 300) and the guide published by Network Impact and the Center for Evaluation Innovation (2014).

“Right Moves” in the marketplace of ideas

Right Moves, Jason Stahl’s new book, explains how Americans debate policy through the history of conservative think tanks. This post contains a brief review of the book and an interview with the Author.


The Review

Right Moves examines the history of conservative think tanks and their impact on the market of ideas in America. When I first heard about it, I wondered what it would add to the existing scholarship on think tanks. Many experts have effectively recognized how the appearance of new players like the Heritage Foundation and the Cato Institute transformed the field of public policy expertise. If until then Brookings had been the model of what a think tank should be in its research operations and modes of influence, with the arrival of Heritage nonpartisanship (or the appearance of nonpartisanship) was relegated from a mandatory to an optional approach, and a weaker one for that matter: advocacy tanks with major investments in rapid production and marketing of ideas first obtained the favor of the Reagan administration and then took the lead in the nineties, transforming once and for all the idea of what a think tank should do to seek relevance. Goodbye books, welcome briefs.

In some way or another, this fundamental change has been recognized and analyzed by all major experts, be them historians like James Allen Smith, think tank scholars like James McGann, practitioners like Kent Weaver, political scientists like Andy Rich or sociologists like Thomas Medvetz. In fact, the thesis that the advent of this new model of conservative organization displaced the academic production of liberal thinkers, especially of those working in universities, can be found in Smith’s 1993 book The Politics of Expertise and became central in Medvetz’s 2012 Think Tanks in America. So why this book?

Stahl explains how the apparently innocuous notion of the market of ideas became an deological weapon: conservative ideologues’s ideas became respectable because they were conservative, not because they were sound and rigorous.

As I soon found out, Stahl’s book is important because it combines historical accuracy with a cunning look at the development of a concept that may seem natural nowadays but has transformed American policy debates from their root: it is the notion of a market of ideas, which justifies the acceptance of conservative proposals for the sake of ideological balance against dominating liberalism in the progressive era. The author makes a great job at explaining how that apparently innocuous notion became the best ideological weapon, how leading ideologues propagated ideas that were worth of consideration because of their ideological stamp, and not because of their soundness or rigor. What’s more: the book shows how think tanks’ contribution to that notion had a tremendous influence in Washington DC over time. In other words, under Stahl’s thesis the rise of Heritage and Co. not only displaced the notion of what a good think tank should do ––it shifted the perception of what is a valid ideological position, and what type of arguments can enter into a serious policy debate.

The rise of conservative think tanks shifted the perception of what type of arguments can enter into a serious policy debate

Stahl’s approach is very close to the ground: as a historian, he has dived into archives and reconstructed fundamental moments in the genealogy of the current think tank scene. He follows a number of policy debates very closely -on supply side economics, on welfare reform, on the momentous aggressive foreign policy that was already being elaborated before 9-11. Most of those moments describe what are usually considered conservative shops: the consolidation of the American Enterprise Institute, the ascent of the Heritage Institution in the eighties and its symbolic triumph as Alma Mater of the scores of conservatives that were sent to administer Iraq after the 2003 invasion. But what makes Right moves even more important for contemporary readers is its explanatory power regarding the fate of liberal organizations and of the Democratic Party. Instead of serving as a liberal counterbalance to the new forces, the Brookings Institution moved to the right in search of a new center that could be perceived as devoid of bias; and the Democratic Party as a whole moved to the right under the ideological influence of the Democratic Leadership Council and the Progressive Policy Institute, referred to with the unified acronym DLC/PPI throughout the book, which sought to establish a new liberal orthodoxy for Bill Clinton’s presidency under the term New Democrat. That move to the right marginalized the leadership of Jesse Jackson, more solidly aligned with traditional liberal values that connected with the civil rights movement.

The interview


Question: What’s the origin of this book?

Jason Stahl: This book was the topic of my dissertation, although it’s different in the final product. It is part of my work during the last six years, since 2009 to last year.

Q: What is your specific contribution to the existing literature? Some scholars, for example Rich or Medvetz, have already devoted attention to the importance of conservative think tanks.

JS: With this book I am really making a contribution to two big bodies of research. The one that you mention, with Medvetz and Rich as more recent representatives, is certainly one body. The other body has studied conservatism in America. And the specific approach of my contribution can be seen in my training as a historian, which is reflected in how I combine the two themes. I am talking with the tools of a historian, going to the archives and reconstructing institutional development through those sources. Other scholars have used very different methods. They are sociologists or political scientists concerned with how think tanks operate in the present. They conduct interviews and do ethnographic work. In any case, looking at conservative think tanks as sole point of inquiry is new in both literatures.

Looking at conservative think tanks as sole point of inquiry is new in both the literatures on think tanks and on conservatism

What are the think tanks that you analyze in the book?

There are two groups: the most prominent conservative think tanks that most would name, and I mean AEI, Hoover, Heritage and Cato. They are the most prominent, and they all have a place in the book, especially AEI and Heritage.

But another important part is the interrelationships with think tanks that existed before, and of course Brookings is the most important here. These conservative centers established their brands in counterposition with Brookings or borrowing from them in what is a very dynamic process. Especially because, if you look at the development of Brookings over time, it starts out as progressive technocratic, but then in the 1930s it becomes an anti-New Deal endeavor, and it was perceived as being strongly associated with big business. That doesn’t switch until the early 1950s under the umbrella of postwar technocratic liberalism. What I argue in the book is that they reoriented themselves again in the 70s, in large part because of how conservatives reacted to them and to what was perceived as a liberal bias of the postwar consensus, which had had the effect of marginalizing conservative views. So Brookings’ response was to deny their liberal bias with very eloquent gestures that portrayed them as balanced: they brought in a new Republican president and conservative scholars. This dynamic continues to this day. By design, they have been very good at constantly reinventing themselves, shifting, reading the moment of where they needed to be to have the most influence.

Q: What is the role of foundations in this period?

JS: In the 1970s, foundations became a solution for conservative think tanks. The idea that they were fundamentally biased impeded their influence, so initially, between 1970 and 1975, it was a very big solution to say: no, we’re not funded by GE or other corporate funders, but by foundation XYZ, which is more likely to be perceived as a neutral institution that funds research. An example of such funders is John Merrill Olin: an oil business man, Olin decided to start his own foundation in order to funnel money to political projects.

On the other hand, AEI’s Bill Baroody was able to obtain support from the Ford Foundation in spite of the fact that it was viewed as biased towards by suggesting that they needed a more balanced portfolio. For the Ford Foundation, supporting AEI could lend them an aura of neutrality!

Q: What is the key takeaway of this book?

I really want this book to speak to different audiences, both to academics and to a larger public. And I think it’s also very important to have a recognized narrative for this important part of our recent history. These institutions have had an enormous effect on the way Americans debate policy today, and the book can help understand how that frame came to life. Basically, by framing policy debates as a market place, conservatives were able to validate the need of particular types of conservative voices. On the longer run, this strategy has relativized policy debate, has lent enormous new power for conservatism, and has altered how Americans understand the way political arguments happen.

Conservative think tanks have had an enormous effect on the way Americans debate policy today, and the book can help understand how that frame came to life.

Q: Does your book contain any recommendations for the future?

JS: Well, this way of debating policy is just with us. At this point, there might be a need for a broader market of policies. Until now we haven’t had a broad market place, but just a balanced market place. We have two voices, instead of the many possible voices. What we can do is allow for an actually broader marketplace. It’s true that those voices may exist but lack resonance on the mainstream platforms, and on what is heard in Washington; but the media fragmentation that we’re seeing may be signaling to a change in that direction.

Q: We are experiencing a fascinating and transformative election year. How do you think the conservative movement may change in the future?

With AEI positioning itself as an anti-Trump institution and Heritage starting to assume a pro-stance, we can see an interesting institutional split this year.

JS: I think you’re right, one hears so often that we’re in front of a timely transformation that then doesn’t happen, but I think this time we’re seeing a fascinating shift in the primaries towards candidates that don’t fall within the voices that I was mentioning. And it’s interesting to see how different think tanks are positioning themselves in the primary process. What will Heritage do? It seems to me like they are starting to assume a pro-Trump position. So we can see an institutional split, because AEI is working hard to position itself as a Republican anti-Trump institution.

Follow the Jason Stahl on twitter: @stahljason


TPRC42: Predicting trouble & navigating policy solutions

I was in TPRC42 the last weekend thanks to the sponsorship of Georgetown University’s Communication, Culture and Technology program –my program. TPRC is an incredible conference on telecommunications policy research. You can browse the program of the event and, more importantly, download any of the papers of your interest.
TPRC is also quite unique in that FCC staff attend the event in large numbers to understand what’s going on in the cutting edge of the academy – and what should they care about. Or, in words of Mike Nelson (who is a member of CCT´s faculty):

In fact, many of the more interesting conversations in TPRC orbited around the need for more, less or different regulation and policies in the world of ICT around concerns that are more or less common knowledge: privacy, net neutrality, mobile spectrum, cybersecurity etc. The panel “Governing the ungovernable: Algorithms, Bots, and Threats to Our Information Comfort-Zones” evidenced the differences between the proponents of corporate responsibility and bona fide against what Prof. Nelson called the pessimists. The pessimist can see gloomy things when they learn Facebook has developed facial recognition algorithms that are actually better than the human eye.

In a less gruesome and more constructive approach, other attendants demanded ways of making both algorithms and their consequences to the data they crunch transparent to the public.

But, as Mike Nelson noted, algorithms can’t be open because they are protected by patent law. Many of the good proposals and better intentions in the conference revolved around the recognition of complexity and the problematization of existing regulation. In opening remarks of one of the essential references shared during the panel on algorithms:

Regulation is the bugaboo of today’s politics. We have too much of it in most areas, we have too little of it in others, but mostly, we just have the wrong kind, a mountain of paper rules, inefficient processes, and little ability to adjust the rules or the processes when we discover the inevitable unintended results.

Opening Notice and Consent Policies

The 80+ papers discussed in the conference have given me the perfect excuse to narrow down my personal highlights to just one paper that has a bigger affinity with some of my interests, namely customer protection and the creation of initiatives that can ensure a better understanding of law, policies and regulations by the public.

Disagreeable Privacy Policies: Mismatches between Meaning and Users’ Understanding

Led by Princeton’s Prof. J.R. Reidenberg and endorsed by funding of the National Science Foundation, this project analyzed Notice and Choice privacy policies of six major websites (three news outlets plus three e-commerce companies) in order to discern the clarity of their content.

The approach is very smart. Instead of trying to establish what the policies really mean, they were given to three different groups of persons: privacy policy experts, graduate students in computer science and public policy, and crowd workers. Making use of an online tool, each person responded separately to questions about the meaning of the policies.

The conclusions were significant: even amongst privacy experts, the levels of agreement were relatively low in some of the areas covered, with a median across all policies of 75% for experts, 60% for knowledgeable users (grad students) and 50% for crowd workers. This supports the common sense impression that notice and consent privacy policies are too obscure. For Prof. Reidenberg this has two possible explanations:

  1. Companies didn’t realize the general public does not understand their policies
  2. Companies don’t want the general public to understand their policies

If 1) holds true, Prof. Reisenberg could find powerful allies in the private sector for what is his ultimate plan, that is: to create some kind of automated method that can be used to better communicate the consequences of consent to particular privacy policies online. He mentioned for reference a traffic light visual code that would signal invasive policies with red, median policies with yellow and friendly policies with green.

But what if 2) is the case? Well, according to Prof. Reidenberg (and in anticipation of the question I wanted to pose), if companies with a proven record of obscure polices failed to remedy the situation, this paper and similar evidence could be used to start proceedings at the Federal Trade Commission.

Of course, this either/or position is just my summary of the situation. What I really liked about the paper was its commitment with the betterment of a common practice that makes users helpless in regards to their online rights.

People you should follow

TPRC42 had an overwhelming genius concentration. You literally can’t attend the conference and not meet, listen and speak to people with very advanced proposals and powerful insights in the world of Telecommunications. Without any claim to exhaustivity, here are some incredible talents:

Erhardt Graeff, MIT & Berkman Center for Internet & Society at Harvard

Graeff moderated the panel on the regulation of algorithms. According to his blog, Erhardt studies “information flows across mainstream and social media, and explores technologies that help entrepreneurs from marginalized groups, especially youth, to be greater agents of change”. Can you think of a more important topic?

Josephine Wolff, MIT

If you want to understand the internet, you would be off to a good start just by reading Josephine’s articles in Slate. Also a PhD candidate at MIT, her insights in cybersecurity are very revealing.


Luis Hestres, American University

Winner of the student paper contest but a member of the faculty at the University of Texas at San Antonio since this Fall, Luis Hestres is an expert in the use of technological tools in the hands of advocacy organizations. He’s also an alumnus of the Communication, Culture & Technology program from Georgetown University.


Bruce Schneier, CTO at Co3 Systems, Harvard University

Another expert in cybersecurity? Well, no, Bruce Schneier is the expert in cybersecurity. He speaks with the confidence and the clarity of a guru. In the keynote panel on Friday night, he sent a very clear message: given the ontology of the internet, from a cybersecurity perspective it doesn’t make any sense to claim that government intelligence agencies and other legitimate actors should be entitled to actions that cannot be allowed to happen outside of those areas of legitimation, because “everybody is using the same stuff”. Corollaries: we can only choose between private and obscure networks for everybody or highways of open-ended information for everybody. Nothing in between.

Jonathan Cave, Rand Europe and Warwick University

Jonathan intervened in all and every session he attended – and he went to the same sessions I went 90% of the time. An amazing speed of realist, hard-boiled thought with very little space (and mercy) for wishful thinking. It’s just a pity he doesn’t blog.


I will be in TPRC42

The 42nd Research Conference on Communication, Information and Internet Policy takes place from Friday to Sunday this week in the George Mason University School of Law in Arlington, VA. I will be one of the CCT candidates covering the event, which has an amazing number of really interesting contributions.

Friday is Panel Sessions Day

On Friday I will attend two of the Panel Sessions:

900 Questions: A Case Study of Multistakeholder Policy Advocacy Through the E-Rate Lens

Public libraries have played a very important role in the expansion of internet access — in a way, they could be considered the better organized stakeholder promoting a democratic use and a rational organization of online contents. Recognized in the Telecommunications Act of 1996 together with public schools, both types of institutions were the target of the E-Rate program launched in 1995 with the aim of making internet affordable to them and free to the communities they serviced. The consideration of Internet as a public good in America has a lot to do with E-Rate.

Larra ClarkProgram on Networks and Program on America’s Libraries for the 21st Century (AL21C)

Alan InouyeALA Office for Information Technology Policy

Tom KoutskyConnected Nation

Jon PehaCarnegie Mellon

Marijke Visser, ALA Office for Information Technology Policy

Governing the Ungovernable: Algorithms, Bots, and Threats to Our Information Comfort-Zones

This is a great panel to see a top level discussion about the gaps between technological progress and regulation. With the internet of things advancing at great speed, researchers and technology firms are now in the position to plan applications that will alter our day-to-day experience in a way public service can’t even imagine.

Erhardt GraeffMIT Media Lab and MIT Center for Civic Media

Kate DarlingMIT Media Lab, Harvard Berkman Center for Internet & Society, Yale Information Society Project

Davis Hake,  Department of Homeland Security

Michael R. Nelson, Georgetown University