The Metrics Syndrome

Not long ago, I attended a conference where most of the participants were academics and the rest of us worked in community settings. There were several interesting panels that brought together people from both camps (and some who bridged the gap) to talk about community cultural development work and its challenges. Again and again, someone working in higher education posed the same wistful question: “What are the metrics for this work?”

“Metrics” has become common jargon for quantifiable measurements of success. This language originated in corporate America, where it at least makes some sense. If you’re trying to sell computer programs to businesses and want to know how well you are doing, setting up usable metrics isn’t all that hard: What percentage of your target market are you reaching? How many customers buy upgrades and new versions when they come out? How do your sales compare with figures for comparable programs? How “sticky” is your Web site and how many visitors make purchases? If your company is publicly traded, what is the price-to-earnings ratio? And so on.

But every time I hear someone talk about community creativity as if it were widget-world, my blood runs cold. I comprehend that the well-intentioned people seeking metrics for cultural development are trying to make the case for something they care about. Quite a lot of foundation funding has gone to studies purporting to establish metrics for community cultural vitality. Some approaches are less odious than others. If the members of a community see their network of associations and activities as vital, permeable and susceptible to their own participation and influence, then it makes sense to conclude that such personal testimony is a valid way to judge cultural vitality. But very often, such subjective evidence is dismissed as merely “anecdotal” and the well-funded search for quantification goes on.

The trouble is, the very quest for metrics is contaminated with ideas and assumptions borrowed from worlds that have nothing in particular to do with community and creativity. The notion that everything of value can be weighed and measured — one of the most grotesque artifacts of post-Enlightenment thinking — is antithetical to the deep values of community cultural development. Indeed, in this domain, the search for metrics actually harms what it seeks to help. Please bear with me while I explain why I see it this way.

The Metrics Syndrome is a manifestation of what the philosopher Friedrich Hayek 60-odd years ago termed “scientism.” Scientism means taking methods and ways of thinking that work very well in the physical sciences and misapplying them to highly complex human endeavors, where they don’t work at all. If you can arrive at solid truth about the behavior of minerals or fluids by weighing and measuring them, this thinking goes, you should also be able to reduce social systems or circumstances to quantitative data, and this should enable you to understand and intervene in them with equal success.

As Hayek and others have pointed out, this is a huge and deeply unscientific mistake. Unlike working with a box of rocks, it is only possible to obtain quantifiable data on a few aspects of human events or situations. Researchers can devise tests and grading systems, tabulate how much time or money is spent in a certain endeavor, or even stretch to the kind of absurdity one state agency achieved in establishing a scale of “artist-client contact hours.” In the domain of cultural development, the data thus obtained will always be limited, and will almost never reveal the most important aspects of whatever is being studied. Everyone who works on the ground in communities knows that all kinds of unquantifiable factors affect the quality and worth of the experience: feelings, ideas, relationships, beliefs and more. In the quest for metrics, such things are dismissed or devalued precisely because they can’t be adequately demonstrated by quantitative measurement. Before long, the idea takes hold that only the factors that can be quantified are relevant and the rest — indeed, the heart and soul of the work — is just some soft stuff that has to be scraped away to get at the facts.

This error is easily compounded when, based on the woefully inadequate information that comes from measuring just a few quantifiable things, people form a hypothesis about what constitutes success, such as a high score on a mathematical scale. When the magic number is reached, we call it success. (The “artist-client contact hours” standard I mentioned earlier was finally dropped when someone who knew a little math pointed out that the most successful project would then be one in which a single artist addressed a vast group of “clients,” thus racking up thousands of contact hours in a single blow.) Cognitive scientists sensibly tell us that human brains succumb to what is called “confirmation bias.” Simply stated, we easily accept whatever confirms our own ideas; it takes extra effort to interrogate our own assumptions, to admit that even though some people believe they are needed to make the case for cultural investment, numerical scores don’t necessarily convey real value or translate into lived feelings of satisfaction, into something we can see and feel in the texture of real life.

Scientism has distorted our understanding of value in many domains outside business and hard science. For example, the phrase “scientifically based research” appears 111 times in the No Child Left Behind Act, where it is defined at length, stressing control-group research that yields quantifiable data. The Act’s mandated student testing focuses on math, reading and science, and many schools have made significant changes in the style and content of curriculum to improve these test scores. This is commonly called “teaching to the test.” According to the Council for Basic Education, a conservative group that generally supports No Child Left Behind, school principals have reported significant increases in class time in reading, writing, science and math, with corresponding decreases in foreign languages, the arts and elementary-school social studies.[1]

The No Child Left Behind Act was passed in 2001. Look around you, at the schools in your own communities that have eliminated arts classes, letting the Act’s metrics dictate their philosophy of education. How is that working out? Are school children being prepared to face the real challenges of 21st Century life and experience the real pleasures of being born with inquiring minds and acute senses? Or is that type of education mostly open to those whose families have the resources to choose private school? There is always the exceptional teacher, the dedicated volunteer, the wise principal who succeeds in any system, but where I live most schools are failing to do the job the great Welsh writer and educator Raymond Williams described half a century ago, articulating education’s purpose as “society’s confirmation of its common meanings, and of the human skills for their amendment.”[2] He was saying that culture, this amazing thing we all co-create, is the matrix for all social and individual development, and that knowing as much as we can of it is important, but it is equally important to experience ourselves as able to change it. To be sure, schools teaching to the test aren’t teaching that.

Just as the people who are searching for a metrics of community cultural development are motivated by positive intentions, no one sets out to harm children. The people who created the dominant approach to education believed that national standards and test-oriented curriculum would reduce inequities so that everyone could obtain a good education. But they succumbed to the typical pitfall of such metrics-driven schemes, which is that they pull energy away from the intended beneficiaries — the children — toward serving the plan itself. No Child Left Behind is only one example of this approach. Many arts advocates have tackled the impossible task of trying to justify increased arts funding on the basis of economic multiplier effects or the community economic development impact of what is sometimes called “the creative class.” Meanwhile, the real value of public arts funding and the arts’ share of private funding have both steadily declined. The scientistic model contradicts the way we actually experience ourselves as makers and sharers of culture. Imposing its orthodoxies on our creativity isn’t improving our ability to understand or intervene in the world. All it really accomplishes is keeping us busy trying to spin straw into gold. When our ways of describing reality are as inadequate to the task as these, no matter how firmly rooted they seem, it’s time to change them.

When it comes to community cultural development, the Metrics Syndrome also has another fatal, intrinsic flaw: it is based on the idea that factors can be derived from past experience, isolated and applied to future actions in such a way that success is guaranteed. But one terrible byproduct of scientism is that it generates theories with absolutely no predictive power. How is it possible to force creative vitality into being? We have only to look at planned communities where the Metrics Syndrome shaped development to see that two and two don’t make community cultural development. Consider the instant villages where prefab shopping districts and common areas attempt to mimic the fabric of organic village life in the hope of reproducing organic cultural vitality. Want to live there?

Every person whose work touches on community and culture knows that just about everything positive that happens in our own work and in any creative realm is a happy accident. It has to be, because when it comes to human endeavor, we can never know enough to manipulate the infinite possible variables to guarantee an intended outcome. Much of the time, we can’t even get the people we know best to do what we want. How often do we find ourselves surprised by the words or actions of someone we know extremely well? Could you ever imagine giving enough data about that person to a complete stranger — even some sort of certified expert in human behavior — such that the stranger would be better able than you are to predict your friend’s next move? When two or three or a million unfathomable humans collide with the complex circumstances of life in the modern world, how can we call the results anything but accidental?

This is certainly evident in art. How many novelists, when asked why a certain character did thus-and-so, reply that they don’t know, that the character seemed to take on a life of his or her own? How many visual-arts effects are the result of releasing intention and surrendering to chance? In dance, in drama and in storytelling, improvisation and interpretation ensure that no work is ever the same twice, that its message always morphs to bridge the ever-changing gap between giver and receiver. Even in science, it is the same. The mathematician, financier and writer Nassim Taleb, who calls himself “an epistemologist of chance events,” has pointed out that in scientific research, “most of what people were looking for, they did not find. Most of what they found they were not looking for.” Penicillin was just some mold inhibiting the growth of another lab culture; lasers at first had no application but were thought to be useful as a form of radar; the Internet was conceived as a military network; and despite massive National Cancer Institute-funded cancer research, the most potent treatment — chemotherapy — was discovered as a side-effect of mustard gas in warfare (people who were exposed to it had very low white blood-cell counts). Look at today’s biggest medical moneymakers: The top-selling drugs treat cholesterol. Statins were discovered by Akira Endo, who grew up on a farm where he developed an interest in fungi. After agricultural school and a biochemistry degree, he worked on fungal enzymes for processing fruit juice. Something he discovered there led him to think fungi might produce chemicals that inhibit cholesterol synthesis. Even that TV star Viagra was devised for another purpose, to treat heart disease and high blood pressure; it made its manufacturers rich by exhibiting a highly lucrative and quite unintentional side-effect.

When we work with people in communities, one accurate way to describe what we are trying to do is to maximize the possibility of positive accidents. If those believing the right metrics can make things happen their way can be compared to builders, measuring out lumber and pounding nails to construct something to their exact specifications, community artists can be compared to farmers: preparing the soil, adding food and water, placing the seeds in their earthy nests, then letting the sun, air and microscopic life forms do their miraculous thing. Our hope is to help create the conditions, the environment in which people’s creativity can flourish and grow, each according to his or her own nature. This can never be quantified.

As artists, we know this. We enact it every day. And slowly, slowly, the larger society is catching on, because as it happens, learning to improvise and trust our senses is more and more what everyone today needs to know how to do. So many of today’s big winners in the markets of social influence and personal achievement are where they are precisely because they pursued their own curiosity and desire through a world of accidents, always on the lookout for opportunity. The people who have become fabulously wealthy by creating Microsoft, Apple, Google, eBay and MySpace could never have set out to make it happen. There was no possible way to plan or train for enterprises that were not even imagined when they started down their life paths. All conventional wisdom and conventional measurements of business success were beside the point. The only thing these entrepreneurs could do was stay true to what they knew, remain aware of what they didn’t know, and keep alert for happy accidents.

The well-meaning arts advocates who have contracted the Metrics Syndrome have lost their way. In their academic departments and research institutes, nearly everyone accepts the quest for metrics as the inevitable road to progress in justifying investment in community cultural development. Overexposed to this way of seeing, they’ve caught the bug. They succumb to the  feeling that they have no other choice. Indeed, they feel they have to keep on enacting the Metrics Syndrome it even though it is a quest that has gone on for decades without any noticeable increase in funding — in fact, with a decline. They’ve lost the capacity to know when to give up a failed enterprise, or to see that the funders who continue to invest in this absurd quest are merely enablers, caught up in a wasteful codependency.

More and more people believe we are living through a liminal time in human history, where an old way of thinking, one that seems secure in its dominance, is actually weakening, beginning to make way for a new paradigm. In this case, scientism is giving way to an integral view of the human subject, wherein we remember that given all our unpredictability and particularity, we can no longer derive our model for social value from the factory, laboratory or corporation. I see artists, especially those who choose to work in community, as standing at the frontier between old and new paradigms. As philosopher Ken Wilber said, when paradigms shift — when the way society understands and organizes reality changes — it can be lonely and uncertain for people who clearly see the emergent reality. Describing them, Wilber used this phrase: “More depth, less span.” We may not be a majority, but despite much social pressure to go along with the old way’s scientism, reductionism and obvious inadequacy to any complex human endeavor, we see it is time to reject all the conventions that reduce us to objects, including the Metrics Syndrome.

I invite our colleagues in academia and civic research to do a very hard and very brave thing: Interrogate every single assumption inherent in the Metrics Syndrome, looking deeply and fearlessly at how this quantifiable approach to value does not serve us, does nothing to advance our campaign for pluralism, participation and equity in cultural and community life. They may have to face peer pressure, but they will be in very good company if the next time someone asks for the metrics, they just say no.

Arlene Goldbard is a writer and consultant soon to be based in Kansas City, Missouri, whose focus is the intersection of culture, politics and spirituality. Her most recent book, “New Creative Community: The Art of Cultural Development,” was published by New Village Press in November 2006. Visit her Web site to read her blog and download talks and writings.

[1] “Academic Atrophy: The Condition of the Liberal Arts in America’s Public Schools,” Council for Basic Education, March 2004.

[2] Raymond Williams, “Culture is Ordinary,” The Raymond Williams Reader, Blackwell, 2001, p. 21.

Original CAN/API publication: October 2008