heravalue

Reflections on the public value of arts and humanities research

Sorting out the competitive research grant lottery

14326101068_b3d4b990ec_z

How should governments allocate resources to research projects?  The usual answer to this is through a mixture of strategic allocations in areas of national significance, recurrent funding for those with proven pedigree and responsive funding for the ‘best’ new ideas.  Funding allows a diverse research ecology with a strong research infrastructure to support a research community in which no blind spots emerge.

Grass-roots – responsive mode – funding is always a bit of a gamble for all those involved.  Funders have limited resources, and developing proposals can be an incredibly time-consuming effort.  Therefore the ‘oil’ in the system is a willingness for researchers to spend their scarce time developing proposals that might not be funded.

That willingness is strongly dependent on the system’s credibility to deliver fair outcomes, and to ensure that if you play the game, follow the rules and come up with a good idea, then you’ve got a decent chance of funding be funded.  And that willingness is strongly influenced by relative success rates – if good research proposals are being rejected with no chance of funding, then the funding system starts to have more in common with a lottery than a credible objective governance tool.  Arguments vary but empirical data suggests that there is a credibility threshold of success rates at around 25-33%.

It’s plausible that three-quarters of proposals just aren’t good enough, because we’ve all been asked to review weak proposals, and most of us accept that sometimes we just get it wrong.  So when success rates can hit as low as 2% as the Dutch Social Sciences open Ph.D. call reportedly did in recent years, then there are strong negative effects, with people being unwilling to invest significant time in developing proposals.

This undermines the role of the responsive funding to support risky, path breaking research, and the Matthew effect means that it is people with existing resources, not necessarily the best ideas that have the time to write proposals for these “funding lotteries”. A real science policy problem, and one which has also affected by UK’s ESRC, as noted by Phil Ward, with outcomes for November 2014 noting that more than half of the excellently-scoring proposals were not funded.

Phil raised the question of what is the solution to this problem, noting that the ESRC “combing through 144 applications to identify the 14 to be funded seems like a colossal waste of their time. It’s inefficient, and it’s not good to appear as such with the Nurse Review just around the corner”.  One suggestion he made was a two-stage process, as was previously used in UK research council Research Programmes, and is now common in European Joint Programming Initiatives and ERA-NET schemes.

He also argued for more demand management at the level of universities to stop proposals with no chance of getting funded going forward.  But that’s asking demand management to make some fairly fine calls, with 45% of those submitted ranking as at least very good, significant value with the change to make a real contribution.

Could universities, as private institutions with their own politics, really be trusted to make a distinction between holding back a superstar Prof’s very good proposal to allow a post-doc’s excellent one to go in?  Why swap a large pool of peer reviewers for a tiny one, full of internal politics?  Why ask researchers to spend their costly time to run their own institutionally-based research councils?

If the effect is as desired driving a further upward shift in proposal quality, demand management is effectively adding to the lotterisation of the system: you buy a ticket with an excellent proposal and hope the stars shine on you.  And ultimately, if you believe universities can be trusted identify the best research proposals (and that’s by no means uncontroversial), then why have research councils at all?

Clearly, there is some scope for more top-down management, but the fundamental point is that there’s a fine line between ‘nurturing’ proposals and stitch-ups that primarily benefit established researchers over innovative researchers.  That could be avoided by having a process with more stages in it and certainty given before projects are definitely finalised.  But that risks being a massive administrative burden, and a bureaucracy squeezing out academic input, and favouring box ticking and inevitably conservatism.

These problems also reflect the shortage of cash.  I was lucky enough to be in the UK during the boom years of the early 2000s when there was suddenly as much money as good ideas.  That money was well invested in creating a generation of researchers who are now aggressively chasing every opportunity with brilliant ideas, with the well-known consequences.

It’s not just Research Council research funds that are under pressure – their admin budgets are also under pressure. There is a present risk that administrative concerns lead them to drop this essentially bottom-up tool to favour a top down centrally guided set of funds.  This can be seen with the popularity of the idea of massive research centres which address strategic themes (or depending on your position, today’s craze).  But they do at least have the plus of being easy to administer compared to the headache of deciding which of 144 excellent proposals get the nod.

We’ve been here before, with the US trying to address the time waste issue already in the 1980s.  Their efforts stranded  so it could be that the problem is just intractable: “you want dynamism, you get waste”.  Any replacement needs to not just address the symptoms of administrative burden but primarily ensure there are incentives for science system renewal that provide substantial public benefit.

Perhaps the 10% success rate is just a blip arising from one meeting in the ESRC case, but clearly, funding rates far below the credibility threshold are a growing problem across Europe needing our serious attention.

(This post is a condensed form of a spontaneous Twitter conversation involving @Nightingale_P @heravalue @AdamCommentism @kieronflanagan @frootle @Cash4Questions) arising from @frootle’s original post; all errors and omissions remain the responsibility of @heravalue ).

Photo appears courtesy of catherinecronin under a CC BY-SA 2.0 License.

Advertisements

An unusual take on the Research Excellence Framework

Image001

 

We are just in the process of preparing the HERAVALUE final report, giving an opportunity to reflect on our progression over the last 18 months.  It is only really in the last six months that the various ideas have started to cohere into a wider analytic framework contributing to debates concerning arts and humanities research’s public value.

At the heart of our findings is the notion of scale: research produces impact as it ‘upscales’: scholars exchange knowledge with users, users diffuse that knowledge in their networks, and these wider networks create new societal capacities to realise the ‘good society’.

Transactions at particular scales each have their own logic.  A researcher may be primarily concerned with ‘excellent’ research, and configure that research’s social utility to fit with excellence.  Conversely, a research user is only interested in its societal utility, and whether the exchange with the researcher is ‘productive’ in answering their own questions problems.

This multi-scalar distinction is useful in explaining the central policy conundrum for humanities’ public value.  We repeatedly encountered the apparent impossibility of agreement between government and their wider research stakeholders on a definition of effective measures of humanities’ public value.

This most obviously manifests itself in a widespread conflict in claims between government and academics.  On the one hand, government demands that scholars hold themselves to account, and on the other, scholars complain that there are being instrumentalised, reducing their research’s overall usefulness.

What a multi-scalar framework makes clear that these two positions are actually both logically correct, but refer to two very different domains, with their own logics and dynamics.

The researcher is quite correct to say that if research follows a trajectory too narrowly tied to particular users, then what is gained in usability is lost in generalizability.  The public value of research lies in the point of upscaling from when private (individual) benefits become public (collective) benefits, and so this generalizability is absolutely vital to public value.

But the policy maker is quite right to say that in return for public funding, the humanities have to be held to account, demonstrating they spend money responsibly through frameworks which command public confidence.  With increasing public interest in being involved in debates and discussions, research has to have a capacity to interact with publics and demonstrably satisfy these demands.

The two issues are not necessarily logically incompatible, but in public debates there has been an elision between the two scales: each side has on the basis of their own scale-dependent assumptions then made deductions about the other scale.  These elisions are contradictory and prevent a consensus position being arrived at.

There is a problem in arguing from an accountability position – research should create public benefits (macro-scale) ??? to arguing all research should be managed to create public benefits (micro-scale).  This is directly contradicted by the position that tying research question choice too closely to users reduces the overall public value.

Likewise, there is a problem in arguing from the academic freedom position – independent choice of question leads to the best long-run public benefits – to making a macro-scale argument that humanities research should not in some way be accountable to the public.  That latter position is clearly untenable as a simple piece of special interest pleading.

That has clear implications for dealing with the tensions: avoid slippery elisions between these scales.  If you want accountability, then hold accountable at the scale of the network or the system, and if you want excellence, then give individual researchers a free hand in the choice of their questions.

It was only when attempting to articulate that concept that it occurred to me that this is precisely what the UK’s Research Excellence Framework (REF) is seeking to do.  The REF has a highly contentious element where 20% of the departmental scores are derived from ‘Impact’.

‘Impact’ has been widely criticised for forcing individual researchers to be too concerned with ensuring that their research is taken up in wider research networks and embedded into artefacts that can be sold (see inter alia Shepherd, Oswald or Fernandez-Arnesto).  That potentially has the effect of undermining the excellence of that research by tying the choice of question and research prosecution too closely to outside private interests.

But what is salient here about the REF when viewed through this multi-scalar perspective is that the REF does not make this elision.  Impact does not operate at the level of the individual – unlike the research excellence score, based entirely on individual publications.

Impact is constructed as a collective variable – choosing the departmental unit of assessment as the collective does not seem unreasonable given that it corresponds most closely to the unit of self-governance (even if management changes in English universities have reduced the level of self-governance departmental units enjoy).

When so viewed, the REF’s has an underlying logic, evaluating individually the things individuals are responsible for, and collectively those things (‘impact’) which not all can be expected to provide but which are part of the justification for public support.

That does not mean that the REF in practice makes sense.  In particular, university managers pressuring individuals to create impact does not necessarily make sense (in the way that pressuring them to improve their publications does).  But universities are self-managing institutions and it is not right to confuse the effects of institutional choices with the logic of the system.

So from this perspective, the REF is logically consistent, attempting to identify societal contributions at the collective level rather than measuring the sum of all individual contributions.  It therefore avoids an important and illogical elision between the different ‘scales’ of public value.

I was initially surprised by this outcome as the REF and impact has been subject to such extensive criticism, creating a ‘no smoke without fire effect’.  But on the other hand, the fact that the multi-scalar framework can raise such a counter-intuitive result suggests that it is an interesting model, and worthy of further consideration.

Paul Benneworth, Enschede, 18th February 2013.

The image appears courtesy of AlphachimpStudio.

 

Financial crises and policy panics in research

Image001

The HERAVALUE research project has been conducted against the backdrop of the biggest financial crisis in living memory that has been associated with a savage period of reductions in public spending.  And inevitably this has had implications for the object of study – the policy debates and processes about what humanities research should be doing.

Most of our research has focused on the elements of the policy process relevant to arts and humanities research.  But the elephant in the corner of these studies has always been the crisis, and in particular, the effect that this has had both on individual policy-makers and networks, but also the wider policy process and discourse.

The effect has varied across our study countries, and in the case of Norway, where its oil production and sovereign wealth fund have stabilised the negative effects of the financial crash, it has not been a salient issue for policy-makers.

In the case of the Netherlands, there is the sense that the crisis became a post hoc rationalisation for things that the government wanted to do anyway.  The Top Sector policy chose to concentrate innovation funding around nine areas of Dutch economic strength, and encouraging research and teaching to make itself more relevant for those sectors.

In Ireland, the austerity cuts have necessitated brutal changes to research.  The Research Prioritisation Exercise has selected areas of research for funding on the basis of their capacity to contribute to economic growth, and concentrated core research funding into those fields.

At the EU level, discussions about Horizon 2020, the successor to the European Framework Programmes, have been dominated by the idea of grand challenges.  A more sceptical reading of them suggests that it is quite simply put an attempt to invest in areas of research that produce a more immediate economic dividend.

That is indicative of a wider trend by policy-makers to try to spend their scarce resources on research which has a direct economic impact.  After all, during a deep recession the primary goal for public policy has to be to restore the growth and the tax base that will allow service to be restored to other areas of public service.

The effect for humanities research has been quite clear – they have felt continually threatened during this period.  They have been continually battling – not against policy-makers themselves – but against a policy environment which seems to assume that they are less useful than a particular set of useful disciplines.

Both the Top Sectors policy and the Research Prioritisation Exercise have embedded these assumptions further – that money spent on the ‘hard’ sciences is useful, whilst that spent on ‘arts and humanities’ is a luxury.  That is an extremely difficult, almost impossible, position for the field to be battling against.

And that has certainly put a wide range of humanities communities on the back foot in science policy discussions.  There have been both eruptions of vitriol in the field from those that claim that the point of humanities is not to be useful, to those who feel that unless the value is made clear then humanities risks swingeing funding cuts.

Why are policy-makers so certain at this moment that investing in useful science is the way out of the crisis/  Certainly, evidence in this area is not particularly encouraging that useful sciences are likely to lead to economic growth.

It seems strange to assume there are a wide range of project ideas that are immediately going to lead to high growth companies just waiting on researchers’ shelves.  There is strong evidence that the relationships between science, innovation and economic growth are less linear than one might suppose.

There is evidence that investing in university research creates economic growth, but that is primarily a Keynsian multiplier effect rather than the effects of creating spin-off companies.  Compared to their multiplier effects, the direct effects of universities on the knowledge economy are at best puny.

In the UK, universities establish 250 companies a year and generate £100m income from intellectual property activities.  This is nothing but a drop in the ocean of the UK’s knowledge economy, where £17.4bn is spent by business employing 160,000 researchers. 

But clearly, the debate is being influenced by this illusion that investing in science, technology, engineering and mathematics is driving economic growth.  It’s not just the policy debate, but it is influencing the way that the humanities community are debating their own contribution.

It has created an unrealistic expectation of what science can deliver, and unsurprisingly, some humanities scholars have pointed that out.  But those comments have had real effects, leading to humanities being depicted as self-interested and self-referential, which has had an effect of pushing the debate further in the opposite (wrong) direction.

We know that the banking crisis was built on a massive, systematic and misguided belief that an unregulated banking sector could lead to unlimited growth and welfare gains.  The crisis we are suffering today is proof of what happens when policy-making becomes driven by articles of faith rather than understanding the way the world works.

It would be ironic if it were this crisis forced policy makers to run from the arms of one false belief another, in the value of hard science as an economic engine, and at the same time risk undoing the great work that all scientific research can do for society.

Paul Benneworth, Enschede, 11th February 2013.

Photo appears courtesy of Infomatique 

Disclaimer: Many thanks to the Irish and Norwegian teams for their factual contributions to this entry, but the views remain the sole responsibility of Paul Benneworth.  The photo is taken from an exhibition in Dublin looking at how the arts can provide a means of coming to terms with the social devastation being wrought by the crisis and austerity in Ireland: the views on the slate are an artistic statement, and do not represent the views of the HERAVALUE project partners.

 

The public value of A&H research as circulating concepts?

Image001

One of the limitations to the HERAVALUE project has been our focus on the micro-elements of the way arts and humanities research meets society.  This reflects two of our starting assumptions.

The first was that the value of (arts and humanities) research is specific to research projects, and there are unlikely to be simple metrics at the aggregate level that capture in a realistic way value beyond a limited number of economic outputs.

The second was that the value was in part negotiated by elite actors in higher education policy systems and it would be traceable through ‘measuring artefacts’, like laws governing higher education and research, or research evaluation system. 

To some extent these assumptions did hold, with the possible exception that there were no stable measurement frameworks that had become ‘artefacts’ and used unself-consciously.  Rather, there were proposals with a degree of stability whose use was dependent on their legitimacy within the various stakeholder communities.

The micro-approach did have advantages in terms of allowing us to explore in detail assumptions often made about arts and humanities research, value and behaviour.  It also helped us discover the upscaling mechanism – because interviewees reported that part of the value was the way in which articular research findings became codified and institutionalised in social, economic and cultural infrastructure.

But what we could not trace in an effective way was how that upscaling mechanism worked.  We could and did study how researchers created knowledge and how that was transferred to users, and through interviews with users, how they in turn incorporated it in their own products, events and services.

We gained some insights into the role of the media in this process, in terms of the way particular outlets regarded “humanities science” stories.  Through their audience numbers or readership, it was clear that there was a substantial public interest in the insights generated in arts and humanities research.

But what we missed out was the next step, which was to look at how these audiences reacted en masse to these insights, and what happened differently in particular social contexts because of the research.  What we couldn’t do was to create a definite link, to say that a researcher had made a discovery, and that had changed the way that people made their definitions.

That is actually quite an important step with humanities research, because humanities research usually is not a discovery in the scientific sense.  So with the Higgs Boson, the story was that the Boson had been proved to exist, and so everyone who knew about high-energy physics could now know of its existence.

With the exclusion of things like archaeological discoveries such as the recently found grave of Richard III in England, that is not the way that humanities research influences society.  Rather, research mobilises concepts that are circulated and change the way that people think in imperceptible ways, mediated through gatekeepers who incorporate those findings in different ways in their own pronouncements.

The archetypal example of this is a social scientist who develops a policy discourse that shapes the way we think about the world.  Or more realistically, a community of researchers who articulate a concept that becomes influential – the idea of New Public Management exploded in the mid 1990s to become an increasingly dominant and normative view of how public services should be arranged.

So it would be nice to be able to trace how these concepts ??? developed by researchers within communities ??? go on to have impact and change the way that people think about ideas.  And this takes us forward into one potential next step for our research.

Computational linguistics provides a means to explore the phenomenon and to understand the processes at work here, at the level of society.  Computational linguistics has a capacity to use large corpora of texts to understand languages in terms of the relationships of words and concepts to one another.

The diagram below, taken from Alonso et al., shows how, from a corpora analysis related to heritage policy, the concept of “heritage” relates to other words and concepts.  It shows how in aggregate, the users of one language in a particular defined domain area imbue a concept with particular kinds of significance.

In the case of the noun heritage, it is related to particular kinds of actions – related verbs, such as to live, to build, to safeguard and to preserve, as well as particular exemplifications – related nouns, including tradition, house, community and property. 

Image002

From Alonso et al., 2012

By looking at how languages evolve over time, by taking corpora at different points in time, one can observe and infer how the ways that people – publics – use words changes over time.  Critically, it is possible to see the shifting meanings that people ascribe to those words, and to see which elements of meaning become more or less important. 

If humanities research was making an impact at this aggregate scale, then one would expect to see more nuanced meanings emerging in these particular domain areas over time, related to when those nuances emerged in research communities.

Hypothetically, one might imagine that the last five years has seen an evolution in the way that concepts around global security have evolved in response to humanities scholars’ influences.  One might expect that there has been an expansion of the way that global security is understood away from technical definitions relating to screening and surveillance to reflecting the role that cultural self-confidence, diversity and tensions play in global security.

That might be quite an extreme example, but it highlights the way which one particular digital humanities technique, computational linguistics, can provide an additional dimension understanding the public value of research. 

If one could demonstrate that the meanings that people gave to words were shaped by ideas circulating in networks of humanities scholars, that would provide additional insights into how humanities research creates societal benefits at that highest level.

More generally, the issue of how concepts circulate is important to understanding impact, and rather than being a chain model of innovation, concepts circulate at three kinds of scales, the

·         micro ??? between researcher and user,

·         meso – between researchers, users and their immediate networks, and

·         macro – from learning networks to the level of the imagined community.

More effort is required to understand these circulation dynamics and the different logics of the various actors within these communities.  Focusing on circulation dynamics would imply trying to measure the properties of the circulation, including the speed, scope and significance of circulation, rather than particular static measures at single points in time.

If arts and humanities research really is about changing the world by understanding it, then it seems only logical that measuring change should be at the heart of measuring its value.

Paul Benneworth, Enschede, 7th February 2013

Photograph appears courtesy of Elvillano

 

Four dimensions of A&H research value: configuring, absorbing, exploiting, codifying?

Image001

We have elsewhere set out our multi-scalar heuristic for how arts and humanities research (and indeed many other kinds of research) creates ‘public’ or societal value.  The basis of the model is that knowledge is created, transmitted, diffused and institutionalised, and by the end of that process, it has become ‘useful’: the value of that knowledge should therefore be understood in terms of this process of ‘becoming useful’.

The model regards each of these spheres or levels as a different ‘scale’.  I here use scale in the geographical sense – a scale is a level in which a process operates in a clearly distinct way.  The distinction is easy to illustrate in physical geography and the hydrological cycle.

The hydrological cycle follows the way that water and sediment interact as rain moves to the sea.  There is a common process of hydrological dynamics, but depending on the scale of that process, it makes sense to model those processes in an entirely different way.

Rain falling on a bare mountaintop is in principle no different to a river flowing through a delta, in both are hydrological processes.  But there is a clear difference in what matters as an object of study because as a scientist you are interested in what shapes observable phenomena.

The raindrop falling on a mountain weathers and erodes the rock leaving visible imprints in the surface.  The river flowing carries many tiny eroded rock particles, and the way they interact shapes the form of the delta.

Each of these processes has a very different logic, and is best described by very different kinds of models, even if the same kinds of underlying factors – water, rock, slope, intensity of water flow – are at play in both situations.

* * * * * * *

Likewise, the dynamics of arts and humanities research impact is best understood using different models for each of the scales.  And if you want to measure the impact or value of that research, then you should ensure that you are capturing all the elements of the model.

There appear to be four scales at play here, the individual (creation), the micro (transfer), the meso (diffusion), and macro (institutionalisation).  And there are very different kinds of processes at play here, or at least different in terms of which are the salient variables for understanding them and whether their research is useful.

1. Do researchers build accessibility into their plans?

At the individual level, researchers make choices about which research paths to follow and not others on the basis of a mixture of rational and opportunistic grounds – whether there is an interesting research question, and whether funding is available to pursue it.  The literature about knowledge exchange suggests that research into which generating impact is planned at an earlier stage will have more impact than that at a later stage, and the effort that is made to ensure that the findings are open and accessible to users.  Therefore the value of that knowledge can be understood at least partly as a function of researcher effort.

2. Are potential users actually bothering to acquire the knowledge?

The micro-level of knowledge exchange is the level of the individual knowledge exchange transaction.  As Spaapen et al. make clear, the researcher does not themselves have to be aware of the transaction, they may be direct, indirect or financial.  But these transactions all indicate a degree of activity by the user, making an effort to acquire the knowledge, and therefore indicating that the user sees value in that knowledge. The value of that knowledge is therefore dependent on the level of these transactions, and the efforts made by users.

3. Are users actually doing anything with knowledge they acquire?

At a meso-level, the next element of the knowledge exchange process is the user incorporating that knowledge (/artefacts) into their own learning processes and creating a vector for the idea.  In this process, the ‘knowledge’ shifts from the academic domain into a learning community where the new knowledge is created; the dynamic of this knowledge is no longer reliant on the knowledge creator, but on the new knowledges combined and created within this wider community.  The value of that knowledge can be understood in terms of the way these vectors flow within these learning communities, and their importance to the emergent solutions in these communities.

4. Do these learning communities change institutional arrangements and cultures?

The macro-level of the knowledge exchange is when particular localised socialised learning activities become institutionalised into formal organisations and informal cultures.  The basis for this process is that a learning community creates a solution which is then adopted elsewhere without these later users necessarily being aware of the collective learning that went into it (think of the difference in Open Source software between authors and users). 

By being embodied the ‘solution’ can diffuse widely and spread into many domains beyond that originally intended by the knowledge creators. The relationship of this macro-impact is undoubtedly hard to trace to research activities, but these is a clear causal chain from the individual to the micro-scale.

* * * * * * *

So our argument, answering one question at the start of the project, are what are the boundary conditions for ‘good’ measures of the public value of humanities research can be distliied as

any kind of measure of value needs to capture four dimensions, researcher planning, user effort in acquisition, user effort in exploitation and codification/ institutionalisation.

Paul Benneworth, 6th February 2013.

Photograph appears courtesy of FrankBoyd

Arts and humanities research contributing to ???a good society????

Image001

A previous post dealt with the question of the different scales at which arts and humanities research can create impact, from individual users to societal structures and institutions.  This suggests that arts and humanities research might be able to create collective benefits at the level of societal institutions if that knowledge becomes embedded in different ways in collective activities.

This sometimes leads to the claim that humanities research is part of what is means to be a civilised country, and even that arts and humanities research can justify its support and privileges because of a ‘civilising effect’ in ways that other kinds of more technologically-oriented disciplines might not. 

But this is not a claim that stands up to any real kind of scrutiny.  The point fundamentally reduces to a claim arts and humanities research makes people better people.

There is an extensive literature of the moral ambiguity of research.  It is the eventual consequences of how that research is used rather the fact that it is humanities research that makes it publically beneficial.  The choices that users make in using it affect whether that research is positive or negative.

The role of Belgian anthropologists in constructing ethnic divisions in what is now DR Congo served the short-term administrative convenience of the colonial state, but its longer term effect was fossilising ethnic divisions that were later the basis of Rwanda’s genocidal civil war. 

There is no rational argument which can be made that says that arts and humanities research is intrinsically a positive benefit to society and unquestionably a force for good. But at the same time, much of the benefits which arts and humanities research brings are in the social rather the economic spheres of society, raising the question of how can those contributions be understood. 

So there is value in attempting to understand the conditions and the mechanisms by which arts and humanities research might be thought to create public goods.  The previous post distinguished that when we talk of the public impact of humanities research, we are really talking in a compressed way about a process that operates at four scales. 

Firstly, knowledge is produced, then it is transferred, then it circulates in communities, and then finally these communities create cultures and organisations which are influenced by the original knowledge (embedded in structures).

This means we can say that research that creates ‘good impact’ is research that helps to create ‘good social institutions’.  This raises the question of what are good social institutions and a very general answer is that these are institutions that contribution to a ‘good society’.

This takes us into an interesting philosophical area of the good society: good institutions can be regarded as institutions that help individuals and collectives to live a ‘good’ life, however that is defined.  The question of a good life is one I prefer to leave to ethicists, but a clear contribution to this debate, coming from Sandel is that the market can never provide everything that is necessary to live a good life.

The prerequisites are those things necessary to participate in society, and Bozeman & Sarewitz argue that common to most societies is an agreement that certain things matter.  These may include low child mortality, no child labour, universal literacy and suffrage are necessary to ensure that everyone has the opportunity to participate in society.  The problem with markets is, according to Sandel, that they have a singular definition of what is good, far more limited than societal standards.

Global production networks may hide child labour, yet we may through market transactions effectively judge that child labour is acceptable if it is invisible. ‘Market values’ hide those things that do not fit with their narrow framing, unless considerable effort is made to reveal and cost in those ‘negative externalities’.

Using the Sandel argument, arts and humanities research might – and it is not a unique claim – be able to contribute to thinking about the ‘good life’ in ways that take us beyond markets.

One might therefore argue that one of the areas where arts and humanities research might be able to contribute is in creating capacities to deliver things that markets are unable to deliver.  This in turn could create a wider range of choices for individuals and collective organisation to achieve better outcomes. 

An important part of ethnic equality in many western societies has been coming to terms with colonial pasts and institutionalised racism, drawing on humanities research to create understanding and capacity to deal with the phenomenon in its contemporary incarnations.

This at least provides a criterion for how humanities might contribute to the ‘good society’ beyond the generalisation that humanities research is part of a civilised society.  Good humanities research is research that creates or become enrolled in the creation of societal capacities that improve societal capacity to live a ‘good life’, however particular societies and collectives choose to make that decision.

There is still a gap between that criterion and the upscaling process from the individual transaction, but that at least clarifies the end point of the journey.  Good impact is about creating new societal capacities and opportunities. Any research measurement system needs to have thought about two things, and come up with clear answers to two questions.

Firstly are the signals that its indicators give as to what they are saying is a ‘good society’ – if they encourage particular activities, what vision does this offer for societal development?  But arguably, and more importantly is the question of what does societal collectives think matters how do they define a good society, and ultimately, how can research contribute to those societal rather than elite administrator visions?

Paul Benneworth, Enschede, 6th February 2013

Photo appears courtesy of LHG Creative Photography.

Arts and humanities research???s public value: cui bono?

Image001

There’s quite a lot of debate over the meaning of the term ‘public value’, particularly when applied to the idea of research.  It has come under particular strain because of attempts to produce measures of this public value, which highlights the disagreements about what people mean when they use the phrase ‘public value’.

There is a clear division between the particular and the general in public; there are particular publics, with similarities that means that it makes sense to treat them collectively.  There are also connotations to ideas of ‘public’ as a part of society: people talk about public accountability or the public sphere.

So there is a key challenge in thinking about the public value of research in thinking about the scale of the public – is it a particular real group or community, or does it refer to public as an abstract category.

I have already argued in this blog that the attraction of measuring economic value is that it finesses having to make this distinction.  If you sell something, then it has a private benefit, but that private benefit creates a ‘transaction’, which creates a domino effect of further activity.

So if I license a drug patent to a company, then as a university, I can pay my employees, who collectively get to share in these benefits. They then spend their income on goods and services, creating a ‘ripple effect’, through the wider economic.  And that ripple effect can be converted into a benefit to the national economy in terms of a contribution to GDP.

And so on the basis of quite a long argument chain, you can plausibly claim that licensing a patent increases national wealth – a clear collective benefit.  And that is in turn based on a sophisticated understanding of the relationships between these different conceptual levels.

There is an individual exchange, that becomes a transaction; the transaction creates a private benefit for employees; employees’ behaviour creates a  wider network effect (a club benefit), that gradually ripples out into a general national benefit (a collective benefit). 

I have also argued that this ‘up-scaling’ process is not well-understood beyond models of economic structure, because there are very few genuinely multi-scalar sociological models. 

It is clear that arts and humanities research is consumed by individuals in ‘transactions’, that might be difficult to measure.  It is individuals who have driven the media attention for the recent discovery that a skeleton in a car park in Leicester is that of Richard III, and it is historians who have imbued that story with meaning by framing it as a narrative of the last of the Plantagenet Kings and the last English monarch to die in battle.

But arts and humanities research can also be taken up in communities, creating a meso-level collective benefit.  Lawyers might be wrestling with a particular legal problem and draw on law research in their deliberations.  The research becomes embodied in wider knowledges that circulate in this larger (but still real) community as they attempt to deal with a particular challenge (such as in Netherlands the periodic tidying up of public and private law books).

There is than a third step in the ripple from where the knowledge circulates consciously in these relatively closed networks, to where it becomes a general asset that is used unself-consciously without the original researcher input being active or obvious.  That can be understood as a kind of institutionalisation process, where active lived knowledge is embedded within routines and structures.

Image002

What we understand quite well are the processes within these different scales, the knowledge creation, the knowledge exchange transaction, learning in communities of practice, and circulation of knowledge in virtual communities.  What is not so well understood are the processes by which knowledge moves across these different scales.

Well, that’s not entirely true.  We do have a good understanding of the first inter-scalar process, with the research knowledge being passed to individuals in transactions, and to some extent the second, when knowledge circulates in learning networks or community of practices.  What is not so well understood is the role played by research in this second process, beyond a relatively simplistic idea that there is a ‘knowledge pool’ that is periodically filled up by research.

And what is certainly not understood is how research becomes institutionalised – formally and informally – in societal routines, habits, but also legal systems and organisational forms.  So one of our clear conclusions on areas that further research is needed is on the meso- and macro-scales of knowledge exchange in the arts and humanities

That will allow us to properly understand how this research informs and shapes our public institutions, and hence enable us to speak with greater clarity about the public value of research.

Paul Benneworth, Enschede, 6th February 2013.

Photo appears courtesy of UIC Collections.

 

What are the right incentives for promoting public value?

Image001

The final challenge in understanding the public value of arts and humanities research comes with attempts to develop measures that stimulate researchers to do more of a particular kind of activity.  And there is a clear problem that it is very hard to measure what arts and humanities researchers do in ways that convincingly prove that it matters.

There is a thorny issue here of how do you make sure that incentives and rewards encourage good activity.  Policy-makers are very drawn to the idea of managing by setting targets, and rewarding those who are good at delivering particular kinds of outputs.

The idea is intuitively attractive; if you give more resources to those who do more of what you want, then you encourage those who are less well-rewarded to conform more closely with your wishes, and hence you increase the total level of output whilst keeping your costs under control.

There has been an extensive debate about whether you can define as a policy-maker what you really want to achieve.  Policy-makers may like the idea of creating impact, but it is actually incredibly difficult to define in a sensible way.

It is a huge step from saying you fund research to create societal impact, and you can define ex ante what counts as good impact in ways that are robust enough to base funding decisions on. There has been much resistance from academics  who have argued that by setting the wrong targets, policy-makers are pushing the system in the wrong direction.

But our insight from the research project was slightly different, namely that the way that impact is generated creates the risk of serendipity and ???windfalls???, which in turn reduce the disciplining effect that the incentives have.  Setting targets as an efficiency measure only makes sense if the agent is responsible for the target.

If external factors dictate which actors produce outputs, then it is not the ???best??? or most efficient producers that are rewarded, but the lucky ones.  And if it is luck and not efficiency that is rewarded, then there is no incentive on actors to be efficient, and hence there is no pressure to improve the system.

It is almost like the apochryphal job application story where the HR person picks up half the application forms at random and throws them in the bin, ???because they wouldn???t want to employ anyone who was unlucky??????  They generate a sense that the underlying system is unfair, and hence  undermine its legitimacy.

This serendipity risk is particularly high in the case of arts and humanities research.  If you define impact in terms of audiences reached, then some subjects are more popular than others. In the UK, the most popular historical research is about WWII and Kings & Queens of England; whilst in the Netherlands, it is about the Golden Age and colonialism. 

Your chance of achieving impact as a historian depends far more on how closely your research fits with the interests of your publics and not with any particular efforts that you might make to engage users or to produce knowledge in ways that are accessible.

Likewise, media outlets have their own agendas about what they find interesting, and if you measure impact in terms of media coverage, then there is a risk of rewarding mediagenic outputs rather than the good management of research to maximise its impact given the underlying topic.

This issue is not unique to arts and humanities research, of course, it is just that the gatekeepers in this process are hidden.  Bozeman makes a clear distinction between the public value of a potential drug for society, and its private benefit to a drug company.

The public value to society lies in the potential of the new chemical entity to improve quality of life, and possibly even extend life.  The private benefit to a drug company depends on the opportunity the company has to make money from exploiting the product.

The private benefit is determined in part by legal regulations which determine how much safety testing is required, as well as whether the drug developer will have a monopoly exploitation period (via a patent).  So there may be occasions where drug developers choose not to develop a new drug because despite it being publically valuable, they cannot find a way to profitably exploit that.

If you really want to fund medical research that has the best public value, then you want to fund the  research that brings the greatest public, rather than private benefits. This means that counting patents is not a good way of understanding or indeed measuring the public value of biotechnology research.

A medical researcher may do an excellent piece of research, and put all reasonable effort into making sure that their good idea can be exploited by a third party.  But if that third party chooses not to exploit it, and you reward the researcher on the basis of exploitation, then the researcher is being punished for something for which the researcher is not responsible.

The question is whether you want to reward effort or outcome; rewarding outcome runs the risk of a Matthew principle, that a few universities that have a few lucky inventions receive the lion???s share of the funds.  Straightaway you lose your incentives for performance improvement because for universities outside the ???winner???s circle???, there is little to gain from responding to these incentives.

So the third main finding from our research was that any kind of performance measurement system needed to be based around researcher behaviour, and seeing the extent to which they had taken steps to allow others to use their research.  The ways you might do this vary with field, and it is important that a measurement system captures the effort.

Of course, the risk of performance management on the basis of behaviour is that it raises the risk of compliance behaviour.  And the other element of that is that if you want researchers to take public value creation seriously, then it has to be part of the identity of researchers, and compatible with ideas of doing good research.

Our research found that that creating impact was important to a range of researcher identities, albeit in two distinct ways (as core to research and as peripheral, depending on how it related to their knowledge creation processes).  The sensible message here appears to be for policy-makers to find ways to encourage these identity???formation processes for impactful researcher identities, and use their policies to support this rather than create a massive ???impact lottery???.

Paul Benneworth, Enschede, 6th February 2013.

The photo appears courtesy of Max (Art3fact).

Where have all the dilettante humanists gone?

Image001

 

One of the most curious arguments that we found in the discussions about the public value of arts & humanities research was the idea that humanities research was an amateurish hobby.  The argument was at the heart of Jonathan Bate’s book, the Public Value of the Humanities. Whilst Bate doesn’t necessarily subscribe to that view, he raises the spectre of the cynical civil servant seeking to understand public value saying ‘whilst I enjoy riding on my horse, I don’t expect the taxpayer to pay for it” (p. 7).

Of course this hobbyist idea of the humanities scholar is part of a series of wider discourses where the idea of humanities as a hobby is further nuanced.  The idea of humanities scholars being located in ‘ivory towers’ fits very neatly with an idea that they are following a hobby, more akin to alchemy than serious scientific disciplines.

It also has the advantage of creating a sense of urgency that something must be done: without strict management, the these amateur humanities scholars will simply indulge their indolent pursuits and waste public money.  What is needed is more direct control and management of those scholars, making sure that public support comes with the strings attached that forces them to create public value.

But if this were the case, then one would expect to find that humanities scholars openly identified with a version of their scholarship that embodied this amateurishness.  One might expect to find arguments being made that there was some kind of intrinsic justification for having a class of researchers who didn’t aspire to do anything useful, and that scholars bought en masse into this justification.

Even at first sight, it is quite hard to find evidence for this; Alan Hughes led a team at Cambridge who analysed data on all academics’ engagement activities.  They found that whilst many more arts & humanities academics described themselves as doing basic research than other subjects, and conversely less doing applied research, the proportions doing user-led basic research (the so-called Pasteur’s quadrant), were rather similar, at around one-third to one-quarter of respondents

Even if some humanities scholars are committed to basic research, that is not the same as claiming that they have no interest in the usability and use of their research.  Moreover, even if users were not widely involved in setting humanities’ scholars research agenda, similar proportions of humanities scholars to other disciplines were inspired by users and the real world in their research question selection.

* * * * * * *

Our own research focused in detail on the ways in which humanities scholars defined their identities as researchers, their definitions of excellent research, and the roles that were played by user engagement and impact in that excellence.  Our hypothesis was that if humanities was really a field of hobbyists, then you would expect to find the field closed and self-referential, with the only values that mattered defined internally.

What we did find was a variety of ways in which scholars linked beyond their own self-referential communities. At the most basic level was a widespread acceptance amongst academics that public funding implied some kind of duty to the public, and that had a real or material dimension. Even those involved in more abstruse areas like philosophy realised that part of their value came in their influence on ideas which ultimately interacted with the public.

There was acceptance that some kind of public engagement and interaction was necessary, and that took place in ways that influenced what was meant by excellent research.  There was a split between those for whom engagement was part of the research method, for example to public philosophers, and more theoretical philosophers whose work nevertheless was part of an ecology of public value.

The issue of public value was therefore more nuanced than more bald ivory tower positions might suggest.  There was a widespread resistance to being constrained by user problems but at the same time, a duty to be useful or relevant was an important part of academic identities.  That is to say that individuals found it important but related to it in a practical way: if forced to decide on what it was that mattered most, then it might be hard for academics to choose relevance over excellence, but conversely many scholars lived lives in which relevance was part of excellence.

* * * * * * *

The Norwegian research package took this further, and looked under the common identities to make distinctions within the different kinds of identities that researchers held.  The basic distinction they made between stable and liminal (peripheral) identities – those with stable identities felt that they were in control of demands for impacts, whilst those with peripheral identities felt challenged and threatened by increasing demands for public value.

Within the stable identities, two sub-identities were distinguished, between those that felt that valorisation and engagement was useful for their research efforts, and those that felt that valorisation and engagement were useful things for researchers to be doing.  The former group were often in the more engaged and applied disciplines, and saw user engagement as being a vital means of understanding and studying the research subject in its real context, whilst the latter group understood their place within a wider network of knowledge dissemination and engagement.

The unstable identities split between those academics who felt threatened and victimised by the increasing funder pressure for engagement/ valorisation.  Although they often made reference to abstract arguments (such as academic freedom or ideas of universities), in practice their own work did not engage that often with The other group were outsiders in policy communities that realised that humanities scholars were doing good research, and wanted to be able to communicate that value in their policy networks, but were exasperated by their inability to articulate it in ways that achieved traction in policy debates.

So these supposedly ivory tower academics are in fact united by a realisation of the importance of being useful but at the same time resisting instrumentalist policies.  But those that do cry out that all these changes are terrible are often those that are least engaged with the practices on which the changes are taking place.  The field of academic identity relating to excellence and relevance in the humanities can be segmented as the diagram below shows.

Image002

 

 

Paul Benneworth, Enschede, 5th February 2013.

Photo appears courtesy of Steve & Jemma Copley.

 

Arguing about arts & humanities research???s value: disagreement or debate?

 

Image001

The first of the major findings of our research project concerned the nature of the policy discussions around the public value of arts & humanities research.  Our initial hypothesis had been that the challenge for finding good indicators for measuring that value was bridging between two sides of an argument.

So in a conference paper at the 2011 CHER conference (see below), we started from the basic position that many attempts to measure that value had come to naught.  The problem seemed to be that there were two basic positions, and that there was no middle ground between them.

The first view was that value was seen as being ‘instrumental’, of things that could easily be measured and compared between research projects, spin-off companies, license deals, book sales and jobs created.  The second view was that the public valued ‘good science’ and wanted original research, that contributed to knowledge, and which was supported by funders.

Our initial view was that these two positions were not irreconcilable, and so if you wanted to come up with measures of value, you had to accommodate these two positions.  What it boiled down to was identifying what were the things that academics did that had external value, but which were still valued by the academics as being part of excellent research.

We had the idea of ‘dualistic intermediaries’ as being a class of activities that fulfilled this dual criteria, it was a valorisation outcome but it was also seen as being part of excellent research.  Our example of these dualistic intermediaries in this paper was seen as being students, that would have employability skills (external public value) but in the process of identity and skill formation would be involved in research outcomes that were themselves intrinsically valuable.

But the problem for us was that any dualistic intermediaries that you could think of were excluded by the various positions.  So if you want to measure research, then whilst academics  in arts & humanities research do often claim ‘students’ as a research outcome, to policy-makers students are seen as being a teaching outcome, and hence reject the idea.

And it is not just policy-makers who are at fault here: when policy-makers seemingly made concessions, and argue that media articles might be a sensible measure of public value, the Stefan Collini pops up to argue that that way lies hell in a tumbril cart (behind paywall, sadly).

The one thing the review of indicators did show was that there was no set of indicators that were seemingly acceptable to both policy-makers and academics.  Things that tracked what academics actually did were seen as being too idiosyncratic for policy purposes, whilst aggegrate measures were too remote from what academics actually did to reflect those academics’ efforts and successes in valorisation.

That problematic stayed us throughout the course of the empirical research: in all the countries we studied, we found two basic things, an agreement on a practical level that particular kinds of research clearly had value, and disagreement at an abstract level about the appropriate criteria for measuring that value.

One useful literature that we have already referred to was the idea of policy concepts, developed in a very different policy arena – territorial planning – by Erik Gløersen and Kai Böhme.  Their point was that policies can start to pursue rather vague goals that mean very different things to different people, for example in the field of territorial cohesion.

What emerges are policy communities which take these very different meanings and produce coherent rationalisations of what matters to them.  But these different positions are not logically reconcilable because they are abstractions of the way that ideas are used in practice.

If you are a European Commissioner, then ‘territorial cohesion’ carries the connotation of sharing the benefits of the single market, whilst if you are a spatial planner, then you want a definition of cohesion that you can measure in order to understand how it affects your territory.

Likewise, there are different communities in valorisation debates that are telling their own stories about the public value of humanities in the abstract because public value features in very different ways in what they are trying to achieve. 

Research funders have a version of public value that relates to justifying to often sceptical Finance Ministries and parliaments that they are spending money wisely to create public benefits.  Humanities scholars have a version of public value that relates to using their expertise to make informed statements in public arenas that contribution to the development of ideas in society.

We identified the various stories told by people active in these arguments, and identified that  there were three areas of fairly fundamental logical conflict between the versions of what matters for public value of arts & humanities research:

·         One contradiction emerged around the logic for public funding, between those who felt that funding’s existence proved that something mattered to society, as against those who argued that funding’s existence had to be justified in terms of a set of external targets and measures

·         A second contradiction emerged around the process of question selection in research, between those who felt that research should follow the most interesting questions unfettered by use considerations, and those who felt societies had to make choices, particularly in times of crisis.

·         A third contradiction related to the issue of comparing arts & humanities research to other fields, between those who felt that its essential humanism was traduced by measurement, and those that felt that an indicator framework that captured what mattered could benefit by showing arts & humanities research as contributing as much as other subjects.

The issue here is that these are discursive positions, that each have their own logics, and are not immediately amenable to a logical decision.  In a sense they are mutually compatible, but in the context of the argument, they were forced into a series of polar opposites. 

Policy-makers would argue it is not enough to say that arts funding is a sign of a good society, but that if it matters then it has to be measured.  That immediately precluded any chance to find a compromise between the different positions, or to use logic and deduction to decide what it was that mattered.

The only conclusion to draw here is that sol
ving this policy disagreement is necessary if you want to find good indicators of public value for the humanities.  There needs to be a proper debate, with give-and-take, and not simply a series of attempts to have the others’ arguments declared invalid. 

Of course, at the level of practice, research councils and academics can agree that particular bits of research matter, and are as useful as science, technology and engineering.  But this resolution does not take place at the abstract/ conceptual level, and that is an area to which we must look in order to understand how we might conceptualise and measure the public value of research.

Paul Benneworth, Enschede, 5th February 2013.

Photo appears courtesy of habeebee.