Reflections on Evaluation
On 5 November Earthen Lamp’s Senior Research Executive Maike Ludley attended an online debate hosted by the Centre for Cultural Value at University of Leeds. The topic was “Who and what is evaluation for? And when is it useful?” and was part of the Centre’s What Value Culture? programme. The session attracted an international online audience with 200 participants taking part.
The Centre for Cultural Value is working towards developing principles for good evaluation in the cultural sector focusing on the philosophical and practical aspects about what to consider when carrying out evaluation. This event was one of four taking place to feed into the development of these principles.
The event opened with three-minute presentations from the panelists (representing the artist, funder, evaluator and cultural organisation perspectives) followed by wider discussion. The second part of the event consisted of a short breakout session in which participants discussed:
• Who and what is evaluation for?
• What should we do to make it more useful?
Here are the main points from the session that stood out for Maike:
Why do we evaluate? For the funders or for organisational learning?
Everyone agreed that evaluation should be useful to the organisation, but at the same time there is awareness that sometimes evaluation can become a box ticking exercise.
Rationality and objectivity
When considering methodologies does evaluation always have to follow rational approaches or is there room for more creative ways of capturing impact? Are funders open to accepting other approaches to evaluation framework and does it fit into the DCMS framework?
The role of the Evaluator(s)
It was noted that often external evaluators are brought in late to a project, and it would be more beneficial to be involved from the start. It is advantageous for the Evaluator(s) to have more than one point of contact in the cultural organisation to ensure the right levels of access and being embedded in the process doesn’t have to mean that the Evaluator loses their objectivity. Within an organisation, evaluation can carry a negative stigma compared to more creative work.
“Evaluation” – a dirty word
There was a discussion about the need for a new word to describe the process, as the term “evaluation” can have negative connotations for some people. What could be a better term?
What data is collected? What is it good for?
Evaluation should fit the organisation and the creative output, as well as the audience. There was discussion around making evaluation and the collected data useful to everyone who is involved in the process. The was emphasis on being clear about what are we counting and why it will contribute towards a better overall evaluation
Scared of failure
Cultural organisations often feel they cannot say when something went “wrong” or just different to the way it was initially planned. This can lead to reports being amended and the relationship with the external evaluator can turn sour. The panelist offering the Funder’s perspective said he actually appreciates honesty and in no way would being open about things that didn’t go to plan affect a future application negatively. It would be beneficial for evaluation reports to have a “learning” and “reflection” section, which would be useful to both the cultural organisation and the funder.
Overall there has to be more open and honest communication about expectations, but there was hope that events like this can lead to positive change in that regard! The Centre for Culture Value hosted a discussion that was both fruitful and productive. Lots of the points resonated with the Earthen Lamp team and the work we do and it’s great to see that so many people are interested in the topic and would like to see further developments in cultural evaluation processes soon.