2 Evaluation of the Cultural Rucksack

2.1 Introduction

This chapter deals primarily with the evaluation of the Cultural Rucksack that was begun in autumn 2005, and with the evaluation report submitted to the Ministry of Culture in 2006. The evaluation was conducted in response to the decision of the Storting (Norwegian parliament) after consideration of the White Paper The Cultural Rucksack (Report No. 38 (2002–2003) to the Storting). The evaluation and the responses from the public consultation are presented, followed by the Ministry’s discussion of the points raised in these processes.

No major investigations of the Cultural Rucksack other than this evaluation have been initiated by the central government authorities. The Secretariat for the Cultural Rucksack has implemented two small studies of organisation models in Sandefjord Municipality and Møre og Romsdal County (Lidén, 2001 and 2004), and also an evaluation of the new system of direct transfer of funds to municipalities with more than 30 000 inhabitants (see Chapter 3.1) .

In 2007 NOK 0.5 million was allocated for an evaluation of the pilot project for the introduction of the Cultural Rucksack in upper secondary school.

There has been considerable interest at the regional level in discovering more about how the various parties have experienced the Cultural Rucksack. A number of counties have therefore commissioned external evaluations of their efforts to implement the programme (see section 2.4). Graduate and PhD theses have also been written on related subjects.

2.2 Evaluation

The task of evaluating the Cultural Rucksack was put out to competitive tender in autumn 2005. The terms of reference were as follows:

“The evaluation should provide an overview of how well the objectives of the Cultural Rucksack are being met and provide a sound basis for decisions on how the programme can be further developed. The evaluation should be suitable for use in developing and strengthening the quality and effectiveness of the programme. The evaluation should reveal any weaknesses in content, organisation, structure or other features, and identify the potential for improvement.”

In December 2005 the Ministry of Culture assigned the project to NIFU STEP, which conducts studies on education, research and innovation.

NIFU STEP was given a mandate to investigate the following questions:

• What forms of organisation have been chosen for the Cultural Rucksack by the various parties?

• What do the various parties do to evaluate the usefulness of the organisational solution they have chosen?

• How does the way in which the productions are acquired influence the quality of the content from the cultural and educational perspectives?

• What are the strengths and weakness of the various organisational forms, and what is the potential for improvement?

The methods used in the evaluation were Internet questionnaires, document studies, group interviews with teachers and pupils, and interviews with artists, presenters of cultural activities and other actors.

The Ministry of Culture and NIFU STEP then discussed the goals of the evaluation in more specific terms. The Ministry specified the terms of reference in more detail and added a question on the adaptation of the programme to pupils with special needs.

The report, Ekstraordinært eller selvfølgelig – Evaluering av Den kulturelle skolesekken i grunnskolen (“Extra-ordinary or self-evident – evaluation of the Cultural Rucksack in primary and lower secondary schools”), was submitted to the Ministry in September 2006. The Ministry conducted a public consultation in which the report was circulated to 123 bodies in the cultural and education sectors and to all municipalities, counties and county governors – a total of almost 600 bodies. The final date for responses was 1 February 2007.

The Ministry received responses from about 160 bodies: 41 municipalities, 16 counties, five county governors, 29 museums, Norges Museumsforbund (the confederation of Norwegian museums), 39 cultural institutions and organisations, seven universities and university colleges, four church organisations and two ministries. Responses were also received from individuals, organisations and societies involved in the Cultural Rucksack in various ways.

2.2.1 Summary of the evaluation report

The main conclusions and recommendations of the report indicate that the relationship between the cultural and the school sector with regard to the programme is complicated and marked by tension, and that this affects implementation of the programme. According to the report the tensions are related to the terms of reference, objectives, financing, organisation and structure of the programme, and to the content and concept of quality. The report also points out that the tensions are most marked at central government level, and decrease with each level in the system.

According to the report, the division of responsibility between the two sectors is the programme’s Achilles’ heel. Thus the Ministry of Culture is responsible for the allocation of Norsk Tipping funds and the development of the programme, while the Ministry of Education and Research is responsible for the arena in which the Cultural Rucksack is implemented.

The report stated that the authorities at central government level must clarify what the Cultural Rucksack is intended to be, what roles the various actors should play and what tasks they should be performing. More specifically, it must be decided whether the programme is a cultural policy instrument that is being offered to schools or whether the schools are to take part in the development of the programme, with the responsibilities and authority this implies. The report recommends that measures are taken to promote genuine cooperation between the two sectors and that the perspectives and wishes of the school sector should be given more weight. The report claims that the organisation of the programme is too complicated and that the system is poorly understood at every level.

The report contains a lengthy discussion on the differences between a monologue and a dialogue as a form of communication. In the case of the Cultural Rucksack, the two forms of communication largely reflect different opinions on the quality of the programme content. The report cites the definition of quality given in the previous White Paper on the Cultural Rucksack, which states that the quality of the programme can be judged on the quality of the pupils’ experience of the encounter with the artist or presenter of the activity. According to the report the parties disagree about what this means in practice.

The conditions for the financing of the Cultural Rucksack are also evaluated in the report. It concludes that since the reporting system is not standardised it is difficult to follow the flow of funds. It recommends that a simpler and clearer reporting system is established, but that this must not involve more bureaucracy.

2.2.2 Summary of the responses of the consultation bodies

To a large extent the consultation bodies commented on their own experience of the Cultural Rucksack and that of actors at the local level. The responses can be classified into the categories organisation and structure, cooperation, content, financing, and competence-building.

Organisation and structure

There was broad agreement that the current organisation of the programme facilitated constructive cooperation between the cultural and educational sectors. A majority of the consultation bodies expressed satisfaction with the current structure on the grounds that it facilitated local activity, ownership and influence, while at the same time ensuring that the activities were of a high quality. A large majority of the bodies advised that no major organisational changes should be made.

Like the evaluator, the consultation bodies considered that measures should be taken to make the actors more familiar with the system. It is important to ensure that all actors understand the objectives of the programme and the division of roles. The objectives outlined in the steering documents (particularly in the White Paper on the Cultural Rucksack and the White Paper on art and culture in schools ((Report No. 39 (2002–2003) to the Storting)) need to be specified more clearly.

Rikskonsertene stated that:

“The ministries should be more specific about the nature of the Cultural Rucksack, so that there is a common understanding of what the programme is intended to include. The potential value of the programme for the individual pupil must be the subject of continual discussion, and the conclusions used as a basis for future overall guidelines from the ministries. This would guarantee that the programme is not static but continually evolving in pace with new developments and the increase in general competence in the cultural and school sectors”.

Cooperation

With a few exceptions the consultation bodies agreed that cooperation between all administrative levels, and between the school and cultural sectors, was essential for the success of the programme.

Many of the counties and municipalities have formalised their cooperation, and they claimed that this functioned well in most cases. However, clearer terms of reference and division of work are definitely needed. It was pointed out that this would contribute to a common understanding of the intentions behind the Cultural Rucksack, which in turn would ensure continuity, high quality and diversity. Most of the consultation bodies agreed that the Cultural Rucksack should not be an ordinary cultural or school project. It is a unique, innovative programme that incorporates the perspectives, experience and competence of both the school and the cultural sector.

Sogn og Fjordane County wrote that:

“The Cultural Rucksack is one of the best cultural programmes for children and youth that has ever been established. Cooperation between the school and the cultural sector must be based on mutual respect and a clear division of roles. The parties have a joint interest in creating well-functioning models and developing the content of the Cultural Rucksack in cooperation. Both parties are equally important in their own way. It is in the everyday school arena that culture and the arts must flourish”.

Content

There was broad agreement on most aspects of the content and quality of the programme, even though special interests were also mentioned in many cases. Most of the consultation bodies stated that practical cooperation between the various actors was a fundamental condition for ensuring satisfactory content. The evaluation stressed that cooperation must not be based on conflict and that if the content is to benefit the pupils, it is important that both cultural and educational objectives are maintained.

The consultation bodies considered it constructive that professionals with difference experience, competence and interests worked together. However, they pointed out the importance of giving the individual professionals a share in the programme, which will foster a sense of ownership and thereby increase their motivation. This applies at all levels and to both sectors.

A number of consultation bodies pointed to the fact that the Cultural Rucksack is intended to be implemented by external professionals and to supplement the ordinary teaching of aesthetic subjects in the school. The concept of quality should, however, be discussed regularly. This is in accordance with the conclusions of the evaluation report, which states that the criteria for artistic and cultural quality cannot be either standardised or objective.

Nordnorsk kunstnersenter (North Norway artists’ centre) commented:

“In our experience the encounter between the artist and the children and their joint creativity results in genuine engagement. The aim is not primarily to be good at something, but to grow as a person and in relationships with other people”.

Financing

The responses indicated that most of the consultation bodies approved of the allocation key for the Norsk Tipping funds. However, several of them pointed out the need for a simpler reporting system, contractual standards, and a standardised system of remuneration, per diem allowances and transport. Several of them considered that the Norsk Tipping funds should also be used to cover administrative costs and competence-building, and some pointed out that the system of annual allotment of these funds made it difficult to plan for more than a year at a time.

Tromsø Municipality also wrote:

“Central government control of funds to the Cultural Rucksack should be kept to a minimum in order to maintain local ownership”.

Competence-building

A broad majority of the consultation bodies pointed to the need for a higher level of competence among those involved with the Cultural Rucksack. Thus a number of bodies stated that the schools needed to acquire greater knowledge of cultural matters and that the artists needed to know more about how to tailor their productions to a school audience. Many of the consultation bodies advocated using Norsk Tipping funds to cover competence-building measures, since lack of knowledge could compromise the quality of the programme. A number of bodies pointed to the need for more knowledge about forms of cooperation between the cultural and school sectors and the various county organisational models. It was pointed out that competence was spread among a large number of professional groups, and that no one institution or sector possessed the key to success. Thus there was a need for organised cooperation and an ongoing dialogue.

Volda University College stated that:

“Since the goal is that all pupils, whatever their abilities, should be able to appreciate and take part in cultural activities, it is important to facilitate competence-building, cooperation and guidance in connection with artistic production and the choice of presentation methods. Dialogue, cooperation and the exchange of knowledge and experience between key institutions in the Cultural Rucksack are in themselves a way of building competence, and add to the quality of the programme. Efforts to promote contact of this kind should definitely be intensified.”

2.3 Investigation of the system of direct transfer of funds to municipalities

In 2004 individual municipalities and cooperative groups of municipalities with more than 30 000 inhabitants were invited to take part in a trial scheme whereby Norsk Tipping funds were transferred to them directly. This meant that these municipalities would each receive their share of the Norsk Tipping funds and that they themselves would be responsible for designing a Cultural Rucksack programme for their schools (see also Chapter 5.4).

The municipalities of Bergen, Bodø, Karmøy and Lørenskog accepted the invitation and were allotted their share of the funds for the school years 2005–2006, 2006–2007 and 2007–2008. In their applications the municipalities had to present their plans for including the various forms of art and culture in their programmes. These had to include evidence of political involvement, use of resources, participation in networks and established cooperation between the cultural and school sectors. The main condition was that, like the counties, the individual municipality could show that it was in a position to offer its schools a range of art forms and cultural expressions of a high quality provided by professionals.

In April 2007 a questionnaire was sent to the four municipalities and the relevant counties. The questions concerned objectives, content, scope, organisation, financing and cooperation with other municipalities and with the county. The four municipalities vary considerably in size, and the survey showed that there were considerable differences in the way they had organised their programmes.

Bergen is the largest of the four. It has produced a catalogue of the activities available, with its own website. The schools have to take the initiative to sign up for each activity, but there are no minimum requirements regarding the number or the content of the activities offered to each pupil. Bergen reported that in the school year 2006–2007 all schools had signed up for one or more activities. The individual schools were followed up, but whether all the pupils at the school had participated was not investigated.

The other three municipalities have introduced a system whereby participation in the programme is obligatory for all pupils in the municipality. In Bodø all pupils in the same grade are offered the same activity, while in the other two municipalities the school may apply for other activities in addition to the obligatory programme.

All four municipalities were satisfied with the method of organisation they had chosen, and had not registered any, or very few weaknesses in it. However, the direct transfer system has certain disadvantages that should be noted.

Since Bergen is a city, it has a great many resources to draw on in the form of art and cultural institutions, and the municipality has a good quality assurance system. Great emphasis is placed on contemporary art and cultural diversity. On the other hand, it was found that the cultural institutions often drew on their own resources in their cultural presentations to the children, even though many of them are also responsible for their region as a whole. The county stressed that this was a problem.

From the interviews and feedback given by the various bodies, the Ministry gained the impression that in smaller municipalities responsibility for the day-to-day administration of the programme often devolved on a few enthusiastic individuals. Enthusiasm and zeal are strengths, but relying only on these individuals makes the programme vulnerable. It was also found to be difficult to quality assure the content of the programme in these municipalities, and an alternative solution would be to introduce inter-municipal cooperation on the Cultural Rucksack.

2.4 Evaluation of the Cultural Rucksack in the counties

Several counties have commissioned research-based evaluations of the Cultural Rucksack.

In 2002 the Telemark Research Institute conducted an evaluation of the establishment and expansion of the Cultural Rucksack in Buskerud County (Åsne Widskjold Haugsevje, 2002). The evaluation showed that the programme offered pupils a greater number and a better quality of artistic and cultural activities than they would otherwise have had, and that the professional standard of the productions had improved the schools’ own efforts to provide cultural activities for their pupils. The evaluation emphasised that the county considered it important to integrate art and cultural appreciation into the teaching, and stressed that good teaching materials, preparation and follow-up were essential in achieving this. The evaluation revealed that the system of appointing cultural coordinators at the schools had functioned well, and pointed to the importance of setting aside time and resources for this work.

In 2002-2003 the Telemark Research Institute also evaluated Turnéorganisasjonen for Hedmark (Hedmark Touring Organisation) (Sigrid Røyseng and Ellen Aslaksen, 2003). The report concluded that it was important to strengthen the contact between the organisation and the schools. It recommended establishing more arenas and communication channels for dialogue, including how and how far to integrate the Cultural Rucksack into the ordinary school day. This means that schools will have to set aside resources for this work, for example through the cultural coordinators, and the organisation will have to approach the schools more often and devote more time to dialogue. The report also pointed to the need to clarify roles and expectations between the various actors in the cultural field.

In 2003 the research foundation Østfoldforskning conducted an evaluation of Turnéorganisasjonen i Østfold, which is responsible for the Cultural Rucksack in Østfold County (Anne Rønning, 2003). It was concluded that the main objectives of the programme had been fulfilled, but that in certain fields more focused efforts were needed. Information to the schools needed to be improved and the efforts to involve the school sector needed to be intensified. Roles, responsibilities and the issue of resources should be clarified.

In autumn 2006 a user survey of the Cultural Rucksack in Akershus County was conducted by the Faculty of Art, Design and Drama of Oslo University College (Line Prøis Kristiansen, 2007). The conclusion was that the cultural institution “kultur.akershus” has fulfilled the objectives of the Cultural Rucksack to a large extent. However, the implementation of the programme depended on whether the individual municipality set aside resources for a municipal cultural coordinator and cultural coordinators at the schools. Furthermore, the benefits to the pupils of the professional productions depended on the attitudes and interests of the school and the teachers. It was pointed out that schools and teachers should be made more aware of the kinds of roles they can play in the Cultural Rucksack.

2.5 The Ministry’s assessment of the evaluation and the responses of the consultation bodies

The primary aim of the evaluation performed by NIFU STEP was to provide a sound basis on which the Ministry could further develop the Cultural Rucksack. There was considerable variation between the consultation bodies on the accuracy of the report on certain points. The experiences of the various actors involved seemed to depend on their place in the system and their geographical location. This underlines the complexity and diversity of the programme, which is a complicating factor in the Ministry’s assessment.

Several of the consultation bodies emphasised the importance of the programme. For example, Rikskonsertene wrote that:

“The Cultural Rucksack is a unique programme, and nothing like it exists in any other country. This is something we can all be proud of, especially the politicians in the Storting!”

Some of the museums commented that the Cultural Rucksack had inspired them to think in new ways (the Museum of Science and Technology, the Eidsvoll 1814 museum and Aust-Agder kulturhistorisk senter), and others that they had been inspired to intensify their cooperation on putting together a cultural programme. Many of the municipalities were very pleased with the programme organised by the county.

The Cultural Rucksack is an ambitious cultural and school policy project. Its implementation depends on cooperation between many different bodies in the cultural and school sectors at the national, regional and local levels. Its implementation requires a continual process of finding new solutions and arenas for cooperation across cultural fields and between the various administrative levels.

One of the main principles on which the programme is based is that local and regional bodies are free to develop their own “rucksacks” adapted to local conditions. The intention was that there should be as little centralised control as possible. As indicated by the evaluation and the responses of the consultation bodies, this principle has resulted in a situation where there is not one single cultural rucksack, but many different ones. On the one hand this is a positive development, since it gives local and regional actors freedom of action and provides opportunities for local engagement and creativity. On the other it makes it more difficult to establish a centralised tool for maintaining an overview of how the programme functions in all parts of the country.

All in all, the Ministry considers that the evaluation report has paid more attention to tensions and differences than on finding out how the programme functions in practice. The Ministry considers that the empirical data given in the report are too weak, and that the conclusions alone do not provide a sound enough basis for decisions on the further development of the programme. Furthermore, in some cases the conclusions differ from the responses of the consultation bodies.

The consultation responses and the regional evaluations have given the Ministry a more positive impression of the functioning of the Cultural Rucksack. The programme has resulted in a more systematic and carefully planned presentation of culture to schoolchildren. A larger number of cultural activities are available and the quality is higher, and in general more attention is paid to quality and methods of presentation. The schools now have a resource for making children acquainted with different art forms and thus developing their appreciation of a broad range of cultural expressions. The programme also has a democratic aspect and promotes social cohesion, both of which are cultural and educational policy objectives.

The years up to 2006 were a period of expansion and development. The Ministry considers that the implementation of the programme has proved to be successful in most municipalities, and that working with the Cultural Rucksack is experienced as meaningful and instructive for the parties involved. The programme now has a significant place in children’s education, in schools and in cultural life. In many places it has stimulated the professional presentation of art and culture, and revitalised the role of culture as a factor in regional development. The Ministry regards this as a very positive result.

On the basis of the consultation responses the Ministry does not consider that the evaluation report has provided sufficient grounds for making any major or comprehensive changes in the organisation of the Cultural Rucksack. Major changes at this stage would probably do more harm than good.

The Ministry realises that cooperation between the two sectors can be challenging, but considers that the position of the programme in the interface between education and culture is also a strength. With a clear division of roles, and close dialogue and cooperation, the programme should prove to be a resource for the various parties. In this way the Cultural Rucksack will be an out-of-the-ordinary, but at the same time self-evident, supplement to other school activities. The cultural sector should ensure that the programme offers high-quality, professional productions. The school sector should facilitate the incorporation of these activities into the school day, set aside time for preparation and follow-up, and adapt the teaching to the programme content. The school and the pupils should have the opportunity to give systematic feedback after each presentation or project.

It will of course be necessary to address many of the problems highlighted in the evaluation report and the consultation responses. The Ministry realises that the objectives, principles and division of roles must be clarified. A more coordinated reporting system is needed. The further development of the programme should include competence-building on a more systematic basis and a continuous research process. The school sector should be more closely involved in the planning, development and implementation of the programme at the local and regional levels.