Kristin Clemet, Norwegian Minister of Education and Research

Lecture in connection with the Norwegian Broadcasting Corporation’s celebration of the
P2-channel’s tenth anniversary, October 2003.

Mantra or meaning - Quality in education and research

It is an honour to be given P2’s most important rostrum in connection with the Norwegian Broadcasting Corporation’s celebration of the channel’s tenth anniversary.

The title of this lecture is “Mantra or meaning – quality in education and research”. The topic has been chosen for several reasons:

• The word quality is a key concept in public debate. It is associated with everything from care of the elderly, public transport and running hospitals to education and research.
• The debate on quality is important, in part because it concerns democratic processes and the demands, wishes and expectations of the general public.
• At the same time, the word quality can be misused by repeating it endlessly with no defined content – almost like a mantra.
• The term is also complex: it encompasses very different elements depending on the topic under discussion, and not least on those who are holding the discussion.

In this lecture I will talk about the general concept of quality, but first and foremost about quality in research and education. I will dwell upon

• why it is important to discuss quality in education and research

• how the concept of quality can be defined and given content
• how we can assess quality

Finally I will attempt to debunk some common myths about quality in research and education.

Why should we focus on quality in research and education?

Most people will advocate a high level of knowledge and will probably favour all proposals for improving Norwegian research and education. But one of the problems with the so-called quality debate in Norway is that it often ends up as a discussion on resources or quantity. It is easy for such debates to attract attention, but if we are already using considerable resources it will be at least equally interesting to find out what we are actually getting from our input.

A prerequisite for making improvements is knowing where we currently stand. We must identify our weak points if we are to do something about them, and our strong points if we are to consolidate them. So what do we know about the strengths and weaknesses of the Norwegian knowledge system?

As for our general level of knowledge we are in a particularly good position. International surveys show that our population is among the best educated in the whole of the OECD area. More specifically we can state that:

• We have an inclusive school system where most pupils enjoy their education. We have a high teacher/pupil ratio, and we are ranked among the best with regard to the availability and use of ICT in secondary school education.

• The capacity within higher education has more than doubled in the last 20 years, and international evaluations have shown that university teaching maintains a high academic level.
• Norwegian research does well in several fields, for example climate research, geology, brain research, mathematics, ICT and social sciences. Norwegian doctorate theses are also regarded as being of a high international standard.

All these are strong points that we can be proud of. And presumably most people will agree that there are many examples of high quality within basic schooling, higher education and research.

Of course we could decide to rest on our laurels, but we would then choose to ignore some important challenges. Let me mention some of them:

• Too many Norwegian pupils lack skills in reading, arithmetic and general science. We are among those countries that have the greatest differences between less able and able pupils. A total of 20 per cent of Norwegian ten-year-olds lack elementary reading skills.
• It has been shown that in higher education Norwegian students take a relatively long time to complete their studies and that many of them discontinue this education.

• One of the factors foreign expert groups have pointed out in their evaluation of Norwegian research environments is that efforts are fragmented. There are too many small research groups, and there is a need for stronger scientific leadership along with improved collaboration and distribution of work.

In addition we must realise that the world does not stand still. We must continue our development simply to maintain our strong points, and advance even more if our ambition is to improve.

What is quality?

I have already mentioned several characteristics that are connected with quality in research and education. But we also need a wider awareness of what we really understand by the term quality.

It has been claimed that quality is "the most complex, multi-dimensional concept that has ever been reduced to seven letters". Others have said that "quality is impossible to define, but we recognise it when we see it". When I asked my children today, they said that quality refers to something that is "proper" or "better" and that it has "something to do with safety".

So quality is far from an absolute and unambiguous concept. Basically the word refers to the idea of disposition, nature and characteristics – in other words it must tell us about the goodness of something. But we must first know which characteristics we want to evaluate, and then in relation to what these characteristics are to be assessed. In other words quality is a relative term, and an empty term until it is given content.

Many attempts have been made to find more precise definitions of the term.

• For some, quality will be linked to the unique and outstanding, for example in the awarding of the Nobel prizes. This is a very exclusive understanding of the term, both because the quality designation is reserved for the chosen few and because the quality judgement requires great expertise.

• At the other end of the scale we find cases where quality is connected with the ability to meet requirements. These may often be minimum requirements or the requirement that the product or the process must not contain errors. Quality is thus not something unique but – on the contrary – a feature that is expected as a minimum. For example, the international standardising organisation ISO defines quality as the ability to meet established requirements.

Between these two extremes there are a number of other interpretations of the concept of quality. The term can also be connected with the ability to utilise resources in the best way possible, the ability to improve results in relation to a given level, or the ability to adapt products or services to new conditions.

In many areas of society it will be natural to emphasise one of these definitions. Within public transport for instance, quality will to a large extent be associated with the absence of deviations from the norm. The person waiting for the bus is more interested in the journey taking place without delays or accidents than the method of transport being unique.

In the field of research and education it is more difficult to assign priority to one interpretation of the quality concept. In these areas we have in fact extremely complex expectations:

Quality in research

Let me start with research. In this area we have comparatively fixed criteria for what is good research. However, over time various criteria have assumed key importance.

Galileo Galilei’s discovery of Jupiter’s moons was an enormous scientific breakthrough. The moons that circled this large planet provided clear evidence that our own world could not be the centre of the universe as the Church claimed. Galileo’s assertion was regarded as heresy and, as we know, the story ended with Galileo being summoned before the Inquisition and forced to publicly recant his earlier statement. Here the quality of research was measured on the basis of the prevailing religious dogma, and with that as a standard Galileo’s scientific breakthrough did not qualify for recognition. More than 350 years later, in 1992, the Catholic Church made an official statement to the effect that Galileo had been wrongly judged. New criteria had replaced the old.

Current research usually operates with three criteria of quality:

The first is originality, demonstrated when research contributes new knowledge that usurps old truths and forces us to think afresh. Many people also use the expression "pioneering research".

The second is solidity, which refers to how research is carried out, for example whether the results can be verified.

The third characteristic is relevance or the extent to which the research is interesting or useful to others. “Others” can be both research colleagues and users in trade and industry or society at large.

Naturally enough, researchers assign different priority to these three criteria. In some cases the criteria can also be directly opposed to each other. For example we can say intuitively that there is a direct contradiction between making revolutionary scientific breakthroughs and keeping to time-honoured theories.

When Charles Darwin published The Origin of Species, the book was received with great interest and the first edition quickly sold out. But that did not stop it being severely criticised, and many ridiculed the hypotheses he presented. In this case then, originality clearly contradicted what was regarded as solidity and scientific relevance at that time.

A survey conducted some years ago reveals that most Norwegian researchers give priority to originality as the most important criterion for good research. This is not unexpected. Much of the point of research is that it tells us something we didn’t know before.

We should therefore perhaps believe that research of a really high quality originates in individualistic systems where each isolated and independent researcher can challenge established theories and methods. But this is not necessarily the case. The Hungarian scientist and philosopher Michael Polanyi was among the first who noted the relationship between originality, solidity and relevance in research. One of his main points was that scientific progress is not achieved by exclusively cultivating the original but by cultivating the tension between originality on the one hand and relevance and solidity on the other. According to Polanyi it is the researcher’s contact with established methods and theories that provides favourable conditions for rebelling against these same theories and methods.

This can largely be confirmed by the studies that have been carried out to find out what characterises good research environments. One interesting common feature of the best environments is that they bear the stamp of extensive communication and collaboration, both inside the environments themselves and externally. Particularly important of course is cooperation across national borders and contact with the front line of international research.

Good research environments also appear to bear the hallmark of diversity with regard to both the age and background of the researchers and the mix of professional activities within the research unit - in the form of specialisation, teaching, dissemination and contact with trade and industry. I would like to emphasise that it is not diversity as such that promotes quality but the environment’s ability to utilise diversity.

This tells us that it is extremely difficult, and not even desirable, to select one individual criterion for quality in research.

Quality in education

In the education system it is even more difficult to find one quality concept that everyone can agree on. We know that most of today’s children and young people in Norway will complete 13 years of schooling, and it is estimated that 65 per cent will take some form of higher education. The entire education system is thus a system that embraces the majority of children and young people today. It goes without saying that a system of this kind must have broad objectives and must therefore also operate with a broad concept of quality.

Several attempts have been made to find criteria to define quality in both higher education and basic schooling. Most of these produce a long list of criteria, but there are three particular perspectives that seem to recur.

• In the first place reference is made to the external frameworks for teaching such as resources – for example allocations, number of teachers, premises and equipment.
• The second criterion emphasises the actual learning process and includes such factors as the content of the teaching, how the teaching is organised and the methods that are used.
• The third quality criterion is often called quality of results. Briefly, this refers to the benefit the pupil or student gains from the education, not only in the form of grades but also as overall benefits of learning.

As with research, it would be inappropriate to choose one of these criteria and claim that it provides the actual definition of quality in schools. A unilateral focus on the frameworks for teaching can for example easily hide the fact that the learning benefits are not in proportion to the use of resources.

And even though many will argue that the learning benefits represent the primary goal, we could also imagine a situation where the academic level is high because the pupils are "whipped on" to achieve good results – in other words using methods we cannot accept. In such a case it is important to have a quality concept that is wide enough to allow us to ask whether the learning process is of good quality.

In higher education we have moved relatively quickly from a situation characterised by higher education for the few to one offering higher education for most. Norwegian universities and colleges of today are therefore faced with totally different challenges and expectations than they were some decades ago. The purpose of the ongoing Quality reform in higher education is to get more students to complete their studies while at the same time raising the academic level.

Some critical voices have nonetheless claimed that it is more a quantity reform than a quality reform, partly because the reform entails emphasis being placed on the number of candidates and examinations taken. However, in my opinion it is not a sign of quality when a large number of students fail or discontinue their studies after a very short period of time.

Of course we can say that the quality is excellent and that there is something wrong with the students. But I prefer to ask some critical questions, for instance about the quality of the study programmes offered to the students and about the educational institutions’ incentives to offer quality.

Higher education gives no guarantee of personal growth, but it offers the student an opportunity to develop cultural awareness and the ability to reflect as well as attaining academic qualifications. The Quality reform sets greater demands to specialisation, commitment, making choices and consistent effort, giving and receiving guidance, and to looking beyond and across national and traditional academic boundaries. Rather than reducing the possibility of overall development, it provides the opportunity for greater fulfilment.

How can we recognise and assess quality?

Measuring quality is a theme many find particularly difficult. The crucial point is of course what is to be assessed in each case. Most people are well acquainted with product evaluation, and we accept that there can be large differences in the quality of everything from cars and clothes to cheese and wine.

Quality is also measured in parts of the public sector: for example in the public health service we have over time come to general agreement about the criteria hospitals are to be measured by.

But there is more disagreement about the evaluation of the quality of schools – at least so far. It is claimed that the school is such a complex organisation that a single criterion such as academic results cannot give a satisfactory picture of the actual situation.

So before talking about measuring quality, we should ask ourselves why such an exercise is to be carried out. The point is not simply to measure for the sake of measuring.

From my viewpoint the real reason lies in our desire to improve one area or another. And if we are to improve, we must know where we stand at present. "Where does the shoe pinch?". "Where must we work more or in a different way?" etc. To answer these questions we need information that is understandable and is available to those who will use it. In some cases such information can be converted into indicators that enable us to draw comparisons, for example how something has developed over time.

So the question is whether such indicators can give information about quality in education and research. In my opinion the answer is yes. But I will also emphasise that although an indicator says something about quality, it doesn’t say everything.

I also think that there are a number of characteristics and values that neither can nor should be measured. Concern, kindness, sense of humour – these are all examples of important qualities but in my opinion such qualities should not be measured or graded.

A key task is therefore to identify the variables that both can and should be measured and that may prove to be useful indicators. I think it is appropriate and right to conduct an open and thorough debate on the quality criteria to be used in schools. Quality criteria are far from exact. They can vary over time, for example depending on the challenges we are facing. Moreover, children are involved, and this intensifies the requirements for choice of methodology. In this case protection of privacy, human considerations and ethical deliberations must play a crucial role.

This represents an obvious challenge. But instead of abstaining from measuring anything because the process is too complicated, we should make efforts to acquire information that can give us knowledge about the school we send our children to. The object is not to exercise control but to acquire a management tool for schools, local authorities and all those who are working to improve schools.

During spring 2004 we will set up a special website called the School Portal. Among other things this will contain information on pupils’ learning environment, the benefits they gain from learning, and the resource situation at each school. At the same time national tests will be introduced in reading, writing, English and mathematics. The purpose is to give pupils, teachers and school-owners information that can be used to make improvements. This is not a specific Norwegian innovation – on the contrary the scheme has already been introduced in several other countries.

The national quality evaluation system will not allow a one-dimensional ranking of schools. Schools conduct their activities from very different platforms, which makes it difficult to make direct comparisons without correcting for these varying conditions.

It is therefore at least as important that each school is given the chance to measure its own progress in the various fields, or the opposite: to find out where it will subsequently be necessary to use extra resources, whether this applies to the efforts of the schools or of the local authorities. Comparisons with others can nonetheless be of some interest as we can learn from each other. Most schools have both strengths and weaknesses. Consequently most schools have something to learn from others and something to teach others.

We are also introducing systematic quality evaluations in higher education. Such evaluation systems were established in most western countries as early as the 1980s, but in Norway this work did not start properly until the late 1990s. We have recently set up our own national quality assurance body – the Norwegian Agency for Quality Assurance in Education – and are now more in line with our neighbour countries.

A different set of rules applies to research. When Alfred Nobel wrote his famous will, he chose criteria without any particular considerations other than his own priorities. His most important condition was that the Nobel prizes should go to those “… who, during the preceding year, shall have conferred the greatest benefit on mankind”. The limitation regarding the preceding year was later removed as in many cases it is only in the future that it is possible to see what can really be regarded as scientific breakthroughs.

Nobel was free to choose criteria, and in addition he defined an extremely exclusive target group. In cases where we are to evaluate research quality on a broad front, the criteria must be chosen with great care.

I previously mentioned three features that characterise quality in research: originality, solidity and relevance. Indicators have been developed that take these characteristics into account, as well as saying something about productivity and progress. This can for example be the number of publications or how often a researcher is quoted by other researchers. Such indicators are useful tools, but all tools have their weaknesses.

A standard tool for recognising and acknowledging quality is peer review. This means that other competent researchers evaluate the results before they are published in a recognised journal. But this method also has its weaknesses. In some cases the method can preserve established facts because it may under-evaluate original contributions. It is therefore important to apply a whole set of indicators to attain as comprehensive a picture of reality as possible.

It is also important to listen to critical voices. In early eighteenth century the Norwegian dramatist and philosopher Ludvig Holberg pointed out that the number of publications in itself is not always an expression of quality. In an essay on university teaching he maintains that a country’s honour does not depend on many books being written but that few and good books are written. Having said that, there is nonetheless wide agreement that the best researchers are often also the most productive.

Debunking some myths in the concept of quality

So far I have tried to show that research and education are areas where we have such complex expectations that a wide concept of quality is needed. At the same time I have tried to demonstrate that it is both meaningful and possible to measure quality in research and education on condition that we measure along several axes.

General agreement has been reached over time on these main points. However I am of the opinion that there is a need to debunk or dislodge some of the myths that recur in the quality debate:

Firstly there seems to be a widespread perception that investment in quality in one area will necessarily be made at the expense of other considerations. A typical example from schools is the notion that more focus on academic performance will be detrimental to pupils’ well-being and self-image. But most studies show that there is no antagonism between personal well-being and academic performance. I also disagree with those who see reading, writing and arithmetic skills as merely “hard” utilitarian subjects that are almost opposed to the development of the whole person and quality of life. The basic subjects are useful, but they are also a fundamental condition for enjoyment and for the fulfilment of one’s dreams, hopes and talents.

Another example is some researchers’ opinion that we cannot aim for top performance without reducing scope. Experience shows the opposite. Special backing for the best and most promising researchers has proved to produce positive effects far beyond the selected environments.

Another myth I would like to challenge is that we can abstain from measuring quality in research and education. In theory we are of course free to abolish all quality evaluations. But in practice this would have enormous consequences. Barriers between different countries’ education systems are being removed in the sphere of higher education. Comparable quality requirements represent a key objective and instrument in this process, for instance in increasing mobility across national borders. It would be difficult and very inopportune for Norway to abstain from this development. Institutions need quality evaluations to avoid working in the dark.

There is also an ongoing debate on openness about quality and measuring quality. In my opinion knowledge and openness are the basic building blocks of democracy – and a prerequisite for progress. The knowledge system must accept that society’s requirements regarding access have been intensified. People have quite simply become more interested in knowing what type of teaching is being given in schools, what the quality of the studies at universities is like, and what the allocations to research actually result in. We can choose to see this as a problem, or we can see it as part of the process of democratisation.

In my opinion it is also a myth that openness about quality only helps to create losers. Many have expressed their criticism of making quality evaluations of schools available to the general public for exactly that reason. There may of course be grounds for this if the measurements are one-sided or deficient. But even given broad quality evaluations, there are many who are in principle against such openness. I am not afraid of it. I believe that openness promotes curiosity, debate, dialogue and demands, and I think that is positive! My conviction is that such openness will inspire an increase in quality in the entire education and research system.