10 Quality and learning outcome

In the Commission’s opinion it is important that MOOCs being developed in Norway maintain a high level of quality. In the development of MOOCs, there is much to learn from the systematic quality work performed in connection with the development of flexible education and web-based programmes.

The Commission also believes that there are some new, specific challenges and opportunities involved in assuring the quality of MOOCs.

10.1 Quality in web-based higher education

10.1.1 NOKUT’s quality assurance work

In recent years, NOKUT has worked with quality assurance of web-based higher education. This relates particularly to applications for accreditation of web-based higher education. In 2013, NOKUT revised its guidelines for accreditation applications, and clarified the requirements related to quality.1 The following will give a description of these requirements.

Learning objectives, content and structure

The Bologna process adopted a pan-European Qualifications Framework for higher education, cf. Chapter 9. The qualifications achieved by a candidate upon completing a study, shall be described as knowledge, skills and competence.2 The descriptions are an important basis for assessing the quality of a web-based study. The syllabi’s descriptions of the learning outcomes will also be essential to the online student's preparations and expectations, which is also important for quality. Furthermore, NOKUT requests an account of the contents and structure of the study, including its academic scope, depth and context. It should also be clear what parts of the study that support development of the qualifications described in the learning outcome.

Work scope

NOKUT requires an account of the work scope in a web-based programme, including an overview of how many hours of teaching and guidance are facilitated. In addition, a description of the anticipated extent of self-study and exam preparations must be available. The extent of guidance and follow-up from a lecturer is particularly important to online students.

Types of education and the educational rationales

The work and education forms in a web-based programme could comprise a number of educational instruments and use of a wide spectrum of technologies. The work and education forms must, like in other educations, be selected on the basis of what the students are going to learn.3 This means that the descriptions of learning outcomes will form the basis for the educational reasoning relating to the choice of web-based teaching forms and instruments. NOKUT requires a reflection on the choice of technological tools, and the choices shall be substantiated based on educational and academic considerations, what the students are going to learn and what kind of students are in the target group for recruitment. Facilitating communication and collaboration is also an important part of the educational facilitation of a web-based programme, where the subject content and the descriptions of learning outcomes must be the decisive factors.4

The subject contents must be facilitated for the web in digital learning resources that are appropriate for the students’ subjects. Selection, production and adaptation of the subject content, as well as the educational and technological methods for presenting it, will determine the quality along with the choice of digital learning platform.5

Digital skills

NOKUT emphasises that the composition of the academic environment, its size and overall competence shall be aligned to the study content as described in the study programme. At the same time, the academic environment must be large and robust enough to safeguard research and professional development work. In addition to overall educational competence, the lecturers must also have satisfactory digital skills.6 This means knowledge and experience of online education, and skills related to use of relevant technological tools. This applies not only to lecturers, but to all employees involved.7 The size of the academic staff is also significant. It must be ensured that there are enough available lecturers to teach and instruct the online students, and ensure that the lecturers are given enough time to follow up the students.8 The students must also receive training in online study methods and use of digital tools.

Student recruitment

How students are recruited and how admissions are carried out is vitally important to the facilitation of a good online study environment. In web-based studies it is important for the study environment, study groups and other subject matter activities that require collaboration, to ensure that the students have approximately the same academic basis and progression. NOKUT therefore emphasises that the institutions must ensure that the recruitment of students strengthens and promotes a good study environment.9

10.1.2 Quality norms for web-based education

In addition to NOKUT, there are several other players in Norway working on quality in web-based education, e.g. Flexible Education Norway and Norway Opening Universities. Both institutions have cooperated with NOKUT and provided input to the work on developing requirements relating to assessment of web-based studies.

Lessons learned from flexible education

Flexible Education Norway (FuN) is a national member organisation for institutions that offer flexible education in the form of online studies, online studies in combination with workshops and other flexible adaptations.10 FuN has prepared quality norms for web-based education.11 These norms were last revised in 2011, and constitute a set of quality standards that FuN believes are reasonable to expect from providers of web-based education. The norms are detailed and grouped into the following principal elements: quality management and quality work, programme development, information and instruction and programme execution. The objective of the norms is to help stimulate quality in the education programmes, in parallel with and pursuant to the quality assurance work carried out by NOKUT.

Lessons learned from the work of Norway Opening Universities

Ownership in the institution, resource use and the “hybrid expertise”

Norway Opening Universities (NUV) is an administrative body under the Ministry of Education and Research. NUV shall contribute to promoting the development of ICT-supported learning and flexible education in higher education, and collaboration between higher education institutions and working life. Norway Opening Universities e.g. funds development projects at universities and university colleges related to development and use of technology for learning and flexible education.12 In the same way as NOKUT, Norway Opening Universities is focusing on learning objectives, choice of instruments, educational rationales and digital skills. Norway Opening Universities is concerned with the significance of strategy, management focus and organisation as regards quality. Studies from Norway Opening Universities show that it is important for the quality of web-based education that the activities are owned and supported by the institutions’ administrations, and that they are integrated into the strategies and plans of the institution. The management's focus on, follow-up of and will to pursue this kind of education is of vital importance.13

It is also important for quality that the development work and implementation of the education programmes have a dedicated staff with diverse but complementary competence. The academic environment should consist of academic employees, administrative personnel and representatives from the IT department or employees with other technological or digital support skills. Broad participation could contribute to a better understanding of what the technology could provide, as well as the educational and academic challenges related to the development of a web-based study programme. Expertise in the intersection between pedagogy, administration and technology is referred to as “hybrid expertise” in certain contexts, and is described as essential in the development and implementation of flexible education.14

Evaluation and documentation, transparency and sharing

Norway Opening Universities believes that it is important to document and evaluate the processes under way in the educational innovation efforts, and in the development and implementation of flexible studies. The opportunities for reflection on lessons learned and the development of own knowledge on education and development work may contribute to a quality culture and better education quality.15

Transparency and sharing may promote quality. Use of technology in education may contribute to a transparency that can benefit students, academic employees and the institutions. Access opens up education for the outside world. It has to bear comparison, it must be professional and updated, This sharpens the quality mindset.16

Norway Opening Universities' expert group on quality

Over the last three years, Norway Opening Universities has had an expert group on quality in ICT-supported and flexible education.17 The expert group’s work will, together with other work carried out in this field, form the basis for developing a set of criteria for work on quality in web-based education. The basis for the work is that national stakeholders, higher education institutions, employees and students will contribute to ensuring quality in flexible education. The work in the expert group will be continued in 2014 with focus on detailing and specifying the criteria, as well as on making the criteria and basic data for the work available to the public.

10.2 Educational opportunities in MOOCs

The educational perspectives on media and technology development have historically been distinguished both by optimistic visions and profound scepticism related to the consequences that this development represents in terms of changes to the education institutions and educational practice. A pragmatic attitude has characterised players within flexible education, who have consistently applied new media as channels for communication of education schemes. Within the rest of the education system there have been visions and expectations over the last hundred years of changes in educational practice, without this coming to fruition.18 This applies, e.g., to how films, education television, video, computers, CD-ROM and other types of media merely have functioned as a supplement to the established educational practice revolving around the teacher and the text book. Over the last fifteen years, a number of critical voices have also been heard concerning the imbalance between the technological visions and use of new media in the educational practice.19 So far, the education institutions have only been marginally successful in changing the educational practice with use of technology from a traditional, communication-oriented model for web-based education, to utilising the opportunities offered by digital media for the students’ involvement in their own learning process, as well as the social collaborative dimension of learning.

Many now claim that this is about to change, not least in higher education. This is first and foremost about how technology development creates new opportunities for the use of digital media in an educational context. One opportunity is in scaling the services, nationally and globally, and how the access to technology and number of users in digital networks have increased considerably in recent years. Another opportunity is in the integration of various types of technology, such as video formats, social media and new learning platforms.

In many MOOC services, the attention is to a large extent directed at the participants as stakeholders in their own learning, while also changing the traditional role of teacher. The teacher has, e.g., a facilitator's role in support of collaboration and interaction between the participants. The communication and cooperation takes place in other ways such as through social media. Technology may be used to support students based on needs, capabilities and interests. This may take place by collecting and analysing data from the students’ learning process, and then preparing an education programme and course materials based on the progress of the relevant participant and contributions from other students. In some programmes the social interaction between the participants could be an important component. In other programmes the main focus could be on individualised, tailor-made learning processes through adaptive learning, i.e. systems that check what a participant knows about a subject, retrieve relevant learning objects from a database, check goal achievement and demonstrate learning progress over time.

Flipped classroom entails that the students study the subject matter outside the classroom, e.g. through video lectures, and use their time in the classroom for student-active learning activities such as presentations, seminars, exercises and reflection. Blended learning refers to educational schemes with varied learning activities that combine both web-based education and campus education. Blended learning and flipped classroom are not new phenomenons resulting from the MOOC development. However, the emergence of MOOCs gives students and the education institutions access to a far greater number of programmes than previously used in such educational schemes.

Over the last years, there have been several national initiatives to use digital exams in higher education.20 The work shows that digitisation of exams involves a number of opportunities and challenges, not least from an educational, administrative, technical and legal perspective. The Education Committee in the Norwegian Association of Higher Education Institutions (UHR) and Norway Opening Universities have jointly established an expert group to continue working on the legal perspective, as well as assessing the opportunities for future digital forms of evaluation.21 In recent years, an increasing number of lecturers and academic communities have used the opportunities offered by digital technology to evaluate students in new ways. The benefit of the digital evaluation types could, e.g., be more feedback to the lecturer on the student’s learning progress, more feedback to the student on the understanding of the subject and time saved for the lecturer in the grading process.22 The development of MOOCs has begged the question of how technology can stimulate new, efficient types of assessment for larger numbers of online students, and how digitisation of the assessments can help stimulate the quality of higher education. In connection with MOOCs, self-assessment and peer assessment have also received a lot of international attention.23 Research indicates that there is a correlation between peer assessment and teacher assessment. There is less knowledge as to whether there would be a similar correlation between peer assessment and teacher assessment in MOOCs, and whether the peer assessment would contribute to better learning outcomes.24

10.3 Learning analytics

In the literature on MOOCs, learning analytics is constantly highlighted as one of the key changes involved in the fully digital study programmes. Such analyses collect data from different levels in ways that previously were not possible: macro level (international, regional, national), meso level (institutions) and micro level (participants and participant groups). This applies particularly to higher education where data at the micro level has been missing.

Collection of such data is often referred to as data mining, and the objective is to identify and test various learning patterns. This knowledge is used to develop models to predict how different participant groups will succeed in their studies, how they learn and what needs they have. When this knowledge is integrated with analysis of the individual participant’s “learning history” (micro), the student will be able to follow his/her own progress, in addition to receiving individual and relevant follow-up. In the literature, this last process is referred to when we speak of learning analytics.

It is common to differentiate between five different types of learning analytics, but the field is rapidly expanding.

What is referred to as basic learning analytics entails using the analysis functions embedded in most of the learning platforms used today. A simple visualisation of data logs may give the individual participant a quick overview of his/her own results compared to others, and give the teacher an overview of the participants' activity. NOKUT points out that simple types of learning analytics are already in use in Norway, as the current learning platforms provide opportunities for visualising data logs and clarifying activity. Furthermore, NOKUT refers to the fact that some of the Norwegian education institutions also connect student data from several systems (e.g. LMS and Common Student System (FS)) in order to give a broader picture of the student activity.25

A more advanced form is referred to as predictive analytics. This involves combining static data, e.g. demographics and previous academic results, with dynamic data, e.g. log-in patterns on learning platforms, which documents the participants are working on, or the extent of participation in online discussions. The goal is to predict how well each individual participant will do, identify characteristic learning patterns for different groups, and applying relevant measures at an early stage. Students in an assumed risk group may be offered extra follow-up, and particularly proficient participants may be motivated to additional efforts by getting some extra challenges. Predictive analytics also assume that it will be possible to predict what kind of learning activities have the best effect on the individual student.26 Even if predictive analytics are still at an early stage of development, there are several examples of application of such analyses. In “School of one”, a mathematics programme introduced in public schools in New York City, learning algorithms are used to analyse the pupils’ manner of learning and math skills. The algorithm produces a personal mathematics “learning play-list” for each pupil, which follows the progression considered optimal for each individual student. In this programme, the predictive analytics are intended as a supplement and not as a replacement for teacher follow-up.27

Adaptive learning analytics builds models of the participants’ understanding of specific subjects. It makes it possible to automate individual feedback to the participants, for example which parts of the curriculum they have understood and on what level they have understood it. This may be integrated in learning platforms in such a way that the participants are continuously presented with learning content in line with their academic level.

Analysis of social networks is used to clarify relationships. The objective may be to identify participants who are not socially and academically integrated, or whether the teachers’ interaction with the participants is too biased in favour of one participant group. The Social Networks Adapting Pedagogical Practice (SNAPP) is an example of technology being applied to analyse behaviour for the purpose of learning. SNAPP analyses social networks and forum activity in LMSs, and provides the educators with diagnostic instruments to evaluate the digital participant activity in terms of learning. Such analyses are supported in modern research showing that there is strong correlation between offline activity, i.e. cooperation with other course participants or others who are familiar with the subject matter, and the test result. Participants who cooperate with others do achieve better results than those working alone.28

Discourse analysis is commonly considered the most complicated form of learning analytics. Here the systems must be able not only to log and identify the participants’ and teachers’ contributions and activities, but also the quality of what has been written. The system would then be able to give specific feedback to the teachers and participants on the quality of their contributions. Even if technologies exist that can analyse certain qualitative aspects of a text, they are not good enough for advanced learning purposes.29

Learning analytics, and particularly the most demanding types of such analytics, is still in its infancy. Researchers at HarvardX and MITx, who have analysed data from these institutions' MOOCs on edX in 2012 and 2013, wanted to discover which learning patterns gave good results for the course participants. However, in their initial reports they conclude that it is not possible to identify such patterns at the moment:

Everything predicts MOOC performance, because doing anything in this space separates you from the thousands of people who are doing relatively little – thus doing anything predicts doing anything else.30

In spite of the fact that the researchers possess data from nearly 850 000 participants, they explain their problems in finding such patterns with insufficient data. Because learning in MOOCs takes on so many forms, even more data is required to identify the learning patterns. An important conclusion in the project is that learning analytics appear to require extensive cooperation between the many institutions that possess data from MOOCs. The researchers believe that the institutions must enter into binding partnerships and share data in order to realise the potential inherent in learning analytics.31

10.4 “MOOC for ICT in learning” at Sør-Trøndelag University College

One example of how quality assessments can be carried out in the development of MOOCs, is the course “MOOC for ICT in learning” at Sør-Trøndelag University College (HiST). HiST has received funding from Norway Opening Universities in order to develop the course.32 The purpose of the project is to transform the existing web-based subject “ICT in learning” to a MOOC that can be offered free of charge and at a large scale to Norwegian teachers, as well as developing a model for how Norwegian players can offer courses and educational pathways as MOOCs that are both financially sustainable and have a high academic and educational quality.

In terms of the specific contents of the MOOC, it is, as mentioned, a question of developing a model for how Norwegian players can provide MOOCs. The educational objectives of the course are described as challenges that the institution offering MOOCs may encounter in terms of educational, assessment-related, administrative, technological and financial challenges. As regards the educational challenges, it points out the need to assess suitable teaching methods that ensure a good learning environment and learning outcomes. Design of both subject matter and learning activities are mentioned as key success criteria. Issues related to selecting suitable ICT tools and services to create good learning in various MOOC schemes and for different disciplines are also mentioned. The course will focus on how to motivate MOOC participants, how to reduce withdrawal rates and ensure good learning outcomes, the social interaction between participants, as well as use of personal learning environments and learning networks.

The project “MOOC for ICT in learning” provides good insight into issues related to what HiST calls “moocification” of a web-based subject, and what is characteristic for the development of MOOCs. In the project, HiST has selected certain criteria they believe must be given special attention in the development of MOOCs. Box 10.1 lists the emphasised quality criteria. HiST also points out that for each criterion, a description must be made for how the task is to be solved, as well as an objective for the outcome. It must be possible to verify and evaluate this process.

The learning activities to be used in the programme are also described, along with the technological solutions, organisation and implementation, what academic resources are to be used, how the lessons learned from the course can be applied in further work, and how it will be evaluated. They emphasise that planning, development and implementation of MOOCs require carefully considered use of technology. The project intends to use technology to make the cooperation between different players more efficient, minimize travel time, and not least produce good-quality results. The technology must facilitate the division of the subject matter into smaller parts, with a mixture of text, video, polls and multiple-choice questions, as well as the opportunity to have discussions related to each content block. The project at HiST also demonstrates the need to adjust learning contents to the technology used.

10.5 The Commission’s considerations

10.5.1 Quality in MOOCs

The Commission believes that the development and provision of MOOCs in Norway will require thorough work on the part of the institutions in order to ensure good-quality courses. It is the Commission’s opinion that much of the quality work related to campus education in general and web-based education in particular, will be highly relevant for the work on quality in MOOCs as well. This issue is also pointed out by the UK Quality Assurance Agency for Higher Education (QAA). QAA believes that UK’s “Quality Code for Higher Education”is a good basis for higher education institutions’ work on quality assurance of MOOCs as well.33 The Commission believes that the institutions should learn from the overall quality development and quality assurance work when they develop, provide and evaluate MOOCs.

One example of how such assessments can be applied to the development of MOOCs, is the course “MOOC for ICT in learning” at HiST, cf. Chapter 10.4 above.

The Commission feels that the project at HiST aptly illustrates how lessons learned and quality assessments from web-based education can be applied to the development of MOOCs, but that they have to be adapted to the MOOC format.

The Commission believes that learning objectives must form the basis for the quality work. This will be relevant for all types of MOOCs, whether it be a service with a clearly defined learning outcome, or courses with more general subjects where the specific learning goals are defined by the participants along the way. The learning activities must be designed so as to be an instrument towards reaching the learning goals. This also applies to the choice of technological solutions. The Commission therefore believes that there must be a clear connection between the purpose of the course, the choice of learning activities and choice of technological solutions. It is also important to have a good evaluation of the organisation and implementation of the course, e.g. related to the degree of participation, the teacher’s role, forms of communication and forms of interaction. Follow-up of the participants and a good learning environment will also be important. The choice of educational resources must be in line with the rest of the educational scheme. Evaluation and experience sharing should be used actively, e.g. as part of further development of the offer.

The Commission believes that MOOCs have a potential to strengthen both the access to and quality of higher education for people with impaired functional ability, thus making it possible for more people to choose higher education. This assumes, however, that the potential inherent in MOOCs is utilised in the right way, and that student diversity is taken into account in the further development of the programmes. A prerequisite in order for MOOCs to make education more accessible to all, is that the technology and contents are of high quality and adhere to the principles of universal design. The Commission also believes that it is important for the institutions to have the necessary familiarity with universal design when developing MOOCs, both in terms of the existing needs and how the courses can be designed to take these needs into account in the best possible way.

10.5.2 Assessment, exams and identity checks

The Commission believes that digital tools provide new opportunities for assessments and exams.34 Firstly, digital media can facilitate testing of a large number of users, thus streamlining the forms of assessment. Secondly, digital tools, e.g. through simulations, provide new opportunities for evaluating skills that traditionally have been difficult to assess. This could for example relate to cooperation, problem-solving and creativity. Thirdly, it could entail considering new areas of expertise, such as digital skills.

The Commission is of the opinion that the development of Norwegian MOOCs must ensure that exams and assessment schemes support the courses’ other quality work. If course certificates from MOOCs are to have an impact in working life, it is also necessary to have systems that are credible to society.

The Commission believes it is important for the institutions to consider and follow up the recommendations given by national work groups on the educational, administrative, technical and legal sides of digital exams.

Chapter 8.4 refers to challenges in ensuring correct identification of MOOC participants. The Commission believes that there must be systems to check the identity of Norwegian students enrolled in both foreign and Norwegian MOOCs. Such an identity check is e.g. essential in order for foreign MOOCs to be included as part of a degree, or for admittance to studies at Norwegian institutions. A secure identity check is also important if a person seeking employment wants to use MOOCs as part of documenting skills when applying for a job. The Commission believes that it is essential that further work be carried out to develop sound solutions to secure the identity of persons taking MOOCs. The Commission recommends that questions relating to handling of personal information in MOOCs be included in the investigative studies related to digital exams in the university and university college sector.

10.5.3 Measures and premises for quality development

In order for Norwegian higher education to utilise the educational opportunities represented by MOOCs, the Commission believes there is a need for strategic measures in several areas: strategic ownership through national framework conditions and good educational management at the institutions, incentives for developing teaching quality, further development of digital skills among employees and students within the sector, as well as development and sharing of knowledge related to the educational aspects of technology development.

Strategic ownership

In order for strategies for the use of technology in education and development of MOOCs to be sustainable, they must be anchored and supported by the leadership of the institutions, and must be part of the institutions’ overall strategies and plans. Good strategies are a matter of clear educational management at all levels in each institution. At the same time, MOOC development strategies must also be anchored at a national level. National authorities have to facilitate framework conditions and incentives to support the work of the institution. The Commission’s recommendations to national authorities in this report are meant as a contribution to precisely that.

Incentives for developing teaching quality

The higher education sector uses a limited number of incentives at the individual level as regards developing teaching quality. The incentives for research are far stronger. This is not unique for Norway. In a new Swedish report, Swedish institutions are compared with Stanford and Berkeley. The comparison shows that Swedish universities and university colleges have significant potential for improvement, particularly as regards the role of teaching. The report concludes that the role of teaching in a research career must be strengthened, e.g. through clear career paths for those who combine research and high-quality teaching, and through systematic evaluations of teaching quality.35

The Commission believes that an analysis similar to the Swedish one is highly relevant for Norway. In the Commission’s opinion, there are no incentives for educational development work beyond the minimum standards. In the Norwegian Association of Higher Education Institutions' instructive guidelines concerning promotion to professorships in several disciplines, there is only one minimum requirement relating to competence for teaching in the individual subjects.36 The lack of incentives does not stimulate and motivate the individual scientific employees to get involved in the development of teaching quality. The Commission believes that the current minimum requirements for educational competence in the higher education sector are insufficient to ensure educational development work, nor do they facilitate follow-up of the development taking place within technology and new forms of teaching. The Commission believes that there is a need for better incentives for increased teaching quality, as well as for more innovative types of learning. The goals for teaching quality must be far more ambitious. The Commission therefore recommends a review of the general range of policy instruments and incentive schemes for the education area, at the individual, institution and national levels. Instruments and incentives must be coherent and pull in the same direction.

Digital competence

Digital status 2011 shows that, so far, the institutions are not using a lot of resources on developing the employees’ skills in varied use of ICT to promote the students’ learning.37 In order to use technology in teaching and learning, the Commission therefore believes that there is a need for a systematic and permanent skills upgrade for personnel in the higher education sector.

The Commission believes it is entirely necessary to strengthen the digital proficiency related to teaching for personnel in the higher education sector. The Commission strongly feels that the use of technology in learning creates the need for broadly composed expertise of an educational, technological and administrative nature. The learning must be organised in a team-based manner. It is important that academic employees gain experience in developing educational schemes with the use of technology that will stimulate good learning activities and increased education quality. Knowledge and experience must be developed in applying digital instruments, resources and services. The Commission believes that the institutions should take comprehensive, organisational steps by prioritising funds for developing composite expertise and support functions related to the educational development work. The Commission believes that the need for such expertise development is great, and that not all institutions are capable of establishing such services under their own direction. The Commission therefore believes that national authorities must take overall responsibility by making the expectations clear to the institutions regarding development of digital skills, but also by allocating funds to initiate the desired development. As regards the need for and scope of the support services required for developing MOOCs, the Commission is of the opinion that this must be investigated further. The relationship between the institutions’ own responsibility for such services and the need for national initiatives must be considered in such an investigation.

The Commission also believes that it is important that the students have good digital skills. The Commission believes that it is important that the entire educational pathway, from primary and secondary to higher education, is coherent. Students from primary and secondary education must have acquired knowledge and skills relating to the digital tools and methods that are necessary to complete MOOCs. The Digital Commission recommended greater efforts in this area. The Digital Commission believed that higher education institutions should be required to review their descriptions of learning outcomes in all study programmes to ensure that digital skills are reflected in the educational pathway.38 The MOOC Commission supports the need for strengthening students’ digital skills.

Research on educational opportunities and learning activities

In the opinion of the Commission, there is a significant need for more research into the educational and learning aspects of technology development within higher education. It is important to have systematic and research-based knowledge development, both to contribute towards learning across institutions and academic communities, and for academic and economic resources to be used in the best possible way.

The Commission believes that a number of challenges related to quality in MOOCs and the use of technology in learning need to be considered. The Commission believes there is a need for strategic measures in the form of funds for research and knowledge development within this field. Relevant topics include what effect MOOCs will have on education quality, the learning outcome of attending different types of MOOCs, whether all student groups will handle the new forms of learning equally well, how to facilitate those who will face challenges, whether digital forms of learning and MOOCs can work equally well in all academic areas, and new assessment methods such as peer review and digital exams. Another important topic is the use and effect of MOOCs along with other learning activities on campus, in different types of blended learning and flipped classroom scenarios.

Learning analytics as a policy instrument for quality development

The types of learning analytics with the greatest potential for quality development are still in early phases of development. There is some uncertainty as to how demanding the analyses will be, as well as how beneficial they will be. There are questions as to whether the large amounts of data generated and stored for use in learning analytics are problematic in terms of personal data protection. This is also a key issue in the discussion surrounding big data.

Ownership and storage of data is another topic that must be addressed. One relevant question is how to ensure that the large amounts of data generated in learning platforms and stored by the commercial companies owning the platforms, are not sold to other players. Another question is how to ensure that logs containing the participants' activities are made available for research in formats that will secure anonymity, reuse of data and an appropriate aggregation level. In order to carry out advanced forms of learning analytics, access to data from multiple sources will be required. The contracts the institutions enter into with the large MOOC providers stipulate that the data cannot be passed on to a third party without the consent of the institution.

In sum, there appear to be several unanswered questions, e.g. regarding how easy it would be to gain access to data for research purposes, and whether the various countries’ regulations are sufficiently harmonised. There are no standardised formats for this data. In other words, there are some questions of principle and certain unclear issues related to personal data protection, data storage and use of data that should be discussed in relation to learning analytics.

The other main challenge is whether learning analytics can yield anything of substance as regards quality. We need to know what the data can tell us, and equally important – what they cannot. Another obvious issue is that the data are not neutral. This may lead to misinterpretations. Too strong an emphasis on results from a learning analytics may also entail that the focus in the learning process is shifted to certain areas at the expense of others, i.e. that the analyses will govern the learning. This means that important and unclear factors linked to learning analytics and quality should also be discussed.

It is the opinion of the Commission that learning analytics have considerable potential as a tool for quality development in higher education. However, this demands that the numerous challenges mentioned above be assessed in a satisfactory manner. The Commission therefore believes that it will be crucial to strengthen the knowledge basis in this field.

The Commission also believes that the use of learning analytics must be incorporated as part of the educational development work and the bolstering of digital skills at the institutions. At the same time, the Commission sees that it will be necessary for someone to take special responsibility for developing expertise in this area, and contribute to the dissemination and application of this expertise within the sector. The Commission therefore believes that a learning analytics community should be established. The structure and form of this community must be considered vis-à-vis the current players and range of instruments.

The community should be tasked with conducting systematic and research-based knowledge development as regards learning analytics, and through development work and knowledge transfer, contribute to developing and applying learning analytics in Norway. This could e.g. include mapping of global projects and technologies, as well as use and testing of suitable technologies. The community should have a practical approach and connect with relevant stakeholders. Knowledge transfer will be important for the institutions’ will and ability to develop courses, and seize the opportunities provided by learning analytics to further develop the quality of higher education.

10.6 The Commission’s recommendations

• The Commission recommends that experience and knowledge from quality work in flexible and web-based education be applied in the development of MOOCs.

• The Commission presumes that the institutions will base their development of MOOCs on the principles of universal design.

• The Commission recommends that the institutions test new forms of educational assessment and exams in MOOCs.

• The Commission recommends that questions regarding the handling of personal data information in MOOCs be included in the review of digital assessment and exams.

• The Commission believes that it is necessary to strengthen the digital skills of employees in the higher education sector. The scope must, however, be mapped in more detail. The Commission recommends that funds be allocated to bolster digital skills.

• The Commission recommends that the institutions develop the employees’ expertise in use of technology in teaching.

• The Commission recommends that the institutions take responsibility for developing the students’ digital skills.

• The Commission believes that there is a need for stronger incentives for increased quality in teaching, as well as for more innovative types of learning. The Commission therefore recommends a review of the general range of policy instruments and incentive schemes for the education sector at the individual, institution and national levels.

• The Commission recommends a systematic effort towards research-based knowledge development regarding the use of technology in higher education.

• The Commission recommends establishing a community for research-based knowledge development, development work and knowledge sharing related to learning analytics.