Part 1
Background

2 The composition, mandate and work of the Expert Group

Zoom in on image

2.1 Composition of the Expert Group

The Expert Group for learning analytics consists of experts in fields including education, ethics, technology and law.

2.2 Mandate of the Expert Group

2.2.1 Excerpts from the Expert Group’s mandate

The Expert Group will provide the Norwegian Ministry of Education and Research with a better basis for decisions regarding learning analytics and adaptive teaching aids, as well as exams and tests in primary and secondary education and training, higher education and tertiary vocational education. It will also advise on the need for regulation and input to policy development and measures by the Norwegian Ministry of Education and Research and subordinate agencies.

The tasks of the Expert Group

The Expert Group shall assess pedagogical and ethical issues in the use of learning analytics, as well as legal issues and data protection considerations. The group shall advise national authorities on the need for the development of legislation for the aforementioned levels of education. The work must include assessments of future opportunities relating to the tools and how the market for adaptive teaching aids will develop in the future.

Furthermore, the Expert Group will provide input to the education sector on how good practices can be developed for the use of learning analytics, in line with ethical and pedagogical standards and applicable legislation.

Key questions

In its work, the Expert Group will base its work on the following key questions:

• How does learning analytics affect learning?

  The Expert Group will assess whether and how learning analytics affects the professional roles of teachers and instructors, the relationship between teachers and teaching aids, views on teaching and on the pupil/student role. For primary and secondary education and training, it is essential to consider learning analytics in light of both the formative and pedagogical missions of the education, and to consider whether learning analytics affect the breadth of what the pupils are to learn and the differences between various subjects.

• What are the challenges and potential of learning analytics?

  The Expert Group shall assess ethical issues that are closely linked to the pedagogical assessments. Among other things, the Expert Group shall assess how learning analytics can contribute to the inclusion or exclusion of pupils/students or groups thereof from the instruction, e.g., due to special needs, including those requiring universal design or those with linguistic minority backgrounds, and the effects on possible differences in learning outcomes. Assessment of data protection issues and monitoring of the use of the generated data will be key, as well as whether there are various ethical considerations associated with different types of data/data sources. For primary and secondary education and training, there are also ethical considerations, particularly with respect to the age of the pupils, as well as the balance between the need for good supporting data and the desire for data minimisation, and between requirements for the protection of children and the interest in early intervention. Furthermore, social science issues, such as the relationship between the use of learning analytics and public interests, and democratic values such as openness, transparency and privacy, may be highlighted. The Expert Group must assess whether the quality of the knowledge base has consequences for ethical choices, proposed measures and other recommendations for learning analytics.

• How can legislation provide appropriate support to the sector?

  A key question is whether there is a need for additional regulation or guidelines for the use of learning analytics in sectoral legislation or other legislation. Assessments of pupils’/students’ data protection are an important aspect of the ethical issues, especially when processing data on children and vulnerable groups of pupils and students. Consideration must be given to whether there is a need for clarification as to what types of processing of personal data are permitted to safeguard the rights of pupils/students and whether all use of learning analytics will constitute a form of profiling, cf. Article 22 of the General Data Protection Regulation (GDPR).

• What competence do the education sectors require to make good assessments of learning analytics?

  The Expert Group shall assess what competence the education sectors require if learning analytics are to be used in training and education, including the legal, financial and digital competence and competence to assess risks related to data protection, ethics and education in the exercise of various roles.

Structure

The work of the Expert Group shall result in two or more interim reports submitted to the Norwegian Ministry of Education and Research. Basic ethical and pedagogical assessments of the opportunities, benefits and risks relating to learning analytics shall be included in the first interim report. Where there are issues common to the different levels of education, these can be addressed jointly. Where there are significant differences between the levels of education, primary and secondary education and training shall be given priority in the first interim report.

Supplement

In December 2022, the Norwegian Ministry decided that the Expert Group’s main report should be published as an Official Norwegian Report (NOU).

2.2.2 The Expert Group’s interpretation of the mandate

The main task of the Expert Group is to contribute to a better basis for future decisions on learning analytics and adaptive teaching aids, exams and tests in the Norwegian education system. We are also to advise on necessary statutory regulations and input to policy development and measures. We have emphasised providing an overview of a complex field and considering pedagogical, ethical and legal issues in context, rather than separately.

As the title of the report suggests, we are concerned with pupils’ and students’ learning and the ways in which learning analytics affect their learning process. This is also a key issue in the mandate. In order to shed light on the question of learning, it has been necessary to identify what characterises learning analytics in today’s schools, vocational colleges, universities and university colleges. We have considered it crucial to gain an overview of what kinds of experiences various actors in the Norwegian education system actually have had with learning analytics. Through our work, it has become clear to us that this area is understudied and that there are very limited sources to inform us of this subject. Simply put, we know little about learning analytics in practice in primary and primary and secondary education and training, tertiary vocational education and higher education.

Discussions about learning analytics are characterised by a considerable gap between the technological potential envisaged for learning analytics and the pedagogical reality in which instruction and learning take place. Therefore, we have given priority to assessing experiences with and research on learning analytics in Norwegian pedagogical practice. It is this knowledge, including a number of responses and ongoing dialogue with the education sector and other stakeholders, that forms the basis for our recommendations.

A holistic view of learning analytics

The Expert Group understands learning analytics to be a process in which data generated by pupils or students are used systematically to enhance learning and improve instruction. Although our mandate requires thorough legal discussions, we have found great value in considering the issues of learning analytics from various academic perspectives. In accordance with the recommendations made in NOU 2019: 23 Ny opplæringslov [New Education Act] we find it essential that technological, pedagogical, normative and ethical aspects are continuously assessed in all learning analytics. The Expert Group has therefore chosen to discuss these aspects in context, rather than keeping them separate.

We have found it valuable that our mandate is broad in scope and does not reduce learning analytics solely to legal issues. Our intention has always been to assess not only what is legal, but also what are pedagogically prudent and ethically justifiable choices related to learning analytics. Our ambition has also been to place the objective of good learning and education at the centre of the assessments to the extent possible.

A balanced view of opportunities and challenges

Most issues regarding digital technology can quickly end up in a polarised debate involving fixed positions either for or against digitalisation in general. This tendency also applies to issues regarding the role of digital technology in learning situations. The discourse surrounding pupils’ and students’ use of technology is characterised by bold claims and strong emotions. However, both the potential and the adverse aspects of using technology in instruction are well documented, and we have strived to remain objective in the general discussions about the overall effect of technology on schools and educational institutions. We have chosen to give a balanced presentation of the opportunities and challenges that learning analytics create or reinforce in education. Therefore, we will not take a position on whether there should be more or less learning analytics at different levels of Norwegian education. However, we will provide advice on potential value and risks.

Artificial intelligence

Our mandate does not expressly mention artificial intelligence, but as we will explain in section 2.3, this is directly relevant to learning analytics. Artificial intelligence has also become increasingly important in pedagogical issues in the past year, largely due to major innovations in the field that were quickly adopted by pupils, teachers, students and instructors – and which raise a number of new pedagogical, ethical and legal questions. The Expert Group has considered it important to include issues related to artificial intelligence and we have done so where appropriate based on our mandate.

Unclear understanding of learning analytics has impacted work of the Expert Group

We will briefly comment on how the fact that the very concept of learning analytics is so foreign has been a challenge in our work. The Expert Group finds that the definitions of learning analytics found in scientific literature are usually incongruous with how learning technology suppliers, instructors, students, teachers, pupils, parents and administrators in the education sector define the term. Today, many people use pupil and student data to improve instruction but without referring to it as “learning analytics.” We have also experienced the opposite, i.e., that the term learning analytics is used rather uncritically and in an excessively broad manner. The ambiguous use of terminology makes it more difficult to understand and obtain knowledge about when and how learning analytics occur.

The task of the Expert Group is twofold: 1) to assess how learning analytics affects learning, and 2) to advise on good practice for learning analytics in today’s schools, training establishments, vocational colleges, universities and university colleges. To carry out the first task, we need to turn to the research and theoretical definitions of learning analytics. For the second task, we must address how the education sector itself uses the term and discusses the analysis and interpretation of pupil and student data. A number of comments to the Expert Group emphasise the need for more systematic insight into learning analytics in practice. It is also important to have relevant, comprehensive and concrete examples that show how analyses of pupil and student data can support learning, in order to be able to assess the value of learning analytics in primary and secondary education and training, higher education and tertiary vocational education in the future. The Expert Group emphasises that it has been challenging to find good examples of practice in Norwegian education, even though many of the digital learning resources in use in the education programmes have functionality for learning analytics.

2.3 The terms learning analytics, adaptivity and artificial intelligence

Learning analytics and adaptivity are two key concepts in the mandate of the Expert Group. Artificial intelligence is not mentioned in the mandate but is nevertheless part of the Expert Group’s work. The reason is that artificial intelligence is an important component of many forms of learning analytics in general and adaptive systems in particular.

There are many different perceptions of these three concepts. In this chapter, we briefly describe the understanding we have applied in this report and how we perceive the terms to be used in practice.

2.3.1 How the Expert Group understands the terms

The Expert Group assumes that learning analytics is the systematic use of data to enhance learning and improve instruction. Learning analytics can most simply be described as a cycle, and it is a process with several necessary steps, as illustrated in Figure 2.2.

Zoom in on image

The starting point of the cycle is a learning situation. Data is collected from the learning situation, which is then analysed by a computer. Sometimes, this analysis combines data from the learning situation with data from other sources. The results of the analysis are presented in a manner that allows the recipient to use the information to make a decision on learning or the instruction. An example of how to present results (referred to as visualisation in Figure 2.1) may be a report that provides an overview of a learning assignment or a recommendation for new learning activities. In many digital learning resources, we find elements from learning analytics, such as data collection and visualisation or recommendations for activities. However, learning analytics can only be said to have been performed when a change has occurred based on the data from the analysis.

The Expert Group assumes that adaptivity means the automated, individual adaptation of a learning resource using artificial intelligence. An example of such adaptation is that a pupil is automatically assigned assignments in a test based on the pupil’s answers to previous assignments in the test. Other forms of adaptivity may be that the content or display of a teaching aid automatically adapts to the pupil’s preferences, based on information about how the pupil has previously used the resource. Adaptivity is relevant for learning analytics because learning analytics will increasingly be based on data from adaptive systems – especially in schools.

The Expert Group assumes that the form of artificial intelligence that is currently most relevant for the education sector is increasingly based on machine learning, i.e., computer programmes with the ability to experience and act – learn – based on large volumes of data. Artificial intelligence is a prerequisite for stating that a learning resource is adaptive but is not necessarily part of learning analytics. However, some resources that have functionality for learning analytics will have built-in artificial intelligence.

2.3.2 How does the education sector use and understand these concepts?

Through our work, it has become clear that the terms learning analytics and adaptivity are used and understood in many different ways. The term learning analytics is scarcely used in the education sector. The term adaptivity, on the other hand, is almost overused as a general term for all types of adaptations of digital resources. When data is used systematically to enhance learning and improve instruction, it is rarely referred to as learning analytics. At the same time, we see that when the term is used, it is often to describe parts of the steps included in the learning analytics process. Adaptivity – which assumes an automated adaptation for the user with the aid of artificial intelligence – is used to also describe the adaptations the user personally makes to personalise a digital resource.

Artificial intelligence is developing at a rapid pace. In the last year alone, artificial intelligence in education has gained a significantly more prominent position – both in the field of practice and in the public debate. However, not all use of artificial intelligence in education constitutes learning analytics or adaptivity but it can be difficult to define the parameters. Nor is there necessarily a goal to establish fixed boundaries between learning analytics with and without artificial intelligence or to determine when the use of artificial intelligence qualifies as learning analytics. It is more important to focus attention on when pedagogical decisions are in practice made by human beings and when they are made by machines. It is crucial to have a conscious approach to what kinds of decisions should and shall be made by humans and which ones can we leave to the machines.

2.4 The work of the Expert Group

The Expert Group has held a total of nine meetings. Four meetings were held before the group’s interim report was submitted on 1 June 2022 and a further five meetings were held until the submission of the main recommendation on 6 June 2023.

2.4.1 Input and knowledge gathering

During the course of its work, the Expert Group has involved many stakeholders and specialist environments.

Many of the meetings included external opening speakers, who presented and highlighted key topics and initiated discussion. See the list of external speakers in Table 2.1.

In November 2022, the Expert Group organised an industry forum in collaboration with ICT Norway, where Canvas, Cappelen Damm, Conexus, Gyldendal, Hypatia Learning, Inspera, Neddy and Visma InSchool participated. At this forum, the suppliers demonstrated various resources with functionalities for learning analytics and provided comments to the Expert Group. The industry forum included speakers from Microsoft, Google and Apple, who also answered prepared questions.

Rambøll conducted an assessment of learning analytics in primary and secondary education and training and higher education on behalf of the Expert Group. See the results of this assessment in section 3.4.

Open input meetings

In February 2023, two open digital input meetings were held on learning analytics in primary and secondary education and training, higher education and tertiary vocational education.

Invited speakers were Rambøll, Bogstad School in Oslo, the School Student Union of Norway, Union of Education Norway, Gyldendal, BI Norwegian Business School, the National Union of Students in Norway, the Organisation of Norwegian Vocational Students and Universities Norway. Several participants made brief statements. Around 150 participants participated in the two input meetings.

Input meetings

During 2022 and 2023, the Expert Group conducted input meetings with the following actors:

• Norwegian Data Protection Authority

• Norwegian Directorate of eHealth

• Industry: BS Undervisning, Cappelen Damm, Cyberbook, Conexus, Disputas, Fagbokforlaget V&B, Gyldendal, Hypatia, ICT-Norway, Kikora, LearnLab

• School Student Union of Norway

• Fylkesledernettverk for fylkeskommunale ungdomsråd [Network of County Youth Council Chairs]

• Parents’ Committee for Primary and Secondary Education (FUG)

• Lawyers: Emily Weitzenboeck (OsloMet), Jon Christian Fløysvik Nordrum (University of Oslo), Kirsten Kolstad Kvalø, Malgorzata Cyndecka (UiB), Mona Naomi Lintvedt (University of Oslo) and Sebastian Schwemer (University of Copenhagen)

• Norwegian Association of Local and Regional Authorities (KS)

• Municipalities/county authorities: Asker, Lillestrøm, Lørenskog, Oslo, Surnadal (the ICT-Orkidé cooperation), Voss, Møre and Romsdal, Vestfold and Telemark and Vestland

• Medietilsynets ungdomsnettverk for digital oppvekst [the Norwegian Media Authority’s youth network for digital upbringing]

• Nettverket for medvirkning i opplæringen (NEMIO) [Network for Participation in Education]: School Student Union of Norway, Norwegian Ombudsperson for Children, UNICEF, UngOrg, Voksne for Barn [Adults for Children], Norwegian Association of Youth with Disabilities, Save the Children Norway

• Norwegian Association of Graduate Teachers

• Norwegian National Union of Students

• Organisation of Norwegian Vocational Students (ONF)

• Secretariat of the Norwegian Privacy Commission

• Save the Children Norway

• Umbrella Youth Council of Oslo (SUR)

• Sikt – the Norwegian Agency for Shared Services in Education and Research

• Norwegian School Heads Association

• Universities: Norwegian University of Life Sciences (NMBU), Norwegian University of Science and Technology (NTNU), University of Oslo, University of Stavanger and UiT The Arctic University of Norway

• Organisation of Norwegian Vocational Students and Universities Norway.

• Norwegian Directorate for Education and Training

• Union of Education Norway

• Vestland ungdomsutval [Youth Council of Vestland County]

• Viken ungdomsråd [Viken Youth Council]

Written comments

The Expert Group has received written comments from the actors in Table 2.2. The written comments are published on the group’s website1.

2.4.2 Involvement of children and youth

The Expert Group has held several meetings with pupils and youth councils in addition to meetings with organisations representing these groups. See Boxes 2.3 and 2.4 for issues discussed at the meetings. Learning analytics applies to and has consequences for children and youth. Therefore, it was important for us to listen to this group’s perspectives on the issues in question. Furthermore, children have the fundamental right in both Article 104 of the Constitution of Norway and Article 12 of the UN Convention on the Rights of the Child, to freely express their views and be heard in all matters that affect them.

Comments from children and youth have broadened our understanding, and we refer to these comments in the report, where relevant.

Conversations with pupils

The Expert Group has had facilitated meetings with pupils in the upper primary level (grades 5-7) and the lower secondary level (grades 8-10) in different parts of the country. During the school visits, we have had semi-structured conversations with pupils in smaller groups. The purpose has not been to obtain a representative overview of pupils’ perceptions, but rather to enhance our understanding of what pupils find to be important.

Comments from youth councils and networks

The Expert Group has participated in meetings with several of the county youth councils. In the meetings, we have explained what learning analytics are and presented questions that the councils have discussed. We have subsequently received several written comments from the youth councils.

Frameworks for the involvement of children in the investigation work

We have greatly appreciated all the comments submitted by children and young people. It is nevertheless challenging to implement processes for the involvement of children. The challenges have involved limited resources and lack of expertise related to such involvement processes. The Norwegian Directorate for Children, Youth and Family Affairs guide Principles and Advice: Child and youth involvement at the system level (Norwegian Directorate for Children, Youth and Family Affairs, 2021) and the Norwegian Ombudsperson for Children’s Participation Handbook (Norwegian Ombudsperson for Children, 2021) have been useful in this work, but we do not believe that these written guides are sufficient.

One shortcoming is that citizens are inadequately involved at the investigation level with respect to digitalisation reforms in the public administration (Broomfield and Reutter, 2022). In recent years, however, there has been a greater focus on involving children and youth in public investigation work, and there are several examples of thorough involvement processes. Several of these investigation efforts have indicated the need for strengthening the frameworks for implementing involvement processes with children.

The Education Act Committee found itself in uncharted waters when it involved children and youth in its investigation process (NOU 2019: 23). Therefore, it called for more comprehensive efforts to contribute to improved processes and routines for involving children. The Children Act Committee proposes that the Norwegian Government consider establishing a body whose main task is to contribute to justifiable involvement processes with children (NOU 2020: 14). Among other things, the Children Act Committee proposes that such a body contribute by designing plans, creating questions for children, recruiting, interpreting results and, if relevant, implementing or facilitating such processes with its own employees.

The Expert Group supports the proposal by the Children Act Committee for the Norwegian Government to consider establishing a body whose main task is to facilitate involvement processes with children in investigation work. With respect to learning analytics, including the use of artificial intelligence, it is essential to safeguard the perspectives of youth, not only in the investigation process, but also in how learning analytics resources are used.

2.5 The Expert Group’s publications

The Expert Group has submitted the following reports: Læringsanalyse – noen sentrale dilemmaer [Learning Analytics – Some key dilemmas] 1 June 2022 and Læring, hvor ble det av deg i alt mylderet? Bruk av elev- og studentdata for å fremme læring [Learning: Lost in the shuffle?] 6 June 2023.

The first interim report comprises two parts. The first part comprises a primer on learning analytics, discusses different types of learning analytics and provides insight into knowledge development, ethics and activities in the field. The second part delves deeper into four dilemmas related to learning analytics. In the first interim report, we emphasise primary and secondary education and training.

In this last report, we delve more thoroughly into learning analytics in Norwegian pedagogical practice, discuss data types and data quality in learning analytics, and examine the legislation relevant to learning analytics for primary and secondary education, higher education and tertiary vocational education. The second part contains the Expert Group’s assessments of how learning analytics can enhance learning and improve instruction, as well as assessments of the pedagogical and ethical challenges and the need to regulate learning analytics. The third part of the report addresses the Expert Group’s proposals and recommendations.

2.6 Other investigation processes relevant to the work of the Expert Group

New Education Act for primary and secondary education and training

A proposal for a new Education Act was circulated for consultation in the autumn of 2021. Prop. 57 (2022–2023) Lov om grunnskoleopplæringa og den vidaregåande opplæringa (opplæringslova) [Act relating to Primary and Lower Secondary Education and Training (the Education Act)] was submitted to the Storting in March 2023. The bill refers to the Expert Group’s work on “assessing pedagogical, ethical, legal and data protection issues in the use of learning analytics and advising on the need for legislative development and comments on good practice”, which “shall provide the Norwegian Ministry with a better basis for decisions concerning learning analytics in the knowledge sector” (p. 473). Proposals for new regulations to elaborate on the provisions of the Act will be circulated for consultation in autumn 2023. The proposal for a new Education Act will be discussed at the Storting and is scheduled to enter into force from the start of the 2024 school year. The Norwegian Ministry will adopt new regulations in the spring of 2024.

Committee for Quality Development in Schools

The mandate of the Committee is to identify and review the needs of teachers, school administrators, school owners and national authorities for tools, tests and data sources for quality development.2 The Committee shall propose changes to current tests, tools and data sources with the aim of facilitating improved quality development. The interim report Kvalitetsvurdering og kvalitetsutvikling i skolen: Et kunnskapsgrunnlag [Quality assessment and quality development in schools: A platform of knowledge] was presented in January 2023 (NOU 2023: 1). The final recommendation by the Committee will be presented in autumn 2023.

Report to the Storting on professional education programmes

The Norwegian Ministry of Education and Research has announced that it will present a report to the Storting in spring 2024 on professional education programmes in higher education. The report to the Storting will emphasise the education programmes governed by framework plans, including teacher training.

Report to the Storting on tertiary vocational education

The Norwegian Ministry of Education and Research has announced that it will commence work on a new report to the Storting on tertiary vocational education. This report aims to provide additional knowledge on how tertiary vocational education can best utilise its potential and how this sector should be further developed. According to the Norwegian Ministry of Education and Research, the report to the Storting will be presented no later than spring 2025.

3 Learning analytics in Norway today

There has long been considerable interest in questions concerning the role of technology in schools and education. The debate rages on about screen time in schools, the use of artificial intelligence in assessments, children and young people’s privacy and a number of other issues concerning digitalised learning processes. Because the interest is so great, it is surprising that there is hardly any systematic research either on what is collected in terms of pupil and student data, what teachers and instructors believe they need learning analytics for, and what actually constitutes the common use of digital footprints in today’s education system.

As the Expert Group explained in the first interim report, we still have little systematic knowledge about learning analytics in practice at Norwegian schools and educational institutions. Advice on how to develop good and justifiable practices for learning analytics must be based on what we know about current practice and what needs the actors in the education sector believe that learning analytics can address. In this chapter, we will present new findings from an assessment of learning analytics in Norwegian primary, secondary and higher education and training and provide a brief overview of the research. We will also account for the type of needs the various actors in the education say they have for learning analytics and what barriers stand in the way of good learning analytics.

3.1 Brief overview of learning analytics research

In our first interim report, we outlined interdisciplinary research and research and development (R&D) on learning analytics. We pointed out that much of the work done in the field is small-scale practical testing. There is considerable activity, but research is still at a point where it is difficult to envision what will be possible to achieve in practice, where the breakthroughs will occur and what the legal constraints will be (Kluge, 2021; Selwyn, 2022). As the research project GrunnDig3 on digitalisation in primary and secondary education and training shows in its final report, there is a lack of systematic research that examines the effect of different forms of technology use in academic contexts (Munthe et al., 2022). The GrunnDig report also shows that much of the existing research concerns either science or language subjects. The knowledge summary by Misiejuk and Wasson (2017) on learning analytics also emphasises the lack of knowledge about what they refer to as “everyday analytics”, i.e., knowledge about how the data collected is actually used.

There is little systematic research conducted on learning analytics in real pedagogical practice at all levels of the education system. Another major challenge is that the new knowledge actually being developed about learning analytics both in Norway and internationally tends to remain within the research environment. This means, among other things, that the results often only have a limited impact on established pedagogical practice, for the products and for the market.

However, several research projects related to the use of artificial intelligence in education have been planned and initiated. The projects Artificial Intelligence for Assessment for Learning to Enhance Learning and Teaching in the 21st Century (AI4AfL)4 and Gameplay5, which will integrate machine learning with a gaming platform to detect reading and writing difficulties, are already underway. The innovation project Learning in the age of algorithms6 and the research projects Ethical risks assessment of artificial intelligence in practice (ENACT)7 and Artificial intelligence in education: Layers of trust (EduTrust AI)8 have recently received funding – to name a few.

3.2 Different practices at the different levels of education

Although there is little knowledge about learning analytics in practice in primary, secondary, higher vocational and higher education and training, we know that there are some significant differences between the different levels of education. The differences are evident both when we assess which tools are available and in use, and in conversations with suppliers and users.

Practical training in primary and secondary education and training

The general situation in Norway is that much of the commercial development of digital learning resources is primarily aimed at primary and lower secondary schools. This is also where we find the most use of various resources that support learning analytics, and it is here that we find the most learning analytics aimed at individual follow-up, as well as the most adaptive feedback to each pupil. In upper secondary education we find somewhat less use than in primary and lower secondary schools. Although there is more use in primary and lower secondary schools than elsewhere, we can describe the use as limited here as well, and it is often limited to certain subjects. Comments we have received from Research, Innovation and Competence Development in School (FIKS) at the University of Oslo specifically underline this issue. Based on their activities in a number of municipalities, they write that learning analytics is “unfamiliar and is actively used only by a minority of teachers, especially in the lower grades” (FIKS, 2023, p. 1).

Practice in higher education and tertiary vocational education

While it is primarily in primary and secondary education and training that we find most practical examples of learning analytics, most of the research both in Norway and internationally has been conducted on higher education, especially within specific programmes of study with customised tools for planning instruction and courses or related to counteracting dropout. We should note that the use of digital resources in higher education and tertiary vocational education in Norway is primarily limited to various administrative tools such as digital learning platforms, reading list systems and administrative examination systems. This means that the existing use of technology that supports learning analytics is also often intended to address more administrative tasks. For example, the knowledge sector’s service provider Sikt writes in its input that it primarily provides “tools or systems intended to support the instructor or administrative staff in planning and carrying out instruction and assessment, i.e., technical tools that support the educational process” (Sikt, 2022, p. 1).

An important common feature of the administrative systems in higher education is that they are not primarily purchased to facilitate learning analytics, even though they may have functionalities that allow for this. We also note that whereas learning analytics in primary and secondary education and training often involve individual resources developed for specific subjects or areas in subjects, it tends to be more generic and linked to managing entire subjects in higher education. We are aware that some educational institutions have tested specific tools that facilitate learning analytics in individual courses within, e.g., science subjects. Apart from learning management systems and other administrative systems, we do not find the widespread use of tools with functionalities for learning analytics on a larger scale.

One of the reasons we do not see more learning analytics in pedagogical practice in higher education is that the sector finds the legal basis for learning analytics to be ambiguous. This has been expressed in several comments and it is also highlighted in a report on learning analytics prepared by a working group at the University of Oslo (Langford et al., 2022). There is also less interest in learning analytics in higher education than in primary and secondary education and training.

As Universities Norway (2023) emphasises in its comments to the Expert Group; the fact that learning analytics is less prevalent in higher education than in primary and secondary education and training “may be due both to a lack of knowledge and access to tools and opportunities but also to traditions and culture” (p. 1). This difference in culture is also an issue student unions have raised with the Expert Group: They emphasise that a student is not – and should not be – a pupil. Whereas pupils in primary and secondary education and training have a number of rights related to formative assessment, differentiated instruction and close follow-up, students have traditionally had a far freer and more independent role, where they receive less continuous individual assessment. Although students in higher education and tertiary vocational education are also entitled to adaptations, participation and follow-up, this is usually ensured in other ways than in primary and secondary education and training.

3.3 What types of learning analytics do different groups need?

Both the performance of – and the need for – learning analytics appear to vary between primary and secondary education and training and training, tertiary vocational education and higher education. Below, we explain what needs learning analytics can cover for different groups from these educational levels. We also describe the purposes for which the actors themselves believe they need learning analytics.

3.3.1 Primary and secondary education and training

Pupils

According to section 2-3 of the Education Act, pupils shall be actively involved in the education. The general part of the National Curriculum emphasises that the pupils shall both contribute to and take joint responsibility in the learning community they create with their teachers every day. Pupils shall also participate in the assessment of their own work and reflect on their own learning and academic development pursuant to section 3-109 of the Regulations to the Education Act.

In order to achieve such active participation and pupil contribution, pupils require some knowledge of their own learning and academic progress. Learning analytics can help pupils gain more insight into their own learning processes and thereby better equip them to assess their own learning and take a position on issues that concern them in everyday school life. For this to be possible, pupils must understand what the representations of the learning process tell us.

In conversations with pupils, automated feedback is highlighted favourably if it supplements the feedback provided by teachers and is used with due care:

It’s good that the programs can give us feedback more often. After all, teachers don’t always have time for that. We only get feedback from big assignments, not small ones. I need feedback for small assignments just as much. It might be okay not to get it all the time, as it may be demotivating if I’m just making mistakes (pupil, grade 9).

The fact that the feedback is provided immediately is also highlighted favourably: “It’s often good to get the answers right away. Then you know whether you’ve done it right or wrong” (pupil, grade 7). Pupils have also given examples of the type of feedback that might be helpful to them: “It would be good if the programme could give you feedback and suggestions for further work. For example, it would be good if tips on regular mistakes popped up. For example, if I write æ instead of jeg” (pupil, grade 9). However, they have expressed concerns about whether such automated feedback distributed from a programme directly to the pupil could reduce human communication between pupils and teachers. Many also clearly express that automated feedback is of less value to them: “It’s much more true if the teacher says it. I wouldn’t feel happy if the machine just writes ‘good job’, I wouldn’t care. This is not an assessment. It is what is always there” (pupil, grade 7). The pupils also describe automatic adaptations as beneficial: “It is good that apps adapt to how good you are. There are some people who have received math for lower secondary school” (pupil, grade 7).

Parents

According to the Education Act, municipalities and county authorities have a duty to ensure cooperation with parents of pupils in primary, lower and upper secondary education. Parents’ attitudes towards school are of great significance for the pupils’ involvement and effort in school (Drugli and Nordahl, 2016). If parents receive information from learning analytics, it can help them support their child’s development and learning.

In its comments to the Expert Group, the Parents’ Committee for Primary and Secondary Education (2022a) states that learning analytics “may mean that pupils receive more frequent and accurate feedback on the basis of the data available to the technology and education services that are better adapted than what they are currently receiving” (p. 3). The committee also emphasises that the data on the pupil must be accurate, and that the technology is able to use the data correctly. Parents have also expressed to the Expert Group that, it is important to exercise a high degree of caution when using digital devices in education, especially with respect to the youngest children at school (Aas, 2023).

Teachers

Teachers in primary and secondary education and training shall regularly monitor pupils’ learning and adapt instruction as needed. Learning analytics can provide important insight into both academic development and how pupils are working on subject matter. For example, learning analytics can help adapt instruction, provide varied instruction and also document the instruction. Good learning analytics can also support teachers in their integrated assessment work that must be done continuously in the subject, e.g., by informing them of how each pupil has solved various assignments over time, or by providing an overview of how an entire group has solved the same assignment. There is also streamlining potential in learning analytics by analysing each pupil’s assignment solution automatically and simultaneously. This is significantly more efficient than having the teacher assess each of the same assignments individually. In a survey among the members of the Norwegian Association of Graduate Teachers (2023), immediate feedback to pupils was highlighted as the greatest advantage of learning analytics.

In its comments to the Expert Group, Union of Education Norway (2022) maintains that it is the teacher who is the most important factor for pupils’ and that it is therefore important to distinguish between methods that are wholly or partly aimed at replacing the teacher, and methods that can give the teacher a better basis for the pedagogical assessments. The Norwegian Association of Teachers (2022) believes that it is an open question as to what types of data will be capable of providing information with added value to teachers who “on a daily basis have access to richer data sources through communication with pupils in the classrooms” (p. 1). However, in the survey it conducted among its members, it was highlighted that learning analytics can help to more quickly identify pupils in need of additional support: “I find it to be a good tool to gain an overview of the pupil group, and I find that I can implement measures to assist pupils more quickly because I detect them faster” (Norwegian Association of Graduate Teachers, 2023, p. 1).

School administrators

School administrators have the overall responsibility for the quality of education at schools. Therefore, they have a considerable need for information about instruction and learning. Learning analytics can provide useful insight into a class or a grade’s academic progress in various subjects. Such data can contribute to the school’s quality development. For instance, a school administrator can monitor how a class or group of pupils develops over time, the current results in a class compared to other classes or previous cohorts, or the academic level in different classes in the same grade: “Teachers/school administrators will be able to have a more complete/unambiguous overview of pupils’ skill levels and can continuously measure/monitor pupil development. The frequency/rate of data collection will increase the accuracy of the measurement of the pupil’s development and skill level” (Norwegian School Heads Association, 2022, pp. 1–2).

Such results can also support school administrators in their work on quality development, e.g., where additional follow-up is needed, or if resources need to be allocated differently. The Norwegian School Heads Association also states in its comments that experiences with learning analytics among teachers and school administrators vary, and that there is a need for competence in good learning analytics with pedagogical value. Because school administrators have a responsibility for the pedagogical quality of their schools, school administrators also need competence in how learning analytics can be incorporated into school development.

School owners

School owners are responsible for ensuring that all schools have good pedagogical and administrative services for pupils, staff members and parents. Learning analytics can contribute to more evidence-supported quality development in schools and across schools, and it can offer school owners useful insight into schools’ practices and pupils’ learning. In a comment to the Quality Development Committee, school owners stated that they need to be able to compare data across the municipalities and county authorities (NOU 2023: 1). They also stated they need a timeline that follows cohorts of pupils throughout their education. To achieve this, school owners say that they require analytical tools that can guide them and support them in alignment, interpreting and analysing quantitative data and other information.

School owners need data that provide them with information at an overarching level. This applies to schools in the municipality and in the county authority (Skedsmo, 2022). The Norwegian Association of Local and Regional Authorities (KS) network Aggregated Management Data for Large Collaborating Municipalities has expressed that one problem with this is that it is difficult to access data at the municipal level (ASSS, 2022). In order to provide schools with sufficient support to analyse information and develop quality, municipalities want more access to the schools’ results than they currently have (NOU 2023: 1).

3.3.2 Higher education and tertiary vocational education

Students

Having an overview of one’s own academic development can support students in higher education and tertiary vocational education with respect to progress in their study programme. The Organisation of Norwegian Vocational Students (2022) states in its comments to the Expert Group that few students are currently familiar with the concept of learning analytics. Vocational college students therefore have little experience of how learning analytics can affect learning, even though there are student groups at vocational colleges who are more aware of this in their everyday studies, particularly online students. Despite the somewhat low interest among students to date, ONF believes that by collecting data from teaching situations, educational institutions can identify what challenges individual students and the class as a whole have with regard to learning.

In individual learning, it may be beneficial to be made aware of one’s own routines. This can be done by gaining access to your own data, e.g., from task solving, time use and the like. This can enhance your reflection on your own learning process. Botnevik et al. (2020) found that students are willing to share personal data if this benefits them in the form of better marks, improved instruction or a better learning experience. The student unions, for their part, have expressed concern that the analysis of the data may be so instructive that there is no room for self-reflection and self-assessment, or that it invades privacy. The Organisation of Norwegian Vocational Students (2022) states that the use of the analyses must not be at the expense of the formative aspect and the independence that comes with undertaking a longer education.

Instructors

Those who teach at universities, university colleges and vocational colleges have a clear and unambiguous responsibility to structure their instruction in a pedagogically justifiable manner. There is currently a shift towards increasing the use of coursework requirements and student evaluations in various courses, which underlines the desire for good student follow-up during the course of study. In this context, learning analytics can play a role: It can provide insight into how students relate to the subject matter and the status of their academic progress. Universities Norway (2023) notes that learning analytics offers many opportunities to gain additional knowledge about what produces good learning and how to adapt the education to each student: “It can be used by instructors who want to develop their own teaching practice and to support students’ learning processes in real time” (p. 1). In higher education, some of the instruction takes place in large groups in courses with limited opportunities for the instructor and individual students to interact. In this context, learning analytics can contribute to improved insight into and an overview of students’ academic development.

Currently, however, the data on learning that instructors in higher education and tertiary vocational education can obtain via digital resources remain very limited for the vast majority. Digital resources that teachers generally have available are what are referred to as Learning Management Systems (LMS). For the most part, such systems are used to provide an overview of which students are in which groups, facilitate the collection of assignments, share academic resources and facilitate communication and interaction. The need that digital learning platforms mostly appear to cover at present is the documentation of the instruction that has been provided. Learning management systems often contain resources such as slides from the instructor’s presentations, an overview of all the assignments students have received, student responses, assessments of these, and data that shows which students have visited certain resources at different times.

Administrators

Learning analytics can provide important documentation of how different courses are organised with far more detailed insight than a course description would typically provide. Learning analytics can also provide details about how students across courses relate to different resources, assignments and discussions shared on the learning platform (Misiejuk et al., 2023). Sikt (2022) notes that learning analytics can contribute to a “more holistic and coherent design of courses and study programmes” (p. 2). Universities Norway (2023) emphasises that institutions can use data at an aggregated level to make decisions on e.g. the purchase of digital solutions, procedures and legislation.

For programme coordinators in higher education and tertiary vocational education, data from learning platforms can be useful as part of the quality assurance work for individual courses, cohorts or entire courses of study and study programmes. It is precisely in the area of administrative and programme administration across courses that the use of data from learning platforms appears to be greatest at universities, university colleges and vocational colleges today. The extent to which such data are actually included in learning analytics is nevertheless difficult to ascertain. Sikt (2022) also proposes other sources of data that may be included in learning analytics in the future, such as curriculum systems, video services, plagiarism controls and the Common Student System (FS). Sikt further claims that, overall, such data can contribute to quality development in study programmes, among other things by providing new insight into patterns in the manner in which students relate to the various digital resources beyond the learning platforms.

3.4 Assessment of learning analytics in pedagogical practice

In order to obtain systematic knowledge about learning analytics in pedagogical practices in Norway, Rambøll has, on behalf of the Expert Group, assessed the scope, pedagogical practice, attitudes and challenges related to learning analytics. For capacity reasons, this assessment is limited to primary and lower secondary school, programmes for general studies in upper secondary education and higher education.

Rambøll’s research design consists of two different phases of data collection. The first stage was qualitative. During this phase, Rambøll conducted exploratory interviews with experts (3) and focus groups (14) on questions related to practice and attitudes and how the various groups actually speak about learning analytics. Rambøll interviewed teachers, administrators and instructors. In the next phase, quantitative methods were used to investigate the extent of, attitudes towards and variations and distribution in practice. In the quantitative section, the respondents were administrators and ICT managers in primary and lower secondary schools, university colleges and universities.

The quantitative survey was conducted between December 2022 and January 2023. Through a recruitment survey (primary and secondary education and training) and manual registration via contact information from the institutions’ websites (higher education), contact information was gathered from 878 schools and educational institutions. Of these, 625 were primary and lower secondary schools, 215 were upper secondary schools and 38 were university colleges or universities. Based on contact information from the 878 undertakings, Rambøll drew up a list of respondents where everyone was sent the final questionnaire. The survey was distributed electronically to 1560 recipients. There were 673 respondents to the entire survey, while 143 respondents answered some of the questions without completing the entire survey. In total, 43 per cent of the respondents completed the entire survey. Rambøll’s entire survey and a complete account of the methods used can be read in the report Digital læringsanalyse i norsk utdanning: omfang, pedagogisk praksis og holdninger [Learning analytics in Norwegian education: scope, pedagogical practice and attitudes] (Rambøll, 2023).

In the forthcoming review of knowledge regarding learning analytics in primary and secondary education and training and higher education, we combine the data from Rambøll with comments we have received, in addition to previous research from a Norwegian context. We have found that there are systematic differences in how primary and secondary education and training and training and higher education relate to learning analytics. In the forthcoming review, we have therefore chosen to distinguish between what characterises learning analytics in primary and secondary education and training and training and what characterises learning analytics in higher education. Where relevant, we also comment on learning analytics in tertiary vocational education. With regard to barriers to learning analytics in pedagogical practice, we have chosen to discuss this collectively for all levels of education as we see clear common features.

3.4.1 Challenges in assessing the use of learning analytics

Comments to the Expert Group show that the concept of learning analytics is not well established in Norwegian education (Parents’ Committee for Primary and Secondary Education, 2022a; Norwegian Association of Graduate Teachers, 2022; Organisation of Norwegian Vocational Students, 2022; Sikt, 2022; Norwegian School Heads Association, 2022; Universities Norway, 2023). The fact that the term is so unknown in the education sector has meant that it has not been easy to assess learning analytics in practice. We have found that the definitions of learning analytics that derive from the field usually do not correspond with the manner in which instructors, students, teachers, pupils, parents and administrators in the education sector understand the term.

Rambøll’s (2023) study also has a methodological problem as the respondents often seem to have a significantly broader understanding of the terms learning analytics and adaptivity than the definitions would suggest. The high proportion of don’t know responses to some questions indicates that many of the informants in the survey are uncertain about the terms learning analytics and adaptivity, or what the terms actually mean in practice. As Rambøll emphasises, there is a “discrepancy between the conceptual (in research) and contextual understanding of digital concepts in the field of practice” (p. 12), which creates validity challenges. Rambøll has attempted to resolve this both in the focus groups, by repeating and specifying what parts of digital practice were important to assess, and in the survey, by providing definitions of learning analytics and adaptivity in all questions that specifically asked about these terms.

With regard to the term adaptivity, the assessment indicates that respondents in the sector have a far broader understanding of adaptivity than the way it is defined by the field of research and the Expert Group. We note that the respondents in the survey state that they also use adaptive tools in subjects where scarcely any adaptive tools exist on the market today. There is reason to believe that the respondents interpret adaptivity as an adaptation that can be done apart from the tools, while the technical definitions of adaptivity emphasise that adaptive systems refer to automated, individual adaptations to the pupil’s situation using artificial intelligence. In reality, a number of the programmes that respondents in Rambøll’s survey refer to as adaptive do not use artificial intelligence to adapt their content with the aim of supporting pupils’ or students’ learning. We believe this is an important finding in itself: The education sector is not well acquainted with terms such as learning analytics and adaptivity – a point the sector itself also conveys in its comments.

We are not surprised that the respondents are uncertain about the terminology and we emphasise that we fully understand how demanding it can be to navigate all the terms related to various digital solutions. Although we have not systematically evaluated all products on the Norwegian market today, we note that the rhetoric used to market much of today’s learning technology is characterised by an imprecise use of terminology, excessive optimism and promises that are difficult to keep (Egelandsdal et al., 2019) – which is a well-documented and long-lasting trend internationally (Cuban, 2001; Selwyn, 2022).

We also emphasise that the complexity of learning situations with digital resources makes it difficult to compile an exhaustive overview of resources that contain functionalities for learning analytics. As can be seen from the comments we have received from FIKS (2023), the use of licensed resources constitutes a limited part of the overall teaching directed at pupils. This applies in particular to the higher grades of primary school, lower secondary school and upper secondary school. Office support tools such as word processors and presentation tools play a major role across subjects and grades. In such programmes, digital footprints are also left behind, although there is little evidence that such data are used systematically today to track and make pupils aware of their progress in writing and presentation work (FIKS, 2023). Despite the fact that it is not common to include such programmes in discussions on learning analytics in a Norwegian context, we see that this is where the largest volumes of data on pupils’ work is actually found in most schools and in most subjects. We should note that it was precisely such resources that many of the teachers in Rambøll’s survey associated with their digital practice.

3.4.2 What characterises learning analytics in primary and secondary education and training?

Below, we review what we know about learning analytics in primary and secondary education and training. As mentioned, one challenge for the Expert Group is that there is very little prior research to build upon in this knowledge review. Furthermore, it is challenging for both the sector and the participants in Rambøll’s assessment to distinguish learning analytics questions from general questions about digitalisation. Therefore, we have chosen to review the practice highlighted by the informants in the assessment and the extent to which it allows for learning analytics.

Tools and resources

Rambøll’s assessment provides information about the specific resources used in learning analytics at the respondents’ schools. At the primary and lower secondary level, it is the larger publishing companies that are behind the most used resources. Many schools report that they use Salaby (Gyldendal) (68 per cent), Skolen (Cappelen Damm) (54 per cent) and Skolestudio (Gyldendal) (48 per cent), which are fully digital teaching aids in subjects. Well over half of the schools (67 per cent) report that they use Conexus Engage, which is a tool that can provide an overview of data from various examinations and tests such as assessment tests, standardised assessments and digital learning resources. In addition, there are a good number of individual apps in use to varying degrees, such as Multi Smart Practice (44 percent) and Dragonbox (22 percent) in mathematics, the reading tool Aski Raski (41 percent), the vocabulary game Captain Morf (4 percent) and other similar apps.

The resources most upper secondary school respondents report using are subject-specific resources such as Campus Increment (49 per cent) and Kikora (35 per cent) in mathematics, and Duolingo (24 per cent) for language learning, or resources where the teacher can enter content, such as Kahoot! (81 per cent), Quizlet (46 per cent) or WeVideo (2 per cent). Rambøll’s study does not cover various vocational programmes and here there is also minimal prior research. GrunnDig’s final report explicitly calls for more research on digitalisation in vocational education (Munthe et al., 2022).

To sum up, we can divide the resources that primary and secondary education and training uses today into these five main categories. We emphasise that this is not intended as an analytical framework, but rather as a practical overview of the types of tools and resources that are prominent in primary and secondary education and training practice today, beyond office support tools.

1. Teaching aids: These resources are customised to different subjects and grades and have been developed in line with the Norwegian curricula. The teaching aids often have ready-made assignments related to specific subject areas in the form of e.g. multiple-choice questions to which the pupils can receive feedback. Digital teaching aids can cover many subjects, such as Skolen, Skolestudio and Aunivers, or individual subjects, such as Multi Smart Øving (mathematics).

2. Learning apps: These apps tend to focus on specific skills or parts of a subject area – often in language and mathematics. Such resources include Duolingo, Dragonbox, and Kaptein Morf.

3. Administrative tools and learning platforms: These are tools and platforms for digital interaction and for registering submissions and absence, as well as structuring courses. Examples include MS Teams, Itslearning and Visma InSchool.

4. Analysis tools: These are tools that collect data from other sources and align them for analysis and visualisation of both individual learning and of aggregated data. Examples include Conexus Engage and Conexus Insight.

5. Various question tools: These are tools where the teacher can create questions, content or assignments. Examples include Kahoot! and Quizlet.

Subjects and grades

Rambøll’s assessment shows that adaptive resources are most frequently used at the primary school level and in mathematics subjects. The reason for this, according to the informants, is that parts of the mathematics subject are well suited for creating assignments with predefined answers. It is also noted that there is a need for volume training in this context: “There is very little use of learning analytics in the higher age levels and in more reflective, creative and physical subjects […]” (p. 15).

Pedagogical and administrative practice

A significant finding in Rambøll’s assessment is that it appears that the use of digital resources in primary and secondary education and training is primarily driven by a desire to create an appetite for learning and a sense of proficiency among the pupils. The informants who were interviewed stated that the challenge for them is to motivate the pupils, not that they lack insight into the pupils’ academic level. We believe this is a finding worth noting, because it indicates that the perceived need for learning analytics in schools is not necessarily substantial. The assessment also indicates that the adaptive resources and information from learning analytics are to a limited extent used to adapt education and adjust instruction.

The report reveals that the most widespread form of learning analytics in primary and secondary education and training is the one that is both easiest and most accessible, i.e., the one that has an administrative purpose. We are referring to analyses of absence and marks (for grades where marks are given), either at the aggregate level for classes, grades, or entire schools. Furthermore, Rambøll finds that teachers tend to look at summaries of a more practical nature, such as how many individuals have performed a particular task, or how the results in a class or grade are distributed. They are significantly less concerned with the fact that learning analytics can provide more detailed insight into the pupils’ academic progress. This is also true for schools that often use adaptive resources.

Assessment of pupils – formatively and summatively

In Rambøll’s assessment, the informants were asked a number of questions to reveal whether information from learning analytics and adaptivity is used to assess pupils, both formatively and summatively. Several factors emerged that we consider significant. First, it is clear that the respondents have a broader understanding of the term adaptivity than is common in the field of research, as both the focus group interviews and the survey reveal. Second, many administrators in primary, lower and upper secondary school respond that they do not know whether or how often teachers use information from adaptive resources in feedback to pupils. Although few adaptive resources are used as a basis for formative assessment, we can characterise the extent as limited in upper secondary school and somewhat broader in primary school and lower secondary school. The extent to which adaptive resources are used also varies between schools. Several of the schools, in primary, lower and upper secondary schools state that they never use adaptive resources.

With regard to summative assessment and marking, the percentage responding don’t know is also high. When asked to what extent information from learning analytics is used to inform marking in various subjects, the percentage of don’t know responses is between 17 and 25 per cent for subjects in primary and lower secondary school and 39 and 45 per cent for upper secondary school. We find it concerning in itself that the informants (in this case school administrators) respond that they do not know whether information from learning analytics is included in the summative assessment of the pupils or when setting marks. However, it is conceivable that the responses would have been somewhat more precise had it been the teachers themselves who had responded to this question. In any case, the findings paint a picture of little common practice and considerable uncertainty about the actual uses of pupil data.

3.4.3 What are the characteristics of learning analytics in higher education?

Below, we review what we know about learning analytics in higher education. The sector is very diverse and its use therefore varies between institutions and subject areas.

Tools and resources

In Rambøll’s assessment, Canvas (78 per cent) is the tool most respondents in the university and university college sector state that they use, but tools such as Kahoot! (59 per cent), Mentimeter (56 per cent) or Inspera (53 per cent) are frequently reported. Again, we must remark that it appears that the respondents understand questions about learning analytics much more broadly than the academic definition, because they also mention certain digital tools that do not currently have a functionality for learning analytics. For higher education, it was also clear in the focus groups that specific questions about learning analytics were quickly met with general answers about learning platforms and assessment tools.

Both the assessment and comments from the sector emphasise that it is primarily learning platforms such as Canvas and Blackboard that are used. These are systems that can allow for some learning analytics, but in Canvas, for instance, these are additional modules that have to be enabled. It does not appear that many people do so. Several representatives of the sector have told the Expert Group that the procurement of learning management systems occurs without emphasising functionality for learning analytics. In practice, this means in places where learning analytics are currently utilised in university and university college sector with the aid of the learning platform, the platform was actually acquired for purposes other than learning analytics.

Administrative use of learning management systems

Learning analytics in higher education is primarily related to the administrative use of learning platforms, as Rambøll’s assessment clearly shows. The reasons for this are not easy to determine with certainty. It may be that they do not know what learning analytics is and what added value it can provide in instruction. It may be that the legal framework is perceived as ambiguous and that this prevents a number of functionalities from reaching the instructors or places of study. It may also be that they do not find that they need access to the students’ academic progress during the course of study.

Meetings we have had with representatives of higher education indicate that there is considerable uncertainty about the legal basis for collecting personal data and that the centralised schemes for approving tools that can be used by all staff members at a given university would limit access to learning analytics. We are aware of individual examples of more systematic learning analytics in higher education (BI Norwegian Business School, 2023), but there are so few of them that they are not included in this review of what typically characterises learning analytics in higher education in general.

When so much of the use of digital resources for which we receive comments from higher education takes place on learning platforms, it is timely to ask what kind of information about learning and instruction can be gathered there. Part of the reason why learning analytics is limited when exclusively retrieving data from learning platforms is that such platforms are not intended to develop or measure students’ academic benefits or progress. In fact, most learning platforms are in practice a collection of academic resources, practical notifications and submissions of various kinds (Lester et al., 2018). A study of how various academic communities used Canvas before, during and after the COVID-19 pandemic shows that the use of functionalities such as discussions and quizzes has increased. However, the study also found considerable variation between different subjects (Misiejuk et al., 2023).

Assessment of Canvas data

In 2020, a working group explored the use of Canvas data in higher education (Unit, 2020). The ambitions were to assess what kinds of data exist in Canvas (variables, interfaces, format, storage time, etc.) and to initiate a discussion about what we want to achieve via Canvas data collection and how such data can be viewed in connection with other data initiatives (Sikt, 2022). The Expert Group notes that the working group reports that it has “found it challenging to identify the needs/wishes for learning analytics/data analysis among staff members, as they do not understand or know what the possibilities are” (Unit, 2020, p. 1). The working group writes that the academic staff are unfamiliar with learning analytics and that this makes it difficult for them to perceive the opportunities and describe the needs. At the same time, technical-administrative staff lack the pedagogical and contextual understanding to be able to “communicate the opportunity space” (p. 1).

3.4.4 Barriers to learning analytics

Rambøll (2023) reveals that the data collected on pupils’ and students’ learning activities is currently primarily used for administrative purposes – not to follow up on individual students or improve instruction. In other words, most of the use is not encompassed by the definition of learning analytics. An important matter is then to identify the barriers to learning analytics in pedagogical practice to enhance learning and improve instruction. In Rambøll’s study, the question of barriers was connected to the use of adaptive resources.

Figure 3.1 shows several factors that are also highlighted in various comments received by the Expert Group on general challenges with learning analytics. Below, we review the barriers we believe are most significant for primary and secondary education and training and higher education.

Zoom in on image

Source: Rambøll (2023)

Time for familiarisation with relevant resources

As figure 3.1 shows, many respondents in primary, secondary and higher education find one barrier to learning analytics is not having the time to learn how to use new tools. In the assessment, a clear finding is that the respondents find that the pedagogical use of adaptive learning resources and learning analytics largely depends on a certain commitment. Those who wish to make use of such resources have to spend their spare time familiarising themselves with and adopting the solutions as there is usually no time set aside to test them during a workday. If a teacher, school administrator, instructor or programme coordinator does not see any direct added value in learning analytics, they are naturally not willing to invest time in familiarising themselves with specific resources.

The final report of the GrunnDig project emphasises that a small number of the teachers in the study programme on their own initiative seek knowledge about the use of digital resources they can use in their instruction (Munthe et al., 2022). Many teachers who participated in GrunnDig’s survey also report that they prefer to see other teachers try out new technology before using it themselves. This underlines the importance of the professional environment and the role of management.

Employees’ digital competence

Digital technology is the area where most Norwegian teachers say that they need continuing education and training (Throndsen et al., 2019). In GrunnDig’s final report, it is emphasised that teachers in primary and secondary education and training are largely supportive of digitalisation (Munthe et al., 2022). However, it also appears that they are dependent on good support and guidance in terms of local development work and sharing of experiences before they adopt new technology. Rambøll’s survey also confirms that employees perceive insufficient digital competence as an obstacle to performing learning analytics in schools.

With regard to higher education, the 2020 report on the state of higher education in Norway shows that nearly half of the faculty staff at universities and university colleges state that they have not been offered training in the pedagogical use of digital technology, while slightly more than half believe that they need more training (Berg et al., 2020). Although many of the educational institutions offer instruction and learning using technology, such offers are not mandatory – and not everyone has emphasised learning in technological environments in their compulsory offer of basic competence (Ørnes et al., 2021).

The Norwegian Agency for Quality Assurance in Education’s (NOKUT) (2017) inspection of vocational colleges revealed that many vocational colleges fail to clearly describe the digital competence of the academic community, and that, where it is described, they mention competence in the use of tools rather than a more pedagogical and didactically oriented digital competence.

The Expert Group also notes that teacher training programmes have been largely unsuccessful in integrating digital competence into their study programmes and that there are major differences between educational institutions in terms of the type of training the students receive (Gudmundsdottir and Hatlevik, 2018; Instefjord and Munthe, 2016). This is something we also discussed in the first interim report. It is highly concerning that half of newly qualified teachers in one of the relatively few study programmes in this field state that they have a low level of digital competence (Gudmundsdottir and Hatlevik, 2018). This constitutes a significant barrier to good learning analytics in Norwegian schools.

Inadequate interaction between digital resources

Rambøll (2023) identifies it as an obstacle that various digital resources are inflexible and cannot necessarily be combined with other digital resources. Several stakeholders have expressed to the Expert Group that this is a challenge. Among other things, Vestfold and Telemark County Authority (2022) write that it is the market and system suppliers in primary and secondary education and training that largely define the technical instruments and how they can be used in practice. It becomes difficult when the various resources require separate logins, operate with different ways of assessing pupils and collect little information about pupils’ learning. The fact that the resources appear to such a large extent as closed systems makes it more challenging for schools to utilise the access they may have to various resources. Research on different digital platforms in higher education also shows that there are limited opportunities to combine data across different digital resources (Samuelsen et al., 2019) and that various data are stored in different formats that do not adhere to common standards – and which are therefore difficult to align (Samuelsen et al., 2021).

Lack of relevant learning resources

The Expert Group’s first interim report, previous research, Rambøll’s survey and comments received by the Expert Group all note that the current Norwegian market for primary and secondary education and training lacks good learning resources that facilitate learning analytics.

We know that most such learning resources for primary and secondary education and training are found in the subjects of mathematics and languages, while barely any resources exist for a number of other subjects. The Norwegian Association of Graduate Teachers (2022) explicitly states that a lack of resources in upper secondary education is a barrier to good learning analytics. They point out that fewer digital teaching aids are being developed for upper secondary education because the market is not viewed as large enough as long as extensive funding is tied to the Norwegian Digital Learning Arena (NDLA)10.

We have also found that there are currently very few digital learning resources for both Norwegian language forms. The right of Sámi pupils to receive instruction in Sámi is also something that is challenged by the market-driven supply of digital teaching aids in schools. Save the Children Norway (2023) expresses concern that there is insufficient development of digital learning resources that ensure universal design:

Universal design in schools shall promote inclusion, equality and equal opportunity for all pupils to participate in the instruction, in social activities and everything that happens at school […] Save the Children Norway is concerned that digital learning resources are increasingly being developed that do not meet the requirement of universal design, and this is therefore contrary to Norwegian law. (p. 6)

When schools report a lack of resources, it may be that the resources only cover parts of the subject but it may also be that there is a lack of resources with universal design, resources in Nynorsk or resources in Sámi.

In higher education and tertiary vocational education, there are even fewer opportunities for learning analytics in individual subjects. One explanation for this is that it is difficult for the market to offer customised solutions for individual courses. The way specific courses are structured varies depending on the institution and a number of courses are unique to their educational institution and in terms of course description. Furthermore, they do not have as clearly defined learning objectives as in primary and secondary education and training. This makes it challenging for developers to design learning analytics solutions for individual courses in higher education and tertiary vocational education. Therefore, there are few genuine opportunities for learning analytics within individual courses at these levels, even though there is general functionality for learning analytics.

The Expert Group finds that a lack of learning resources is a significant barrier to good pedagogical practice involving learning analytics. This will continue to be the case in the future if the market-driven tendency is allowed to dominate.

Lack of guidance from management

It comes as no surprise that a lack of guidance from management is a barrier to learning analytics in primary and secondary education and training and higher education. Previous research has also noted that the decision to use (or not use) digital tools and learning resources is too often at the discretion of the individual teacher (Gudmundsdottir and Hatlevik, 2018). Rambøll also finds a connection between a lack of guidance from management and a lack of systematic learning analytics, particularly in upper secondary education.

We believe that the lack of guidance from management with respect to learning analytics should be viewed in light of the strong autonomy enjoyed by Norwegian teachers and instructors. There is no tradition in Norwegian schools for management to control how teachers conduct their instruction and follow up their pupils (Mausethagen and Mølstad, 2015; Mølstad and Karseth, 2016). At the same time, this barrier indicates that if the school administration or programme administration wish to have more learning analytics, they must actively facilitate this and support their teachers and instructors in including learning analytics in their pedagogical practice.

The teachers and school administrators in the GrunnDig project agree that good support structures are crucial for teachers to develop in a digital school (Munthe et al., 2022). The fact that so many school administrators in Rambøll’s survey respond that they do not know whether learning analytics is included in various pedagogical practices, indicates that school administrations are often also unclear as to whether schools should have more learning analytics. The Expert Group also notes that this, i.e., the lack of guidance from management, is perceived as greatest barrier in higher education. This may indicate that it would be wise to prioritise the formulation of local guidelines for learning analytics at institutions that can inform educational institutions of their learning analytics needs.

Lack of connection to the National Curriculum

One important finding in Rambøll’s survey is the fact that various digital resources with learning analytics are not explicitly linked to specific competence aims, which is a barrier for respondents in primary and secondary education and training, particularly in lower secondary and upper secondary education. This means that teachers themselves must assess in which parts of the National Curriculum learning analytics can help shed light on the pupil’s competence and academic progress. This barrier can be exacerbated by the above-mentioned problem that teachers do not have enough time to familiarise themselves with new tools.

Possible reasons why the proportion of teachers at the primary level are less concerned with links to the National Curriculum are both that they have significantly more tools to choose from at the primary level and that teachers are not required to mark their pupils’ academic performance.

Uncertainty regarding data protection

It is not a new phenomenon that teachers receive a lot of information about their pupils during their schooling, but what is new in terms of using digital resources is the extent of information and that it is stored and becomes part of the pupil’s digital footprint. The use of digital resources is increasing, as is the proportion of digital data that is continuously collected about each individual. This has led to a growing interest and concern about whether schools are protecting the privacy of pupils and about the increased legalisation of the field. The Norwegian Privacy Commission highlights this tendency in its work (NOU 2022: 11). A number of comments we have received confirm that uncertainty about data protection is a major barrier to learning analytics in education.

ICT Norway (2023) emphasises that there is uncertainty among school owners regarding legislation – which means that different school owners arrive at contrasting conclusions on data protection impact assessments – and that the requirements for addressing data protection in procurements are unclear. We have repeatedly been made aware that school owners find that they spend a disproportionate amount of time assessing the data protection implications of using digital learning resources. There is also a considerable fear of making mistakes and using resources that violate pupils’ right to privacy.

The uncertainty surrounding data protection is perceived as even greater in higher education than in primary and secondary education and training. Many cite this as an obvious explanation for why learning analytics in higher education more or less exclusively involves administrative tools and analyses. In its comments to the Expert Group, Sikt (2022) has stated that an unclear legal basis is a barrier to collecting information and conducting learning analytics.

Stable internet access

Although the vast majority of schools and educational institutions should have good access to the internet and good digital infrastructure, Rambøll’s survey shows that grades 1–10 in particular cite lack of access to stable internet as a barrier. There are fewer respondents in upper secondary education – and even fewer in higher education – that report this issue but it is of course a problem for the few institutions that do report a lack of stable internet access. Other national surveys also show that although Norway is generally a highly digitalised society, there are differences in the extent to which schools experience having a sufficient digital infrastructure (Vika et al., 2021). In its comments to the Expert Group, the Norwegian Association of Graduate Teachers (2023) explicitly mentions that lack of internet access constitutes a barrier to learning analytics in today’s schools:

At some schools, teachers have to plan two lessons for each period, one that involves internet access and another that does not, as the school’s network is often down. In such settings, a guideline or ambition to use learning analytics will only be met with a shrug. (p. 4)

Costs

In order for schools and educational institutions to be able to use license-based tools that facilitate learning analytics, it is essential that they have the means to acquire such licenses and that the market for development is sustainable.

As noted by ICT Norway (2023), there are currently major variations in the amount of money municipalities allocate to purchase digital teaching aids. The variation is both national and local as there may also be differences between schools within a municipality in terms of the budget for purchasing relevant resources. ICT Norway emphasises that schools need predictability in order to make good use of learning analytics: “School administrators must also have good and stable financial framework conditions, with a high degree of predictability, to ensure a wide range of digital teaching aids so that all pupils can receive the adapted instruction and follow-up they need” (p. 5).

3.5 Summary

One important point with respect to all digitalisation of education is that realising the potential of technology is never a given (Lund, 2021; Solomon, 2016; Selwyn, 2022). As GrunnDig’s final report also emphasises in its review of research on digital classrooms, we do not always know whether the potential of digitalisation “is actually a potential or just an imaginary potential” (Munthe et al., 2022, p. 10). The reason we emphasise this is a crucial premise for understanding current practice related to learning analytics: There is not necessarily any correlation between the amount of data on learning collected and the systematic use of such data in learning analytics. Although we have never before had so many digital footprints of pupils’ and students’ academic activities as we have now, there is little to suggest that such data are systematically included in learning analytics. A clear finding in the report from Rambøll, which is also confirmed by comments received by the Expert Group, is that widespread use of digital resources does not necessarily mean that the analytical potential of the data collected is being realised. Despite the widespread use of digital resources, few actors in the sector are interested in the analysis opportunities.

Rambøll (2023) summarises the findings from the qualitative focus groups by stating that “learning analytics is something one wants but does not feel that one needs” (p. 27). Those with more enthusiasm for its use are often interested in the possibilities for adaptation in the subject of mathematics. However, it is noted that this interest is as much about future opportunities as it is about the opportunities offered by today’s solutions. This is something we have also experienced in input meetings – the enthusiasm shown for learning analytics is not about today’s learning resources, but about the opportunities that lie ahead.

It is in primary and secondary education and training that learning analytics influences the practice the most, but here too the scope is limited and often also limited to certain subjects. In higher education, the use of digital resources is limited to the use of various administrative tools. Furthermore, the legal basis for learning analytics is perceived as unclear and there is a fear of making mistakes. Barriers to learning analytics largely concern time, competence and lack of guidance from management. Respondents to the Rambøll survey also reported barriers related to a lack of good learning resources, ambiguous connections to the curriculum, inadequate interactions between the various resources and uncertainty related to data protection.

4 Data types and data quality in learning analytics

One prerequisite for all learning analytics is the access to relevant data that has the potential to provide us with insight into learning and instruction. The quality of the data is always crucial to the quality of the insight the data can provide. In this chapter, we will take a closer look at what types of data are relevant for learning analytics and what is meant by good data quality in connection with learning analytics.

4.1 What is data?

There are many different definitions of data depending on one’s perspective. Data is often perceived as “a way of storing, transmitting and processing information in the form of a specific data format”11.

In this context, we are primarily concerned with data that can be included in learning analytics. During a typical day, many pupils and students use learning platforms, apps and programmes. Such interactions with digital devices create digital data. Virtually everything we do on digital devices leaves traces and generates data. For example, digital data are created every time a pupil taps the screen in a language app, or every time a student watches an instructional video. In addition to such traces, digital data can be based on analogue signals from, e.g., sensors, which are then digitised. When such data are included in the analysis, it is referred to as multimodal learning analytics (Giannakos et al., 2022).

To find the best possible starting point for interpreting the data, we rely on information about the data itself and the way it was collected. Metadata is often described as “data about data” and provides descriptive information about the data we have. An example of metadata would be the date a digital photo was taken or when a particular document was created or was last modified. The context in which the data was collected is also important for interpreting the data. For learning analytics, the pedagogical context will be relevant, such as whether an assignment pupils have written was a collaborative task, or what kind of instruction students received just before taking a particular multiple-choice test.

4.1.1 Data viewed from different perspectives

From a technical perspective, different types of data must be stored in a database or file system in order to be analysed by statistical software or algorithms. The data are then stored in a data format that is readable and understandable to the software in a computer. Data formats include:

• Numeric (integer or decimal)

• text (e.g., plain text, html, xml)

• audio (e.g., WAV, AIFF)

• visual (e.g., images such as JPEG, PNG, TIFF, or video as MPv4)

• instrument-specific (e.g., biosensor, gaze tracker, motion sensor)

From a technical perspective, the following technical terms about data are also important:

• Metadata is data about data, or data that defines or describes other data (e.g., the time the data was recorded, the type of camera used to take a picture, or the textbook from which the data in question originated).

• Multimodal data are combinations of different modalities (e.g. text, image, sensor data, gaze tracking data).

• Dataset is a structured collection of data (e.g., consisting of student number, mark and time spent) or an organised collection of unstructured data.

From an analytical perspective we can categorise data as follows:

• raw data (unprocessed data recorded and collected but not acted upon) or processed data (data that has been manipulated, e.g., by turning it into a format that allows for visualisation, and alignment and comparison with other data)

• real-time data (data that is presented to the user as soon as it is recorded) or historical data (data recorded at an earlier point in time)

• structured data (data organised and defined according to specific rules, which is necessary for exchange and interaction), unstructured data (unorganised data) or semi-structured data (a mixture of structured and unstructured data)

From a practical perspective we can refer to the following:

• raw data, e.g.,

  • content data (deliberately created by humans, e.g., when providing personal information to create an account to use an app or upload a video on a platform) or

  • sensor data (data recorded by a sensor, such as when your movements are recorded by a smartwatch)

• analytical data (processed data created by machines following human-machine interaction)

• functional data (data created by a machine to enable communication between machines)

From a learning analytics perspective, we often use the term activity data (Kay and Harmelen, 2014). Such data are defined as traces of human action in the electronic or physical world that can be detected by a computer or digital device. The term activity data encompasses visible raw and analytical data and invisible functional data. The different types of data are also reflected in the definition from the report Å lykkes med åpenhet [Succeeding with Openness] where activity data from adaptive teaching aids is described as “the information that is created when a pupil performs tasks in a learning tool. This may be the pupil’s answer to an assignment, information about what assignment the pupil has done, how long the pupil spent on the task and whether the pupil answered the assignment correctly or incorrectly” (Norwegian Data Protection Authority, 2022c, p. 3). Metadata are also generated about the situation where the data are collected, such as what kind of digital device is used or which Feide [centralised identity management solution for the education sector] ID12 is logged in.

From a data protection perspective, data are referred to as personal data when they can be used to identify a person, either directly or indirectly. This includes data such as name, address, date of birth, telephone number, email address, national identity number, passport number or other identifiers that are unique individually or in conjunction with other data. In learning analytics, personal data may be collected from pupils, students or others to analyse and enhance learning and improve instruction. However, it is important to ensure that personal data are processed in a lawful, responsible and ethical manner and that appropriate technical, administrative or rights-enhancing measures are in place to comply with the requirements of the GDPR in order to protect the rights and interests of individuals.

Far from all forms of learning analytics must be able to identify an individual. Data is often aggregated at the group or organisational level. Aggregated data is data at a higher level that is obtained by combining data from an individual level. The management level, administrators and researchers use aggregated data for a variety of purposes. For instance, data can be used to assess the consequences of measures, recognise trends and patterns in processes and gain relevant insight into make strategic decisions. When data are aggregated to a group or organisational level, the identity of the individual is only used when data are aligned into larger datasets for analysis. The identity is not available in retrospective analyses.

4.2 Data used in learning analytics

The data from the education sector is generated from a wide range of sources. A systematic review of ten studies on data use in learning analytics in higher education in different countries shows that the most commonly used data are activity data, followed by data from the course management systems (e.g., students’ background information) and data from assessment (Samuelsen et al., 2019).

Being able to process data from multiple sources without losing the integrity of the data when aligning different data sets is important for scaling the learning analytics. In order to succeed in this effort, data and data sets must be available in standardised formats. This also ensures what is referred to as interoperability between the different applications that will work together, so that they can exchange data seamlessly.

We find some examples from recent years where learning analytics uses multimodal data (Di Mitri et al., 2017; Giannakos et al., 2019; Worsley et al., 2021). Multimodal data are typically collected from data sources that contain sensors, such as physiological signal bracelets and gaze tracking, but also audio and video. The use of sensor data in learning analytics is still at an early stage and faces technical challenges such as synchronisation and data integration (Samuelsen et al., 2019). In addition, there are a number of unexplored ethical challenges associated with using multimodal data, especially when there are many data sources involved (Worsley et al., 2020), or when data include health data (Martinez-Maldonado et al., 2020).

Data management

Before anyone can analyse the data, it must be organised and stored in a structured format that allows an application or algorithm to manage it. The most common data storage technologies used for learning analytics data are relational databases, files, spreadsheets, or what are referred to as “learning record stores,” such as the open-source solution Learning Locker13. In some forms of learning analytics, it is desirable to align data from different data sources. Such data can be stored in different formats and have different levels of structure, or it can be real-time data that is included in the analysis without being stored. In order to make data with different formats available for further management, international standards for data structures have been developed. xAPI14 and IMS Caliper Analytics15 are examples of such standards. If someone wishes to structure data from different data sources into an analysis, a unique identifier that can link the data from the different sources is needed. For example, it is possible to use a student’s Feide ID to connect data from different sources.

4.3 Data analysis

Data analysis can involve a variety of techniques and methods. Examples include statistical analysis, machine learning, data mining, and data visualisation. Data analysis can be used to identify patterns, trends and relationships in the data and to test hypotheses. The aim is to uncover and present useful information and support decisions. Developers can also use algorithms to automate such data analysis processes, reducing manual intervention and speeding up analysis.

Selection bias

A source of error that is particularly relevant for machine learning and artificial intelligence, and which is also relevant for learning analytics, is what is referred to as selection bias (Norwegian Ministry of Local Government and Modernisation, 2020). Selection bias may occur if the data sets used in the training of the algorithms only contain information about a part of the relevant data. This may lead to the results referring to associations where they do not exist, or not referring to associations where they actually do exist (Larsen, 2020). Thus, the algorithms may be less effective, or they may contribute to maintaining or reinforcing social biases based on, e.g., gender, background or socioeconomic status.

For learning analytics, this selection bias entails a risk that the learning analytics algorithms will contribute to maintaining and reinforcing existing inequalities and discrimination in education (Lester et al., 2019; Selwyn, 2022). To reduce bias in the learning analytics algorithms, it is important to carefully consider the data used to train the algorithms. This is also part of ensuring that the algorithms are regularly revised and tested for biases. Incorporating ethical principles for how learning analytics should be designed and how learning analytics should be implemented helps to promote fairness and equality in education.

4.4 Data quality

What an analysis can actually tell us about learning and instruction is always inextricably linked to the quality of the data we have available. Data quality is about whether or how well the data correspond to the situation or activity they represent. In other words, data quality is about ensuring correct, complete and current data. It is also necessary to ensure that data are not altered or manipulated, intentionally or unintentionally, in ways that affect the end result. The data must be complete, consistent, accurate, timely, valid and unique in order for it to be described as good data quality (Pipino et al., 2002). Of these six principles, it is often easiest to assess whether the data are complete and valid, and then whether it is timely and unique. The most difficult aspect to determine is whether the data are accurate and consistent.

Complete data

An important principle of data quality is complete data, meaning that no data are missing. In other words, all the data that one expected to collect is actually present. There are various reasons for receiving incomplete data, such as missing values or errors when the data are entered. Missing data or incomplete records can lead to skewed analyses, erroneous conclusions or inaccurate predictions. The steps to ensure the completeness of the data are having clear data collection procedures, validating the accuracy and consistency of the data, and cleaning the data in a manner that addresses missing values and incomplete records. In learning analytics, providing complete data is critical to gaining meaningful insight into pupil and student learning.

Consistent data

Consistent data involves collecting the expected versions of the data and ensuring that they do not contain contradictions or systematic irregularities. A simple example to illustrate this is if an instructor wishes to use information about when students performed a learning activity. If the dates when students participated are recorded in different date formats, the inconsistent entry leads to useless data. It becomes difficult to understand what the data really means, or to align it in meaningful ways.

Accurate data

Accurate data refers to the extent to which the data represents the real phenomenon or information they are intended to represent (construct validity) and how close the data values are to the true values of the underlying phenomenon (validity). Thus, accurate data is data that are correct, precise and represents what it is intended to represent. The procedure to improve accuracy involves quality assurance of data sources, verifying integrity and consistency and methods of cleansing and validating in order to detect and correct errors. In learning analytics, accurate data are critical to providing reliable and valid insight into pupil and student learning.

Timely data

Timely data means that the data must be collected and available at a time that allows for the appropriate use of the information. In pedagogical contexts, this often relates to the proximity of the data to the learning situation. If the data from a given learning situation is not available at the right time, the teacher, instructor or others will not be able to use the information to improve the pupils’ or students’ education.

Valid data

Valid data means that the data provides information in accordance with its intended purpose. If the goal of a learning app, such as a multiple-choice quiz, is to provide information about what pupils know about a given academic topic, and the pupils realise that the longest answer is always correct and therefore always choose this option regardless of the content, the data will not be valid. The data will then not be a valid measure of a pupil’s academic insight into the topic.

Unique data

Unique data simply means that the data must not be recorded more times than it should, i.e., that duplicates are avoided.

5 Legislation relevant to learning analytics

Parts of the legislation that are relevant to learning analytics are basic standards at a general level, such as the Constitution of Norway and conventions. Other parts of the relevant legislation are intended for specific areas. An example of this is the data protection legislation that apply to the processing of personal data. Otherwise, the sector-specific legislation in the field of education plays a key role in regulating learning analytics.

In this chapter, we describe the parts of the legislation that applies to learning analytics. This includes relevant provisions in the Constitution of Norway, the UN Convention on the Rights of the Child and the European Convention on Human Rights (ECHR). Next, we provide an account of general legislation relevant to learning analytics, such as data protection legislation and sectoral legislation. At the end of the chapter, we describe the ongoing work in the EU and in the Council of Europe on general regulation of artificial intelligence.

5.1 The Constitution of Norway and human rights conventions

The provisions of the Constitution of Norway and the human rights conventions may have several functions. With regard to the legal function of the Constitution of Norway, the provisions can firstly establish the frameworks for what is lawful. For example, the legislature cannot ignore the needs of children when drafting new laws. Second, the Constitution of Norway can act as an interpretation factor when interpreting other legislation. Third, the Constitution can serve as a guide in connection with legislative and other policy development.

5.1.1 The right to education

Constitution of Norway

The right to education was incorporated into the Constitution of Norway in connection with the 2014 constitutional revision. At the time, the Norwegian Human Rights Commission found that the provision would not alter the state of the law because the Commission assumed that the Education Act that was force at the time and the Act relating to universities and university colleges were in accordance with the international human rights conventions (Document 16 (2011–2012), section 37.5.1). Article 109 of the Constitution reads as follows:

Everyone has the right to education. Children have the right to receive basic education. The education shall safeguard the individual’s abilities and needs, and promote respect for democracy, the rule of law and human rights.

The authorities of the state shall ensure access to upper secondary education and equal opportunities for higher education on the basis of qualifications.

The right to education is a right in itself, but also a prerequisite for the realisation of other human rights. The wording “safeguard the individual’s abilities and needs” emphasises that the education should not only take place on society’s terms (Document 16 (2011–2012), section 37.5.2.2).

The Constitution of Norway also stipulates that everyone should have access to upper secondary education and that this right applies regardless of qualifications. On this point, the Constitution goes further than what is enshrined in the international conventions. The constitutional provision also contains a duty on the part of the State to facilitate higher education where abilities and qualifications are the determining criteria for access.

The right to education in international conventions

According to Article 2 of Protocol 1 to the ECHR, no person shall be denied the right to education. The ECHR has the force of Norwegian law with the adoption of Human Rights Act of 1999, section 2, first paragraph. If there is conflict between the Convention and Norwegian law, the Convention shall take precedence pursuant to section 3. Norwegian legislation and regulations must therefore comply with the frameworks established by the Convention obligations.

Article 13 of the International Covenant on Economic, Social and Cultural Rights (ICESCR) stipulates that states shall recognise the right of everyone to education. The provision contains broad objectives that “education shall be directed to the full development of the human personality and the sense of its dignity”. Article 13 also states that primary education shall be compulsory, accessible and free to all. Furthermore, higher education shall also be equally accessible to all, on the basis of capacity. Article 28 (education) and Article 29 (objectives of education) of the UN Convention on the Rights of the Child contain similar wordings to those of the ECHR and ICESCR. Both the ICESCR and the Convention on the Rights of the Child have the force of Norwegian law pursuant to Section 2, second and fourth paragraph. A main feature of the conventions is that they grant children a right and a duty to education. The needs of individuals shall be safeguarded, in addition to the authorities facilitating higher education.

5.1.2 The right to privacy

Constitution of Norway

In 2014, the right to privacy was incorporated into Article 102 of the Norwegian Constitution, the provision reads as follows:

Everyone has the right to the respect of their privacy and family life, their home and their communication. Search of private homes shall not be made except in criminal cases.

The authorities of the state shall ensure the protection of personal integrity.

The establishment of the right to privacy in the Constitution of Norway did not constitute a change in the state of the law but was intended to reflect the essence of the international human rights provisions and contribute to highlighting the right to privacy through a principled provision in the Constitution (Document 16 (2011–2012), section 30.6.5). The provision does not mention whether interferences in the right to privacy may be permissible, nor anything about the conditions under which a possible interference may occur. The provision in the Constitution of Norway is related to the principle of legality in Article 113 of the Constitution, which expresses the key principle that “[i]nfringement of the authorities against the individual must be founded on the law.”.

When the Storting’s Standing Committee on Scrutiny and Constitutional Affair considered the proposal, the Committee stated that “the proposal shall be read as meaning that systematic collection, storage and use of information about the personal affairs of others may only take place in accordance with law, be used in accordance with the law or informed consent and erased when the purpose no longer applies” (Recommendation to the Storting No. 186 (2013–2014), section 2.1.9). In addition to the fact that the interference must be founded in law, the Supreme Court of Norway has stated that a law that interferes with privacy or personal integrity must safeguard a legitimate purpose and be proportionate in order to comply with Article 102 of the Constitution of Norway (Supreme Court Reports (Rt.) 2014 p. 1105, paragraph 28; Rt. 2015 p. 93, paragraph 60).

There is a close connection between Article 102 of the Constitution of Norway and Article 8 of the ECHR on the right to privacy. The Supreme Court of Norway has stated that Article 102 of the Constitution of Norway must be interpreted in the light of Article 8 ECHR, but it has stressed that the Supreme Court has an independent responsibility to interpret and develop the Constitution (Rt. 2015 p. 93, paragraph 57).

European Convention on Human Rights (ECHR)

Article 8 of the ECHR establishes the right to privacy and reads as follows:

• 1. Everyone has the right to respect for his private and family life, his home and his correspondence

• 2. There shall be no interference by a public authority with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others.

The most important source for determining the content of state authorities’ obligations and individuals’ rights is the European Court of Human Rights (ECHR). The Court has found that privacy is wide-ranging and has noted that the protection of personal data is of fundamental importance to safeguarding the right to respect for private life. If public authorities store or process someone’s personal data, it will directly affect their privacy, regardless of whether or not the data are used (Marper v United Kingdom No. 30562/04 and 30566/04, paragraph 121). Collecting and processing pupils’ and students’ personal data in learning analytics will constitute an interference with the right to privacy pursuant to Article 8 ECHR. The ECHR contains a framework for how the authorities are to safeguard the fundamental right to privacy in the event of interference and this includes legislative measures.

The central purpose of Article 8 is to prevent authorities from arbitrarily interfering with privacy and this obligation to safeguard the right to privacy is therefore directed at the authorities. Nevertheless, the authorities cannot waive responsibility by delegating duties to private actors and the requirements of Article 8 also apply in such cases (Vukota-Bojić v. Switzerland No. 61838/10, paragraph 47). When it is a private actor that interferes with privacy, the authorities may have a positive duty to safeguard the right to privacy. For example, the authorities may need to take appropriate measures to effectively ensure that the right to privacy is protected (Craxi. 2) against Italy No. 25337/94).

For an interference of privacy to be in line with the Convention, the interference must pass a three-part test. The interference must:

• occur accordance with the law

• further a legitimate aim

• be proportionate to the legitimate aim pursued

The intervention must occur in accordance with the law

The requirement that the interference must occur in accordance with the law means that there must exist a legal basis in national legislation. In addition, the legal basis must be sufficiently foreseeable for the person to whom the interference applies (Satakunnan Markkinapörssi Oy and Satamedia Oy v Finland [Grand Chamber] No. 931/13, paragraphs 150 and 151). It must also contain adequate safeguards against arbitrariness (L.H. v. Latvia No. 52019/07, 2014). What safeguards are necessary must be viewed in the context of the type of interference and the scope thereof (P.G. and J.H. v. United Kingdom No. 44787/98, 2001). The requirement that the interference must be in accordance with the law is closely related to the requirement that the interference is necessary in a democratic society (Marper v United Kingdom No. 30562/04 and 30566/04, paragraph 99).

The intervention must have a legitimate purpose and be proportionate

Legitimate aim means that the interference must be necessary in a democratic society, it must respond to a pressing social need and be proportionate to the need. In its assessment, the European Court of Human Rights (ECtHR) has generally considered whether the interference complies with the fundamental principles of the Article 5 of the Council of Europe’s Convention of 28 January 1981 No. 108 for the Protection of Individuals with regard to Automatic Processing of Personal Data (European Court of Human Rights, 2022, paragraph 105). These fundamental principles concern the minimisation of collected data, whether the data are accurate, adequate and relevant, and whether the data are excessive in relation to the purposes for which they are stored. In addition to this, there are requirements for storage limitations and that the use of the data must be limited to the purpose for which they are collected.

The right to privacy in other international conventions

The right to privacy is also enshrined in other international conventions, including as the Council of Europe’s 1981 Convention for the Protection of Individuals with regard to Automatic Processing of Personal Data – the only legally binding international agreement on data protection. In addition, there is Article 17 of the International Covenant on Civil and Political Rights (ICCPR), which has the force of Norwegian law pursuant to section 2, third paragraph of the Human Rights Act.

5.1.3 Children enjoy special rights protection.

Human rights also apply to children. Children enjoy special rights protection in the Constitution of Norway and in other human rights obligations. In 2004, the UN Convention on the Rights of the Child from 1989 was incorporated into Norwegian law via the Human Rights Act. In the 2014 constitutional revision, article 104 was adopted, which reads as follows:

Children have the right to respect for their human dignity. They have the right to be heard in matters that concern them, and due weight shall be attached to their views in accordance with their age and development.

For actions and decisions that affect children, the best interests of the child shall be a fundamental consideration.

Children have the right to protection of their personal integrity. The authorities of the state shall create conditions that facilitate the child’s development, including ensuring that the child is provided with the necessary economic, social and health security, preferably within their own family.

The constitutional provision on children’s rights aims in particular to highlight those needs that are not covered by the other human rights provisions (Document 16 (2011–2012), section 32.5.1). The constitutionalisation of children’s rights has legal significance, both as an interpretation factor when interpreting legislation and by setting limits for what the legislature can adopt. The provision also has policy and symbolic significance. The policy significance is that decision-makers are to include consideration for children as a goal of policy design. The symbolic significance is that “children are made visible in the Constitution of Norway”.

The first paragraph of Article 104 of the Constitution of Norway stipulates that the right to co-determination in matters concerning the child and the child’s views shall be given due weight in accordance with their age and development.

The fundamental consideration of the best interests of the child is set out in Article 3 (1) of the Convention on the Rights of the Child. The best interests of the child as a fundamental consideration entails that this consideration should not be assessed at the same level as other considerations. Children’s particular situation relates to their dependency, maturity, legal status and, often, voicelessness.

This, in turn, means that children have less of an opportunity than adults to make a strong case for their interests (UN Committee on the Rights of the Child, 2013, section 37). The general comments of the Committee on the Rights of the Child are intended to elaborate on how states parties are to implement the UN Convention on the Rights of the Child and initiate measures that are suitable for fulfilling the Convention obligations and promoting children’s rights. The best interests of the child may conflict with other interests or rights, e.g. of other children, the public, parents, etc. The best interests of the child shall be weighed against other considerations and larger weight must be attached to what serves the child best (UN Committee on the Rights of the Child, 2013, pt. 39). In the proposal for a new Education Act, the Norwegian Ministry proposes to codify the principle of the best interests of the child in a separate and general section (Prop. 57 (Bill) (2022–2023), section 10.5.1). The proposal also entails including pupils over the age of 18 in the scope of the provision.

Under the third paragraph of Article 104 of the Constitution of Norway, children have the right to protection of their personal integrity, which includes protection of privacy. Article 16 (1) of the Convention on the Rights of the Child also protects the child’s right to privacy and family life and reads as follows:

• 1. No child shall be subjected to arbitrary or unlawful interference with his or her privacy, family, home or correspondence, nor to unlawful attacks on his or her honour and reputation.

• 2. The child has the right to the protection of the law against such interference or attacks.

The UN Committee on the Rights of the Child has prepared a separate general comment on children’s rights in relation to the digital environment which contains several statements of relevance to learning analytics. The general comment underlines that the processing of children’s personal data that takes place in schools and the authorities’ collection and processing of data, may pose a threat to children’s privacy (UN Committee on the Rights of the Child, 2021, pt. 67).

5.2 Data protection legislation

Learning analytics will in most cases involve the processing of personal data. Section 1 of the Personal Data Act implements the EU General Data Protection Regulation (GDPR) in Norwegian law. The broad objective of the GDPR is to ensure the protection of natural persons and their rights when personal data about them is processed. The GDPR sets requirements for how the processing of personal data can and should take place.

The GDPR also contains a number of provisions on the establishment of a supervisory authority and its role. In Norway, this role is held by the Norwegian Data Protection Authority. The review of data protection legislation below is based on provisions of particular relevance to learning analytics.

Processing of personal data

The scope of the data protection legislation is broad. Article 2 (1) of the GDPR states that the Regulation applies to “the processing of personal data wholly or partly by automated means”. The meaning of the term processing is not intuitive, but is further defined in Article 4(2):

Any operation or set of operations which is performed on personal data or on sets of personal data, whether or not by automated means, such as collection, recording, organisation, structuring, storage, adaptation or alteration, retrieval, consultation, use, disclosure by transmission, dissemination or otherwise making available, alignment or combination, restriction, erasure or destruction.

The provision includes a non-exhaustive list of various operations involving personal data that can be defined as processing. The concept of treatment may consist of one or several operations that relate to multiple stages of the processing. The provision is technology-neutral in the sense that it is not limited to specific techniques. There is no requirement for the operation to be automatic.

Learning analytics will often consist of several operations. For example, collection, storage, use, alignment and erasure are typical operations for learning analytics.

Personal data

If the information cannot identify a person in accordance with the requirements of the Regulation, we consider the data to be anonymous. If anonymous data are processed, the GDPR does not apply. It is important to note that the legal understanding of which data are anonymous differs from the common and everyday use of the terms anonymous or anonymised data. Personal data is defined in Article 4(1) of the Regulation.

The provision contains four elements that follow directly from the wording: (1) “any information”, (2) “relating to”, (3) “identified or identifiable”, (4) “natural person”. In the case of learning analytics, the elements “any information” and “natural person” will not present interpretive challenges. In the case law of the European Court of Justice (ECJ), it has been clarified that “personal data” is to be understood broadly (C-434/16 (Nowak), 2017, paragraph 33).

Regarding the element “relating to”, the ECJ has found that information provided as an answer by a candidate during an examination constitutes personal data. In its decision, the ECJ also ruled that the examiner’s comments on the candidate’s answer are part of the candidate’s personal data (C-434/16 (Nowak), 2017, paragraph 42).

For learning analytics, a relevant question is when information would be sufficiently decoupled from an individual to fall outside the scope of the definition of personal data in Article 4(1). In many cases, it will be difficult to determine where to draw the line between personal data and anonymous data (Norwegian Data Protection Authority, 2015). As the Expert Group explains in the first interim report, there is an unresolved question of what criteria should be applied as a basis for assessing whether a natural person is identifiable. Two interpretations have been put forth. On the one hand there is the risk-based approach. Here, the decisive factor is whether there is a reasonable probability that the data controller or others can identify a natural person with the aid of advanced technology. On the other hand, anonymisation is regarded as the result of a process that irreversibly prevents identification, rendering it impossible to identify the natural person.

The Expert Group emphasises that since the boundary between personal data and anonymous data is so blurred, it may be difficult to clarify the scope of data protection legislation in learning analytics.

Who is responsible for the processing of personal data in learning analytics?

The GDPR is based on the principle of responsibility set out in Article 5(2). The principle of responsibility means that the controller is responsible for ensuring that the processing is lawful and in accordance with the requirements otherwise stipulated in the Regulation. Data subjects have rights in relation to the data controller and the data controller has an obligation to fulfil the rights of the data subjects.

As defined in Article 4(7), a data controller may be “natural or legal person, public authority, agency or other body”.

What determines whether the data controller is responsible for the processing is whether it, alone or jointly with others, decides the purposes and means of the processing. Municipalities and county authorities are data controllers in relation to primary and secondary education and training. In the case of private schools, the school board is the school owner, and the school board is then responsible for the processing. It is the school owner who is responsible for ensuring that the processing of personal data occurs in accordance with the rules in the Personal Data Act and the GDPR. In higher education and tertiary vocational education, the educational institution is the data controller when personal data are processed in the undertaking.

Data controllers may enter into agreements with other parties or undertakings so that they process personal data on the controller’s behalf. Such an actor is referred to as a processor. In learning analytics, data processors may include suppliers of resources with functionality for learning analytics. The relationship between controller and processor is regulated in a data processor agreement. Such an agreement limits how the data processor may process personal data on behalf of the data controller. The data controller may only use data processors who provide sufficient guarantees that the processing of personal data complies with the requirements of the law in practice and safeguards the rights of data subjects. The Norwegian Privacy Commission notes that it is beneficial that the data processor agreement stipulates that the processor shall use certain built-in solutions that are suitable to safeguard privacy (NOU 2022: 11).

The data processor is not permitted to process the data in any other manner than what is stipulated in the data processor agreement. A key point is that processors who breach the data processor agreement or decide the purpose and means of processing themselves are to be regarded as controllers pursuant to Article 28(10).

If two or more controllers jointly determine the purposes and means of processing, these actors are to be regarded as joint controllers pursuant to Article 26(1). In such cases, the actors shall decide how responsibility for fulfilling the processing obligations are to be distributed, unless this is regulated in the legislation.

Data protection principles

School owners and higher education and tertiary vocational educational institutions are responsible for ensuring compliance with the principles for processing personal data set out in Article 5 of the GDPR. (See Box 5.1 for a description of the data protection principles.) The Norwegian Privacy Commission summarises as follows:

Most provisions of the GDPR contain ordinary legal rules. In addition, six data protection principles have been established, cf. Article 5(1). The principles can be regarded as basic norms for the processing of personal data and provide broad guidelines for what to emphasise in order to safeguard privacy. The principles have been developed over a period of more than 40 years and have long formed the basis for various European data protection legislation. They are always relevant and always mandatory to take into account. (NOU 2022: 11, p. 40)

5.2.1 Requirements for legal basis in the GDPR

In order for the processing of personal data to be lawful, there must be a legal basis for the processing in question. Article 6 of the GDPR contains six possible legal bases for the processing of personal data.

In its first interim report, the Expert Group noted that it is mainly two of the legal bases in Article 6(1) that are relevant for the processing of personal data in learning analytics:

• (c) processing is necessary for compliance with a legal obligation to which the controller is subject […].

• (e) processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller […].

Common to the processing of personal data in connection with a “legal obligation” or that it is a “task carried out in the public interest” is that Article 6(3) requires the establishment of a basis in EU (regulations and directives) or national law. In other words, it is not sufficient to use Article 6(1)(c) or (e) as the sole basis for the processing. A basis must also be found in national legislation.

Recital 41 of GDPR states that when the Regulation refers to a legal basis or legislative measure, “this does not necessarily require a legislative act adopted by a parliament”. In the preparatory works to the Personal Data Act, the Norwegian Ministry of Justice and Public Security finds that “statutory and regulatory provisions may constitute a supplementary legal basis” Prop. 56 LS (2017–2018), section 6.3.2). Thereby, both acts and regulations can be used as legal basis. The Norwegian Ministry also stated that the GDPR’s rules regarding legal basis in national legislation must be interpreted and applied in the light of the requirements in Article 102 of the Constitution of Norway and Article 8 of the ECHR.

Requirements relating to the design of legal basis pursuant to Article 6(1) (c) and (e) in national law

A number of factors apply when assessing whether a provision in national legislation belongs to the category of “legal obligation” or “task carried out in the public interest” pursuant to Article 6(1).

One practical and important consequence is that the processing that occurs on the basis of ‘public interest’ triggers a right for data subjects to object to the processing pursuant to Article 21 (see further details in section 5.2.3). This means that if the basis for processing falls under the category “legal obligation”, it will restrict the rights of the data subject. This implies greater caution in preparing supplementary legal bases in the legislation based on a “legal obligation”.

For a “task carried in the public interest”, it may be sufficient that the supplementary legal basis presumes or orders a public institution to perform a task that requires the institution to process personal data in order to perform the task in question.

When will a provision fall under the category legal obligation?

Where the legal basis is a legal obligation, the aim of the processing of personal data shall be laid down in national legislation pursuant to Article 6(3) of the GDPR. Nevertheless, a legal basis that is a legal obligation need not expressly regulate the processing referred to in the obligation Prop. 56 LS (2017–2018), section 6.3.2).

In parts of the legal literature one unresolved question has been whether Article 6(1)(c) may constitute a legal basis when the public administration is the controller (Kotschy, 2020). In the preparatory works to the Personal Data Act, it is stated that private actors’ processing of personal data may be necessary for compliance with a legal obligation and for the performance of a task carried out in the public interest Prop. 56 LS (2017–2018), section 6.5). The Norwegian Data Protection Authority has found that a legal obligation may constitute a legal basis for a public authority acting as data controller (Norwegian Data Protection Authority, 2022a). The Norwegian Privacy Commission presumes that Article 6(1)(c) applies to a public administrative body that is acting as controller (NOU 2022: 11). The same view is also found elsewhere in the legal literature, where it is noted that it should be more clearly stated in the GDPR or its recitals whether the basis legal obligation should be reserved for private actors (Udsen, 2022).

If a provision on processing personal data only authorises or allows someone to do something, the provision will not be covered by legal obligation (Kotschy, 2020). Where there is legislation entailing that public authorities can take action that requires the processing of personal data, the provision will be covered by Article 6(1)(e) “task carried out in the public interest”. The Norwegian Data Protection Authority states that a legal obligation as a basis for processing indicates that there are no real alternative ways to achieve the aim of the processing set out in the obligation, without processing the data (Norwegian Data Protection Authority, 2022a).

The requirement of necessity and proportionality

For the processing to be lawful, it must be necessary. The requirement of necessity applies both when the processing concerns “compliance with a legal obligation” and the “performance of a task carried out in the public interest”. The GDPR does not define the term necessary.

The requirement of necessity relates to both the data being processed and the actual processing operation(s). Data that are not relevant for the aim of the processing will also not be necessary to process. The requirement of necessity of processing must be viewed in the context of the area being regulated (Kotschy, 2020). In the preparatory works to the Immigration Act, the Norwegian Ministry of Justice and Public Security comments on the criterion of necessity:

The data shall be objectively related to the purpose(s) sought to be achieved through the processing. It is not sufficient that the data may be useful. The data must either on its own, or in conjunction with other data, be significant to the work or to exercise authority. (Prop. 59 (Bill) (2017–2018), section 4.1.3.2)

The Norwegian Ministry of Justice and Public Security provides in an interpretive statement how it interprets the criterion of necessity:

[…] We understand that there is no absolute requirement that the specific processing is strictly necessary, especially that it is not strictly necessary that the processing occurs in a particular manner. (Norwegian Ministry of Justice and Public Security, 2022, section 3.2)

The ECJ has, inter alia, stated that the requirement of necessity may be met in cases where the processing: “contributes to the more effective application” of the legislation in question (C-524/06 (Huber), 2008, paragraph 62).

Whether the criterion of necessity is met will depend on a specific assessment of the relevant legal obligation or task carried out in the public interest and the relevant processing of personal data. We will more closely examine whether this applies to the processing of personal data in learning analytics in section 10.2.

Article 6(3) stipulates that the national supplementary legal basis based on Article 6(1)(c) and (e) shall be proportionate to the legitimate aim. Proportionality concerns the selected means to realise the aim. In this context the means are the type of data (quality), the volume of data (quantity) and the manner in which the data are processed. Recital 39 of the GDPR states that personal data “should be processed only if the purpose of the processing could not reasonably be fulfilled by other means”. The ECJ enunciates the principle of proportionality as follows: “Under the principle of proportionality, limitations may be made only if they are necessary and genuinely meet objectives of general interest recognised by the European Union or the need to protect the rights and freedoms of others” (C-439/19 Latvijas Republikas Saeima [Grand Chamber], 2021, paragraph 105). This entails that the interference with privacy must be justified in relation to the obligation or purpose of the task carried out in the public interest that the processing of personal data is intended to fulfil. In order to meet the requirement for proportionality, limitations or measures related to the processing that reduce the disadvantages will be relevant.

The requirement for a clear and precise legal basis in national legislation

The provisions of the GDPR do not explicitly state that the supplementary legal basis must be clearly and precisely worded. In case of an interference with the right to privacy under Article 104 of the Constitution of Norway or Article 8 of the ECHR, it may be necessary to have a more specific supplementary legal basis in national law than what is indicated in the provisions of the Regulation Prop. 56 LS (2017–2018) section 6.3.2). Recital 41 states that the legal basis of the legislation “should be clear and precise and its application should be foreseeable to persons subject to it, in accordance with the case-law of the Court of Justice of the European Union (the Court of Justice) and the European Court of Human Rights”. The ECJ stresses that interventions must be necessary and proportionate and that the legislation allowing interference “must lay down clear and precise rules governing the scope and application of the measure in question” (C-439/19 Latvijas Republikas Saeima [Grand Chamber], 2021, paragraph 105).

National margin of discretion in the formulation of a legal basis in legislation

Article 6 (3) specifies what the supplementary legal basis established pursuant to Article 6(1)(c) and (e) may contain in terms of specific provisions to adapt the application of the rules in the GDPR. The specific provisions may, inter alia, involve the general conditions on the lawfulness of the processing, the types of data being processed, the data subjects in question, the entities to which the data may be disclosed, the purposes thereof, purpose limitation, storage period and processing operations and procedures.

Recital 10 of the GDPR states that when processing personal data for compliance with a legal obligation or for the performance of a task carried out in the public interest, “Member States should be allowed to maintain or introduce national provisions to further specify the application of the rules of this Regulation”. In this context, the Norwegian Ministry of Justice and Public Security understands the Regulation to mean that, in principle, it is permissible to issue rules that clarify the principles set out in Article 5(1) in particular for the principles of purpose limitation, data minimisation, accuracy, storage limitation and integrity and confidentiality (Prop. 56 (Bill and Resolution) (2017–2018), section 6.5). The Norwegian Ministry states that it is uncertain whether Article 6(2) and (3) allows for tightening the requirements for processing beyond what follows from the general rules in the Regulation. At the same time, the Norwegian Ministry notes that the principles in the Regulation are so discretionary that the distinction between clarifying and tightening rules is fluid.

A key point regarding learning analytics is that Article 6(2) and (3) does not permit the establishment of less stringent requirements than would result from an interpretation of the general rules of the Regulation (Prop. 56 Prop. 56 LS (2017–2018), section 6.5). In this area, the Regulation sets out minimum requirements that national authorities can make more stringent.

Legal basis for processing special categories of personal data

Article 9(1) of the GDPR stipulates that the processing of special categories of personal data is prohibited. The special categories of personal data in Article 9 concern data revealing:

racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation.

In order to process this type of personal data, one of the bases set out in Article 9(2) must be present. Among the possible bases for processing health data is letter (g): “[P]rocessing is necessary for reasons of substantial public interest”. For research, letter (j) may be used as a basis for the processing, i.e., if it is “necessary for […] scientific […] research or for statistical purposes in accordance with Article 89(1)”.

Common to the bases in (g) and (j) is, firstly, that they require a legal basis for their use in national law. Secondly, Article 9(2)(g) stipulates that the processing shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject. When assessing the necessity and proportionality of the processing, the nature of the data will be key in relation to the type of interference and scope thereof. The requirements to ensure suitable and sufficient measures do not provide a clear answer to the Regulation Prop. 56 LS (2017–2018), section 7.1.3). However, the Norwegian Ministry of Justice and Public Security believes that the primary purpose of the guarantees will be to safeguard fundamental data protection principles when personal data are processed. At the same time, the Norwegian Ministry notes that the content of the guarantees will vary considerably and that one possible form of measures may be rules that specify the processing itself.

5.2.2 Requirements for conducting Data Protection Impact Assessments (DPIAs)

In some cases, the controller has a duty to consider the data protection implications of the planned processing of personal data. The duty to consider data protection implications will, pursuant to Article 35(1), be triggered if the planned processing is likely to result in a high risk to the rights and freedoms of the natural persons in question.

Pursuant to Article 35(4), the supervisory authority (the Norwegian Data Protection Authority) shall prepare a list of the kind of processing operations which are subject to the requirement for a data protection impact assessment. The Norwegian Data Protection Authority’s overview includes, among other things, “processing of personal data to evaluate learning, coping and well-being in schools or kindergartens. This includes all levels of education: Primary and lower secondary schools, upper secondary schools and higher education” (Norwegian Data Protection Authority, 2019, section 2). This means that learning analytics that require the processing of personal data are high risk and there is a requirement to consider the data protection implications thereof.

Article 35(7) lists four elements to the content of DPIAs:

• a systematic description of the envisaged processing operations and the purposes of the processing

• an assessment of the necessity and proportionality of the processing operations in relation to the purposes

• an assessment of the risks to the rights and freedoms of data subjects

• the measures envisaged to address the risks, including safeguards, security measures and mechanisms to ensure the protection of personal data and to demonstrate compliance with the Regulation.

Article 35 (9) stipulates that, where appropriate, the controller shall seek the opinions of data subjects or their representatives on the intended processing. In other words, this means that the school owner should obtain the opinions of pupils and parents/guardians or their representatives on the processing. Similarly, institutions in higher education and tertiary vocational education should obtain students’ or representatives’ opinions on the processing.

The Article 29 Working Party (2017) notes that obtaining opinions can take place in different ways depending on the context in question, e.g., with the aid of surveys. If the opinions of the data subjects conflict with the assessments made by the data controller, the controller shall document how the data are followed up. If the controller chooses not to obtain the opinions of the data subjects, this should also be documented.

The Norwegian Data Protection Authority describes such assessments of data protection implications as a continuous process, especially in cases where the processing of personal data changes (Norwegian Data Protection Authority, 2019). Changes to the processing of personal data can often occur when using artificial intelligence.

5.2.3 Rights of data subjects

Chapter 3 of the GDPR contains provisions on the rights of data subjects. These rights enable pupils and students to protect their personal data and rights. Some of the provisions are aimed at the data controller (the disclosure duty in articles 12–14) and the controller is in any case required to facilitate the exercising of data subjects’ rights.

Children’s rights in the data protection legislation

The starting point of the GDPR is that everyone has the same rights in the processing of personal data. This means that children and adults have the same rights. The Regulation does not define children. It was originally proposed to define children as persons under the age of 18 but this definition was not included in the adopted text.

However, recital 38 of the Regulation emphasises that children merit specific protection with regard to their personal data. This is justified on the grounds that children may be “less aware of the risks, consequences and safeguards concerned and their rights in relation to the processing of personal data”.

Regarding information on rights and the communication thereof in relation to children, the controller shall, pursuant to Article 12(1) present the information in a “concise, transparent, intelligible and easily accessible form, using clear and plain language”.

Parents’ and guardians’ exercise of rights on behalf of the child

The data protection legislation does not contain rules that explicitly regulate the right of parents and guardians to assert their child’s rights.

In the European Data Protection Board’s guidelines on the right of access to personal data pursuant to Article 15, the Board emphasises that children have a right to access their personal data and that the right of access belongs to the child. At the same time, the Board notes that depending on the maturity and capacity of the child, the child may need the holder of parental responsibility to act on the child’s behalf (European Data Protection Board (EDPB), 2022).

The Norwegian Data Protection Authority assumes that there is no general age of majority in the field of education and notes that there is no age of majority under the data protection legislation (Norwegian Data Protection Authority, 2023). At the same time, the Norwegian Data Protection Authority assumes that parents and guardians have parental responsibility until the child is 18 years of age and that parents can, in principle, request access to data stored about the child on learning platforms. However, the Norwegian Data Protection Authority also states that this will have to be assessed on a discretionary basis in each specific case, where, among other things, the age of the child, maturity and the type of personal data will form part of the assessment of whether parents and guardians can request access on behalf of the child.

The Expert Group notes that the legal right of parents and guardians to assert rights on behalf of the child is highly discretionary. It can be challenging to make this assessment without clear guidelines, while flexibility makes it possible to adapt assessments to the individual pupil and the circumstances in general.

With regard to parents’ and guardians’ independent right to access information about children, section 47 of the Children Act stipulates that, as a general rule, parents with parental responsibility have the right to information about the child upon request. Any rejections can be appealed to the county governor.

The right to information

Regarding information collected from the data subject, the person concerned shall, pursuant to Article 13, be provided with, among other things, information on the purposes of the intended processing and the legal basis for the processing. In addition, the data subject shall be provided with information about the storage period and the right to exercise the other rights in the Regulation.

In learning analytics, personal data are not always collected directly from pupils and students. When personal data has not been collected from the data subject, the enhanced disclosure duty in Article 14 is triggered. In addition to the requirements pursuant to Article 13, Article 14 entails, among other things, that the data subject shall be provided with information on the categories of personal data concerned and from which source the collected personal data originates.

Right of access

Article 15 stipulates that data subjects have the right to access personal data concerning themselves. In addition, the provision contains an overview of what kind of information data subjects have the right to access. Of particular relevance to pupils and students is the right to know which personal data are being processed, the purposes for which they are processed, the storage period for the personal data and the criteria determining the duration thereof.

Right to rectification and erasure

The right to rectification in Article 16 grants data subjects a right to obtain from the controller the rectification of inaccurate personal data concerning him or her. The right to rectification must be viewed in the context of the purpose of the processing. If, e.g., the purpose is to evaluate or measure the competence of a pupil or student, it is the degree of precision and error in the answers that forms the basis for achieving the purpose of the processing. Such errors will not constitute grounds for rectification under the data protection legislation (C-434/16 (Nowak), 2017, paragraph 53). Nevertheless, situations may arise where an examination answer and the comments made by the examiner thereto may prove to be incorrect within the meaning of the Regulation. One example is if the answer has been exchanged for another, or if parts of the answer have been lost, meaning that the answer is incomplete.

In Article 17, the right to erasure grants data subjects the right to have personal data erased by the controller. This right is often referred to as the “right to be forgotten”. Certain conditions must be met for the right to erasure to apply. Among other things, the data subject has the right to erasure of data that are no longer necessary in relation to the purposes for which they were collected or where the personal data have been unlawfully processed. The right to erasure will also apply if the data subject has objected to the processing pursuant to Article 21 and there are no overriding legitimate grounds for the processing.

This provision does not apply to processing that is necessary for compliance with a legal obligation and the performance of a task carried out in the public interest, cf. Article 17 (3).

Right to data portability

According to Article 20 of the GDPR, the right to data portability, i.e., the opportunity to move data (content) between different services and systems, entails that data subjects, in principle, have “the right to receive the personal data concerning him or her, which he or she has provided to a controller, in a structured, commonly used and machine-readable format and have the right to transmit those data to another controller […]”.

Right to object

The right to object to processing in Article 21 of the GDPR entails that data subjects may, upon request, halt an otherwise lawful processing of personal data. If the conditions for the right to object are met, the data subject may also demand that the processed personal data be erased. The right to object applies if the legal basis for the processing is a “task carried out in the public interest” pursuant to Article 6(1). This means that the right to object does not apply if the legal basis for the processing is a legal obligation pursuant to Article 6(1)(c).

There is one key exception to the right to object in Article 21(1) of the Regulation. If the data controller can demonstrate “compelling legitimate grounds for the processing which override the interests […] of the data subject”, the processing of the personal data may continue. According to the wording, this assessment will be based on “grounds relating to his or her particular situation”.

It has not been clarified how the specific content of the right to object shall be determined in a pedagogical context. The Norwegian Privacy Board has considered several cases concerning the right to object and erasure and has concluded that there were “compelling legitimate grounds for the processing”.16 In the opinion of the Expert Group, none of these cases have direct relevance for learning analytics, as they have mainly concerned erasure of internet search engine results and archiving obligations weighed against the interests of data subjects.

There has been uncertainty as to how the right to object should be managed in practical terms. For instance, what if this right triggers numerous requests? And then there is the issue of how the controller should manage requests to object to the processing (Prop. 56 (Bill and Resolution) (2017–2018), section 10.5.4).

The right not to be subject to automated decision-making

Decisions that are fully automated are regulated by Article 22 of the GDPR. Article 22(1) stipulates that the data subject has the right not to be subject to a “decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her”. Three conditions must be met for the data subject to have the right not to be subject to an automated decision: (1) “decision” (2) “based solely on automated processing, including profiling” (3) “which produces legal effects concerning him or her or similarly significantly affects him or her”.

Firstly, it must involve a decision, i.e., something to indicate that a decision has been made or assessments have been performed that could form the basis for further action.

Second, it is a condition that the decision is “based solely on automated processing, including profiling.” Pursuant to Article 4(4) “profiling means any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person […]”.

The fact that the decision is “based solely on automated processing” presupposes that a person is not able to actually influence the decision. Situations where a person is involved in the decision-making process but does not actively take a position on the automated assessment before the person concerned formally makes the decision will fall under Article 22 (Bygrave, 2020). Article 22 does not apply in situations where decision support is actually considered by the person making the decision.

Third, the decision must have “legal effects concerning him or her or similarly significantly affects him or her [the data subject].” This will typically include administrative decisions, which have legal effect in the sense that the decision determines rights and duties. What might similarly affect the person concerned may be difficult to determine specifically. However, such decisions must have consequences that could seriously affect the well-being of the person concerned. The Article 29 Working Party (2018) provides examples of what may fall under the category and includes the following example from the education sector: “decisions that affect someone’s access to education, for example university admissions” (p. 22). In addition, the Article 29 Working Party notes that the threshold for the decision to significantly affect the person concerned may have been reached in the case of decisions with a clear impact on circumstances, behaviour or choices, which may have significant long-term or permanent effects, and which could lead to discrimination or exclusion of individuals.

Under certain conditions, exceptions may be made to the right not to be subject to an automated decision. Pursuant to Article 22(2)(b), national authorities may lay down legislation permitting automated decision-making, provided that suitable measures have been established to safeguard the data subject’s rights and freedoms interests.

Pursuant to Article 22(4) automated decision-making shall not be based on special categories of personal data referred to in Article 9(1). Nevertheless, there may be exception to this principle if the processing is necessary pursuant to Article 9(2)(g) and suitable measures to safeguard the data subject’s rights and freedoms are in place.

5.3 Legislation in the education sector

There is a comprehensive legislative and regulatory framework that regulates the education sector. In this section, we refer to general provisions that stipulate the objectives of education. However, we discuss the general provisions on the processing of personal data and possible supplementary legal bases for the different levels of education in sections 10.3–10.5.

5.3.1 Legislation in primary and secondary education and training

It is mainly the Education Act and the Regulations to the Education Act that constitute the relevant legislation for learning analytics in primary and secondary education and training. The Independent Schools Act applies to primary and secondary education with the right to government subsidies. To avoid duplicate work, we will not discuss the provisions of the Independent Schools Act and its accompanying Regulations. We assume that assessments and proposals related to the Education Act are also relevant to the corresponding provisions in the Independent Schools Act.

During the final phase of our work, the Norwegian Ministry of Education and Research submitted a proposal for a new Education Act Prop. 57 L (2022–2023). However, the following description is based on the provisions of the current Act.

Objectives of education and training

Section 1-1 of the Education Act stipulates the objectives of education and training in seven paragraphs. Among other things, the provision expresses the values that are to form the basis for education and training and what education and training shall contribute towards and provide insight into. The statutory objective does not directly address pedagogical methods that are to form the basis for the education and training, but the fifth paragraph stipulates that “pupils and apprentices must develop knowledge, skills and attitudes so that they can master their lives and can take part in working life and society. They must have the opportunity to be creative, committed and inquisitive.” The sixth paragraph states that the pupils “shall have joint responsibility and the right to participate”.

Provisions of the Education Act with specific relevance to primary and lower secondary school

Pursuant to section 2-1, first paragraph of the Education Act, children and young people are obliged to attend primary and lower secondary education, and they have the right to “public primary and lower secondary education in accordance with this Act and regulations pursuant to the Act”. Section 13-1, first paragraph of the Education Act stipulates that municipalities must comply with the right of all residents in the municipality to primary and lower secondary education. With regard to the content and assessment of education and training, the third paragraph of section 2-3 of the Education Act stipulates that the Norwegian Ministry of Education and Research may, among other things, issue regulations on “content of the instruction in the subjects and the conduct of the instruction”.

Provisions of the Education Act with specific relevance to upper secondary education

The right to upper secondary education is laid down in section 3-1 of the Education Act, which stipulates that “[p]upils, apprentices, candidates for certificate of practice and training candidates have the right to education and training in accordance with this Act and regulations issued pursuant to the Act”. Pursuant to section 13-3, first paragraph of the Education Act, the county authority must comply with the right of all residents of the county to upper secondary education and training. Pursuant to section 3-4, first paragraph of the Education Act, the Norwegian Ministry of Education and Research may issue regulations, including on the scope and implementation of the education and training.

5.3.2 Legislation on higher education and tertiary vocational education

There are several acts and regulations governing higher education and tertiary vocational education. Universities and university colleges are regulated by the Universities and University Colleges Act. The Regulations on the quality of programmes of study17 relate to quality assurance and quality assurance work in both higher education and tertiary vocational education. The Academic Supervision Regulations18 only apply to higher education. Vocational colleges are regulated by the Vocational Education Act, the Vocational Education Regulations19 and the Vocational Education Academic Supervision Regulations20.

Objective

The statutory objective in Section 1-1 of the Universities and University Colleges Act stipulates that one of the objectives of the institutions is to “provide higher education at a high international level”. Section 1-3 of the Universities and University Colleges Act prescribes the tasks of the institutions. It states that the institutions must, among other things, provide “higher education based on the foremost within research, academic and artistic development work, and experience-based knowledge”. Pursuant to Section 1-5, first paragraph of Universities and University Colleges Act, institutions are responsible for ensuring that instruction is “conducted in accordance with recognised scientific, artistic, pedagogical and ethical principles”.

Section 1 of the Vocational Education Act stipulates that the purpose of the Act “is to ensure the provision of high-quality vocational education and satisfactory conditions for students of vocational education”. Regarding requirements for vocational education, Section 4, third paragraph of the Vocational Education Act stipulates that the education “shall be based on knowledge and experience from one or more occupational fields and be in accordance with relevant pedagogical, ethical, artistic and scientific principles”.

5.4 Regulation of artificial intelligence (AI)

It is likely that the regulation of AI will be expanded in the near future with two new European regulations. The new regulations on AI go further than the general regulation found in the GDPR and the specific rules governing the use of personal data for, among other things, profiling in Article 22. The two new regulatory proposals at the European level aim to regulate, among other things, the development, marketing and use of AI. The first is the EU’s proposed AI Act (European Commission, 2021) and the second is the Council of Europe’s proposed AI Convention (Council of Europe, 2023).

These proposed regulations could affect how AI is used in learning analytics. These developments may lead to the codification of certain ethical principles, which may in turn lead to a more transnational development of learning analytics technologies within Europe. New actors may then emerge and other mechanisms may be established, which will be relevant for the education sector.

5.4.1 EU regulation of artificial intelligence

In April 2021, the European Commission proposed an AI Act: Artificial Intelligence Act. The broad objective of the proposal is twofold. On the one hand, the objective is to make it easier to utilise the potential of AI, e.g., by eliminating trade-related barriers. On the other hand, it is about protecting societies and individuals from harm, especially in terms of individual safety and human rights.

The Act will apply to systems with artificial intelligence, which are defined very broadly in Article 3(1):

An AI system is a machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.

A risk-based approach has been used to both define the level of regulation of each AI system and to the application of the Act (Mahler, 2022). With regard to the level of regulation, the proposal distinguishes between four main categories of risks. There are AI systems that (1) contain unacceptable risks and which are therefore prohibited; (2) that are high-risk systems that must comply with specific requirements; (3) that represent a limited risk and entail fewer requirements, and (4) that involve a minimal risk and where no requirements apply (Veale and Borgesius, 2021).

AI in education is categorised as high-risk in specific areas in Annex III (3) of the Act:

• (a) AI systems intended to be used to determine access or admission or to assign natural persons to pedagogical and vocational training institutions at all levels

• (b) AI systems intended to be used to evaluate learning outcomes, including when those outcomes are used to steer the learning process of natural persons in educational and vocational training institutions at all levels.

Chapter 2 of the Act contains a number of requirements for high-risk systems, including a risk management system, good training models and good data governance, technical documentation, record-keeping of data processes, transparency and provision of information to users, human oversight, accuracy, robustness and cyber security (Articles 8-15 of Title III). Articles 16-51 describe in detail the obligations of suppliers and users of high-risk AI systems.

5.4.2 Council of Europe Convention on Artificial Intelligence

In autumn 2019, the Council of Europe appointed a committee to assess the opportunities and threats that artificial intelligence entails for human rights (Norwegian Ministry of Local Government and Modernisation, 2020). Following a preliminary report, the committee was formalised in 2022 as the Committee on Artificial Intelligence (CAI) (Council of Europe, 2023). In January 2023, the Committee submitted a draft convention: “Revised Zero Draft [Framework] Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law”. As in the EU proposal, attention is geared toward systems and the definition of AI is broad. However, the draft convention places greater emphasis on functionalities:

artificial intelligence system means any algorithmic system or a combination of such systems that, as defined herein and in the domestic law of each Party, uses computational methods derived from statistics or other mathematical techniques to carry out functions that are commonly associated with, or would otherwise require, human intelligence and that either assists or replaces the judgment of human decision-makers in carrying out those functions. Such functions include, but are not limited to, prediction, planning, classification, pattern recognition, organisation, perception, speech/sound/image recognition, text/sound/image generation, language translation, communication, learning, representation, and problem-solving […]

The scope of the Convention may be broader than the EU proposal, as the Convention addresses the entire life cycle of AI systems, regardless of whether public or private actors are involved in their design, development or use (Article 4). Articles 5–11 contain a number of state obligations. This includes a duty to ensure that the use of AI in administrative decisions respects human rights, to minimise harm from using AI systems, and to assess potential risks. Education is explicitly mentioned in Article 8(a):

Each Party shall, within its respective jurisdiction, ensure that: […] the application of an artificial intelligence system in provision of goods, facilities and services in essential areas, such as but not restricted to, health, family care, housing, energy consumption, transport, food supply, education, employment, finance, environmental protection, digital information, media and communication is fully compatible with its domestic law and any applicable international law insofar as these require relevant public and private actors to respect human rights and fundamental freedoms.

The remainder of the Convention contains principles relating to the design, development and different types of use of AI systems (Articles 12–18), monitoring mechanisms (Articles 19–23), and risk assessment and training (Articles 24–26). Key principles include equal treatment and non-discrimination, respect for privacy and data protection, compliance with the law, accountability, transparency and security procedures. In addition, the principles involve preventing harmful innovation processes, facilitating public debate and contributing to increased digital literacy in the population. States are also to ensure that suppliers and users take into account and assess AI-related risks.