Structured Assessment System for Improved Student Learning

Providing students with timely feedback - assessment - is important for student learning. Assessments may be formative, aimed at providing feedback to help students improve, or summative, focused, instead, on evaluating student learning. Assessment methods must be based on research to ensure that they contribute to learning in a constructive manner. This is particularly important when the educational contents are changing to include computing, the use of computers to solve problems in specific subject areas, an area where the research basis for effective assessments is currently sparse. The goal of this project is to develop, validate and implement an assessment system to be used for teacher-, peer- and self-assessment for general courses as well as for courses that integrate computing. The system will structure the way assessments are performed to ensure adapted feedback to students. We will accumulate data on student learning that will be used to improve feedback and initiate interventions. The system will be developed by CCSE at the University of Oslo (UiO), piloted in formative assessments and exams in selected science courses at UiO, and then broadly disseminated at UiO, nationally, and internationally. Through this project we have developed methods to assess students’ computational competence. The methods have been developed through interviews with teachers and students and have resulted in a rubric which has been tested across teachers, courses and institutions. The original plan was to use such a rubric as a basis for automatic feedback to the students, but the developed of artificial intelligence and large language models have made the originally proposes system largely outdated. Through the project we have therefore instead developed methods for using natural language processing. This was first developed in well-controlled situations to address the development of concepts in science education literature. These methods are currently being extended using text embeddings from large language models to map students’ textual answers and provide feedback in a project at INTED, the center for interdisciplinary education. We are seeking further financing for this project through national and international sources.

This project has allowed the Center for Computing in Science Education to develop new methods for assessing students’ computational competence. The project has developed rubrics that are currently used in assessments at several institutions. And the project has provided a basis for developing a framework for assessing computational competence. In addition, the project has provided the foundation in the application of natural language processing methods to categorize and characterize texts. Initially, this has been used to understand developmental trends in education research. However, the methods developed form the basis for applying these methods to assess students’ texts and provide adapted feedback to student work. The project has also initiated to a long-term collaboration between the University of Oslo and Michigan State University on assessment methods and rubrics. This has both improved education and assessments and provide a basis for future collaboratory projects.

Effective and timely assessments for students in higher education is important to improve student learning, but providing assessments and feedback to students requires significant resources. This is the basis for the growth in methods and technology for peer-assessment, self-assessment, computer-aided assessment and various methods to make teacher-assessment more efficient. In addition, significant curricular changes are occurring as computational methods are integrated in educations across fields. At the University of Oslo (UiO) the newly established Center for Computing in Science Education (CCSE), a Center for Excellence in Education, is at the forefront of this change. However, new curriculums and new learning outcomes require new, effective formative and summative assessments. The goal of this project is to develop, validate and implement an assessment system to be used for teacher-, peer- and self-assessment for general courses as well as for courses with integrated computing. The system will provide students with timely and adapted feedback in a cost-efficient manner and ensure more valid and reliable grading of exams. The system will be developed by CCSE and implemented in the science education at UiO. A successful assessment system depends on research to develop effective methods for assessment of both regular and computational learning outcomes. We need research to develop and validate effective assessment structures, such as rubrics or checklists, that are adapted both to traditional curriculums as well as concepts and understanding linked to a computational approach to science. Also, the structures must be suitable for teacher-, peer-, and self-assessment roles with a focus on how feedback from assessments can be tuned to produce as good student learning as possible. Research will be conducted by CCSE and the educational research group at the Michigan State University.