Machine Learning Models for the Surveillance of Physical Power Markets

For power markets to function well and deliver promised socio-economic benefits, they must be (among other things) free from manipulation by those active in the market. A market subject to manipulation, especially if this is widespread, suffers from reduced market efficiency, lower public confidence, and ultimately lower socio-economic benefits. The on-going monitoring of power markets by regulatory agencies and the market operators for potential manipulation is a critical part of ensuring that these markets are well functioning. Due to the large volumes of data involved and increasing speed and complexity of power markets, such monitoring requires advanced tool to automatically process and assess market developments for potential manipulation events, and which can then be passed on to human analysts for further assessment. In project HAWK our focus was on developing and testing innovative machine learning (ML) algorithms and models for the daily monitoring of physical power markets. Project HAWK was undertaken by Optimeering together with researchers from NTNU and market monitoring analysts from NVE (the Norwegian power market regulator). The project consisted of several research packages. The first consisted of in-depth research interviews with regulators and market participants, as well as assessments of market surveillance practices and the literature, in order to map current surveillance practice and challenges, and to identify promising research areas. Resulting from this, the subsequent work packages focused on developing new ML methods and algorithms for identifying anomalous market outcomes and actor behaviour in the day-ahead market where the majority of volume is traded in the Nordic power market, and RK (regulating energy) market. The packages included developing methods for monitoring day-ahead market (DAM) price levels and dynamics, DAM price-volume relationships, market bid curves at a market and regional level, and portfolio bid curves and bidding behaviour. In the final work package, the developed algorithms were assessed on historical data by NVE analysts, and compared to their assessments made "live". Overall the new methods were shown to be highly useful, identifying potential new periods of manipulation, as well as highlighting many periods already assessed by the analysts. Post project completion the algorithms will be installed at NVE for live use in on-going market monitoring activities.

The project has provided significant competency building for Optimeering´s employees. The project results will form the basis of new commercial market surveillance software products and new business lines for the company. The project has and will lead to new specific surveillance tools and methods used by NVE to monitor the Norwegian power market, delivering improved and more comprehensive surveillance activities and results. The R&D work also is a substantial component of a PhD project at NTNU. The project improves overall efficiency and effectiveness of the power market via more effective market surveillance operations, lowering surveillance costs and overall costs of power market regulation, increasing market transparency and trust, and help make the transition to a renewable-dominated power system possible. The project will contribute to competence building within the regulator on machine learning and introduce these techniques to groups within market regulators and exchanges.

The goal of our project is the development and implementation in software of machine learning models for the operative surveillance of physical power markets. The developed models and software will be designed to run automatically and be used by power market regulators such as NVE, market operators, brokers and TSOs in their daily surveillance activities in physical power markets, and by market actors trading in physical markets to monitor and regulate their own trading activities. Power markets often exhibit low elasticity of demand. In combination with large and flexible power storages (e.g. hydro power and, increasingly, batteries in combination with renewables) this results in market places that are particularly vulnerable to abuse. Surveillance of markets with high renewable and storage penetration is especially complex and difficult. Auction markets where all participants face the same price may be particularly exposed, especially in periods with congestion. Currently, most models and tools used for market surveillance in the power market based on sets of rules used to define market abuse scenarios. Rule-based approaches have several weaknesses, such as a tendency to produce large volumes of alerts, an inability to assess and priories the alerts produced, and difficulty in catching new behaviours. Machine learning is well suited for such monitoring, and can execute much more effectively than rules alone. The monitoring will be more automated, produce more relevant alerts (and fewer false positives) and enable users to detect market behavior that would otherwise be obscured. Such tools will also be more robust, since they will be less dependent on specific persons. This will in turn enable more effective and efficient mitigating actions to be planned and and taken, increasing the transparency and efficiency of the power system, especially given increased use of storage and renewable generation.