Forpowermarkets to function well and deliver promised socio-economic benefits, they must be (among other things) free from manipulation by those active in the market. A market subject to manipulation, especially if this is widespread, suffers from reduced market efficiency, lower public confidence, and ultimately lower socio-economic benefits.
The on-going monitoring ofpowermarkets by regulatory agencies and the market operators for potential manipulation is a critical part of ensuring that these markets are well functioning. Due to the large volumes of data involved and increasing speed and complexity ofpowermarkets, such monitoring requires advanced tool to automatically process and assess market developments for potential manipulation events, and which can then be passed on to human analysts for further assessment. In project HAWK our focus was on developing and testing innovative machinelearning (ML) algorithms and modelsforthedaily monitoring ofphysicalpowermarkets.
Project HAWK was undertaken by Optimeering together with researchers from NTNU and market monitoring analysts from NVE (the Norwegian power market regulator). The project consisted of several research packages. The first consisted of in-depth research interviews with regulators and market participants, as well as assessments of market surveillance practices and the literature, in order to map current surveillance practice and challenges, and to identify promising research areas. Resulting from this, the subsequent work packages focused on developing new ML methods and algorithms for identifying anomalous market outcomes and actor behaviour in the day-ahead market where the majority of volume is traded in the Nordic power market, and RK (regulating energy) market. The packages included developing methods for monitoring day-ahead market (DAM) price levels and dynamics, DAM price-volume relationships, market bid curves at a market and regional level, and portfolio bid curves and bidding behaviour.
In the final work package, the developed algorithms were assessed on historical data by NVE analysts, and compared to their assessments made "live". Overall the new methods were shown to be highly useful, identifying potential new periods of manipulation, as well as highlighting many periods already assessed by the analysts. Post project completion the algorithms will be installed at NVE for live use in on-going market monitoring activities.
The project has provided significant competency building for Optimeering´s employees. The project results will form the basis of new commercial market surveillance software products and new business lines forthe company. The project has and will lead to new specific surveillance tools and methods used by NVE to monitor the Norwegian power market, delivering improved and more comprehensive surveillance activities and results. The R&D work also is a substantial component of a PhD project at NTNU.
The project improves overall efficiency and effectiveness ofthepower market via more effective market surveillance operations, lowering surveillance costs and overall costs ofpower market regulation, increasing market transparency and trust, and help make the transition to a renewable-dominated power system possible.
The project will contribute to competence building within the regulator on machinelearning and introduce these techniques to groups within market regulators and exchanges.
The goal of our project is the development and implementation in software ofmachinelearningmodelsforthe operative surveillance ofphysicalpowermarkets. The developed models and software will be designed to run automatically and be used by power market regulators such as NVE, market operators, brokers and TSOs in their daily surveillance activities in physicalpowermarkets, and by market actors trading in physicalmarkets to monitor and regulate their own trading activities.
Powermarketsoften exhibit low elasticity of demand. In combination with large and flexible power storages (e.g. hydro power and, increasingly, batteries in combination with renewables) this results in market places that are particularly vulnerable to abuse. Surveillance ofmarkets with high renewable and storage penetration is especially complex and difficult. Auction markets where all participants face the same price may be particularly exposed, especially in periods with congestion.
Currently, most models and tools used for market surveillance in thepower market based on sets of rules used to define market abuse scenarios. Rule-based approaches have several weaknesses, such as a tendency to produce large volumes of alerts, an inability to assess and priories the alerts produced, and difficulty in catching new behaviours.
Machinelearning is well suited for such monitoring, and can execute much more effectively than rules alone. The monitoring will be more automated, produce more relevant alerts (and fewer false positives) and enable users to detect market behavior that would otherwise be obscured. Such tools will also be more robust, since they will be less dependent on specific persons. This will in turn enable more effective and efficient mitigating actions to be planned and and taken, increasing the transparency and efficiency ofthepower system, especially given increased use of storage and renewable generation.