New Machine Learning Algorithms for the Calculation of Water Values

In all power markets, the best - or optimal - use of the generation and storage resources available is essential in order to achieve the best outcomes for both actors in the market and as society as a whole. In hydro-dominated markets, such as the Nordic market, this optimisation is particularly challenging, as water can be stored from one period to the next. Water used today cannot be used tomorrow; conversely forgoing generation (use of water) today enables its use tomorrow. To figure out what is the optimal use of generation resources today, we thus need to take into account the likely best use of generation resources in the coming days, months and years. Over many decades the industry has developed the concept of "water values" as the basis for conducting this optimisation. A reservoir's water value is defined as the likely benefit (e.g. market income, lower costs) of keeping the water in storage and saving it to generate electricity at some point in the future. Put simply, if the water value is higher than the benefit of using the water now it is optimal to keep it in storage, and if it is not the water should be used now to generate electricity. There is a clear need for accurate, timely calculation of water values for a well-functioning electricity system. Today's methods fall short of this however. They both take a long time to calculate and are overly-simplified. Our project focused on developing novel machine learning (ML) models to calculate water values, building on recent generative AI developments (specifically, reinforcement and adversarial learning). Our project consisted of three parts: the first, an investigative component to research the RL and related literature and algorithms, to determine our broad plan of attack for our project; the second to iteratively build and test candidate algorithms; and the third to test and assess the candidates against real-world water value calculations. Following part one, we developed and tested a 4 alternative algorithm "families", before selecting the best-performing reinforcement learning approach to test against real-world operation of two hydro power stations over the period 2014-2020. The algorithms outperformed the actual operation of the two hydro plants over the period, measured on income and operational benchmarks. To followup this work it is planned to apply for additional funding to further develop the selected algorithm into a prototype tool for optimising hydro production and storage-release decisions, and to test this in a realistic environment.

Outcomes: The project has developed several candidate algorithms for water value calculation and operation (that is, production scheduling), using machine learning/reinforcement learning. These algorithms have been tested against actual operation of hydro plants over a 7 year test period, and have outperformed actual operational results on a set of income and operational benchmarks. The developed methods are planned to be extended in a further pilot innovation project, and is expected to ultimately lead to commercial operational software. For Optimeering, this end result (commercial software) will represent a new commercial product, that will be sold to hydro power producers and other storage providers in the Nordics and other power markets. For Lyse and other power producers, this is a promising alternative to existing methods for hydro storage water value calculation and operation, and a successful commercial product can lead to improved hydro asset operations. For Statnett, TSOs and regulators the project will result in improved operational decisions, lower costs, and improved grid investment decisions. The project has in addition contributed to competence building within the all participants’ R&D environments on machine learning and will lead to the further application of these techniques to other application areas in their business activities. Impact: The project will provide socio-economic benefits to consumers and producers by directly enabling more optimal valuation and operation of hydropower and other storage technologies (e.g. batteries), and help make the transition to power systems with more renewable production possible, both nationally and internationally.

In power markets featuring hydro generation such as the Nordic market, the optimal use of the available hydro power resource is essential to a well-functioning power system, in order to achieve desired environmental goals and to maximise the socio-economic benefit of the power system. Over many decades the industry has developed the concept of “water values” as the basis for conducting this optimisation. A reservoir’s water value is defined as the marginal expected benefit (e.g. market income, lower costs) of keeping the water in storage and saving it to generate electricity or provide power system services at some point in the future. Put simply, if the water value is higher than the benefit of using the water now it is optimal to keep it in storage, and if it is not the water should be used now. All actors - producers, grid operators, regulators, use and calculate water values to make bidding, operational and investment decisions. The methods used today to calculate these suffer from known computational limitations that reduce their accuracy and increase time-to-solve. The increase in renewable generation and market complexity (e.g. via new reserve markets) additionally challenge existing methods. There is an unmet need in the power industry for better methods to calculate water values that better capture realistic market, geographical and time detail, a full range of uncertainties and their impact, and can calculate these quickly. Our proposed project will research and develop innovative machine learning (ML) models to calculate water values for use in both operative applications and market simulation models. Our approach will build on recent developments in reinforcement and adversarial learning, to develop a system that “learns” how to optimally value (operate) storage hydro, and that uses the computational advantages of machine learning to do so at an increased level of detail and accuracy.