Generic Hydropower Optimization Using Deep Reinforcement Learning (GHO-DRL)

A key challenge for all hydropower producers is to figure out how to use the water in the reservoirs in the most efficient way. To solve such a problem many factors such as inflow, amount of snow in the mountainside, expected power prices, and so on, must be considered. This research project has applied artificial intelligence to the hydropower scheduling problem. And we have developed new techniques and software that can be used to optimize water usage in hydropower river systems. The project was motivated by the fact that the Nordic electricity market for the last years has experienced a huge fall in energy prices. The reasons for this fall are many, but the undisputable consequence for the Norwegian hydropower sector is that income is going down and there is an urgent need to lower costs and run the operations more efficiently. Preliminary results indicate that deep learning can be used to identify optimal reservoir policies for a given situation. If the project as a whole succeeds, the value creation will be more efficient utilization of the water resources available at Agder Energi, and potentially for the rest of the hydropower sector.

The project has shown that it is possible to use deep learning to understand how water reservoirs should be utilized optimally. The main results of the project demonstrated how different search algorithms can be combined with synthetic data and used to train neural networks. The trained neural networks can be used to suggest production plans or water values based on price and inflow information. The algorithms have been tested in daily operations for Kvinesdal power station. Agder Energi is now using the methods in its daily work. The project shows an area where artificial intelligence has great potential and in the future the methods developed could form the basis for new tools for the rest of the hydropower industry. The promising results also warrants further research on deep learning-based hydropower optimization.

The project is motivated by the fact that the Nordic electricity market for the last two years has experienced a huge fall in energy prices. The reasons for this fall are many, but the undisputable consequence for the Norwegian hydropower sector is that income is going down and there is an urgent need to lower costs and run the operations more efficiently. To this end, there is a need for more practical applicable simulation software. However, traditional solvers based on linear and dynamic programming techniques have significant shortcomings in operational use. The underlying idea in this research project is to apply and adapt recent breakthroughs in so-called Deep Reinforcement Learning (DRL) to the hydropower scheduling problem. To our knowledge, this has never been done before. We will develop DRL based models, algorithms and an accompanying hydraulic-economic model that can utilize inexpensive and massive parallel computing platforms offered by Graphical Processing Units (GPUs). The new software will be tested in an operational setting at Agder Energi and compared to traditional optimization techniques based on linear- and dynamic programming methods (LP, DP). Given the success in this project the value creation will be more efficient utilization of the water resources available at Agder Energi. Today this energy resource is about 4 percent of the Norwegian hydropower production and represents a huge part of Agder Energi's income. It is clear that even small incremental improvements in how water resources are used (1-2 %), will create huge increase in income for the Norwegian hydropower sector. Furthermore, societal costs related to flooding may be reduced with improved techniques for hydropower optimization.