Operational Optimization of PV-Assisted Heat Pump Systems Using Reinforcement Learning

Abstract

Heat pumps are increasingly integrated with PV systems to supply domestic hot water and space heating. End user purchase prices for electricity are typically significantly higher than feed-in tariffs for PV energy. From a financial perspective, it therefore makes sense to increase the self-consumption of local PV electricity and reduce the amount of energy drawn from the grid. One approach is to overheat the thermal energy storage when the heat pump can be operated with PV energy. However, with increasing temperature, heat losses increase and the heat pump’s COP decreases, making excessive overheating uneconomical. The optimal degree varies with time and depends on various factors, such as heat demand, storage capacity and the purchase-to-feed-in price ratio. Using detailed system simulations, we inves-tigate the application of self-adaptive reinforcement learning agents that can dynamically determine the optimal degree of overheating. The agents continuously evaluate environmental and operational inputs to make informed control decisions, enhancing cost-effectiveness. The results of our empirical evaluation demonstrate that the agents reliably learn to avoid unnec-essary hot water overheating and adapt their space heating overheating strategy to the char-acteristics of each system, achieving net electricity costs close to the optimal solution. Across multiple configurations, agents operating in reduced environments—i.e., with fewer state vari-ables—converge more quickly and develop more stable control policies. In contrast, restrict-ing the action space alone does not consistently enhance performance. Although the overall cost reduction potential is modest, amounting to EUR 35–90 per year, the proposed approach has the important advantage of guaranteeing that user comfort is never violated.