Deep Learning Method for Heliostat Instance Segmentation




Deep Learning, Instance Segmentation, Heliostat, Tracking


Heliostat instance segmentation (HST-IS) is a crucial component of the heliostat tracking system at Heliogen’s Lancaster test facility. The system estimates the mirror normal of each heliostat by performing a nonlinear optimization-based fitting strategy using approximations of the non-shaded, non-blocked sunlit pixels on each heliostat, and the tracking system uses these estimates to improve performance.

HST-IS is fundamentally challenging due to variability in lighting conditions and heliostat size relative to the capturing camera. Deep learning-based convolutional neural networks (CNN) have emerged in recent years by demonstrating noteworthy precision in tasks such as object recognition, detection, and segmentation. CNN-based methods offer a robust augmentation to HST-IS methods as they capture a context-less hierarchy of image features.

In this study, we developed deep learning models to automatically segment heliostat instances from elevated images taken from the field. We study various image parameters and architectural customizations to optimize for scalability, robustness, and accuracy in our predictions. We perform robust evaluations of our best model to quantify gaps between model development and real-world deployment and provide evidence for utility in the field.


Download data is not yet available.


Sonn, Alexander, et al. "Estimating orientations of tracking heliostats using circumsolar radiance." AIP Conference Proceedings. Vol. 2445. No. 1. AIP Publishing LLC, 2022. doi:

Kribus, Abraham, et al. "Closed loop control of heliostats." Energy 29.5-6 (2004): 905-913. doi:

Sattler, Johannes Christoph, et al. "Review of heliostat calibration and tracking control methods." Solar Energy 207 (2020): 110-132. doi:

Bouaddi, S., Ihlal, A., Fernández-García, A., 2017. Comparative analysis of soiling of CSP mirror materials in arid zones. Renew. Energy 101, 437–449. doi:

Röger, Marc, Christoph Prahl, and Steffen Ulmer. "Heliostat shape and orientation by edge detection." Journal of solar energy engineering 132.2 (2010). doi:

Coventry, Joe, et al. "A robotic vision system for inspection of soiling at CSP plants." AIP Conference Proceedings. Vol. 2303. No. 1. AIP Publishing LLC, 2020.

El Ydrissi, Massaab, et al. "Dust InSMS: Intelligent Soiling Measurement System for dust detection on solar mirrors using computer vision methods." Expert Systems with Applications (2022): 118646. doi:

Deng, Li, and Dong Yu. "Deep learning: methods and applications." Foundations and trends® in signal processing 7.3–4 (2014): 197-387. doi:

Dhillon, Anamika, and Gyanendra K. Verma. "Convolutional neural network: a review of models, methodologies and applications to object detection." Progress in Artificial Intelligence 9.2 (2020): 85-112. doi:

Rawat, Waseem, and Zenghui Wang. "Deep convolutional neural networks for image classification: A comprehensive review." Neural computation 29.9 (2017): 2352-2449. doi:

Sultana, Farhana, Abu Sufian, and Paramartha Dutta. "Evolution of image segmentation using deep convolutional neural network: a survey." Knowledge-Based Systems 201 (2020): 106062. doi:

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Communications of the ACM 60.6 (2017): 84-90. doi:

Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

Redmon, Joseph, and Ali Farhadi. "Yolov3: An incremental improvement." arXiv preprint arXiv:1804.02767 (2018).

Yi, Zhang, Shen Yongliang, and Zhang Jun. "An improved tiny-yolov3 pedestrian detection algorithm." Optik 183 (2019): 17-23. doi:

He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017.

Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." European conference on computer vision. Springer, Cham, 2014.

Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances in neural information processing systems 28 (2015).

He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.




How to Cite

Liu, B., Sonn, A., Roy, A., & Brewington, B. (2024). Deep Learning Method for Heliostat Instance Segmentation. SolarPACES Conference Proceedings, 1.

Conference Proceedings Volume


Measurement Systems, Devices, and Procedures