Occupant Counting and Position Detection via Machine Learning on Multimodal Image Data
DOI:
https://doi.org/10.52825/isec.v2i.3319Keywords:
Occupancy Detection, Machine Learning, Camera SensorAbstract
High-resolution occupancy data can be used in control systems to precisely adapt building services systems to actual room usage patterns. This can improve energy efficiency and comfort. In addition to high accuracy, further improvements in system performance can be achieved by using additional information, such as the number and position of people in the room. Camera systems offer the potential to provide the necessary presence information with high accuracy and depth of information, but they are not widely accepted. Data processing in the device can offer higher acceptance rates. One goal of this work is therefore to investigate the extent to which a binary occupancy status can be derived from data processing at the edge device, extended to include information on number and position, but without identity. The underlying data streams can vary greatly depending on the camera system. These include RGB, depth, and thermal imaging data. The study therefore also examines the effects on accuracy and reliability that can be achieved with different multimodal camera data. The accuracy and reliability of data stream-related models for machine learning are compared with ground truth data from a structured laboratory study. All models demonstrate high accuracy. Segmented analysis of the area of interest facilitates precise localization. The convolutional neural network proves to be advantageous for person recognition. Implementation on a resource-constrained edge device enables fully decentralized data processing.
Downloads
References
[1] H. Yoshino, T. Hong, and N. Nord, “IEA EBC annex 53: Total energy use in buildings—Analysis and evaluation methods,” Energy Build., vol. 152, pp. 124–136, Oct. 2017, doi: 10.1016/j.enbuild.2017.07.038.
[2] International Energy Agency - Energy in Buildings and Communities (IEA EBC), “IEA EBC Annex 66 - Definition and Simulation of Occupancy Behavior in Buildings.” Accessed: Aug. 15, 2023. [Online]. Available: https://www.annex66.org/
[3] International Energy Agency - Energy in Buildings and Communities (IEA EBC), “IEA EBC Annex 79 - Occupant-Centric Building Design and Operation.” Accessed: Aug. 15, 2023. [Online]. Available: https://www.iea-ebc.org/projects/project?AnnexID=79
[4] D. Yan and T. Hong, “IEA EBC Annex 66 Final Report - Definition and simulation of occupant behavior in buildings,” International Energy Agency - IEA, 2018. Accessed: Aug. 15, 2023. [Online]. Available: https://annex66.org/sites/default/files/2018FinalReport/Annex%2066%20Final%20Report%20-%20read.pdf
[5] S. Hammes, M. Hauer, D. Geisler-Moroder, J. Weninger, R. Pfluger, and W. Pohl, “The impact of occupancy patterns on artificial light energy demand - simulation and post-occupancy-evaluation,” in Proceedings of Building Simulation 2021: 17th Conference of IBPSA, Bruges, Belgium: IBPSA, Sep. 2021, pp. 3536–3543. doi: 10.26868/25222708.2021.30508.
[6] A. D. Galasiu, G. R. Newsham, C. Suvagau, and D. M. Sander, “Energy Saving Lighting Control Systems for Open-Plan Offices: A Field Study,” LEUKOS, vol. 4, no. 1, pp. 7–29, Jul. 2007, doi: 10.1582/LEUKOS.2007.04.01.001.
[7] V. Garg and N. K. Bansal, “Smart occupancy sensors to reduce energy consumption,” Energy Build., vol. 32, no. 1, pp. 81–87, Jun. 2000, doi: 10.1016/S0378-7788(99)00040-7.
[8] T. Labeodan, C. De Bakker, A. Rosemann, and W. Zeiler, “On the application of wireless sensors and actuators network in existing buildings for occupancy detection and occupancy-driven lighting control,” Energy Build., vol. 127, pp. 75–83, Sep. 2016, doi: 10.1016/j.enbuild.2016.05.077.
[9] T. Kitzberger, J. Kotik, and T. Pröll, “Energy savings potential of occupancy-based HVAC control in laboratory buildings,” Energy Build., vol. 263, p. 112031, May 2022, doi: 10.1016/j.enbuild.2022.112031.
[10] Z. Pang et al., “Quantification of HVAC energy savings through occupancy presence sensors in an apartment setting: Field testing and inverse modeling approach,” Energy Build., vol. 302, p. 113752, Jan. 2024, doi: 10.1016/j.enbuild.2023.113752.
[11] I. Chew, D. Karunatilaka, C. P. Tan, and V. Kalavally, “Smart lighting: The way forward? Reviewing the past to shape the future,” Energy Build., vol. 149, pp. 180–191, Aug. 2017, doi: 10.1016/j.enbuild.2017.04.083.
[12] X. Guo, D. Tiller, G. Henze, and C. Waters, “The performance of occupancy-based lighting control systems: A review,” Light. Res. Technol., vol. 42, no. 4, Art. no. 4, Dec. 2010, doi: 10.1177/1477153510376225.
[13] G. Diraco, A. Leone, and P. Siciliano, “People occupancy detection and profiling with 3D depth sensors for building energy management,” Energy Build., vol. 92, pp. 246–266, Apr. 2015, doi: 10.1016/j.enbuild.2015.01.043.
[14] S. Naylor, M. Gillott, and T. Lau, “A review of occupant-centric building control strategies to reduce building energy use,” Renew. Sustain. Energy Rev., vol. 96, pp. 1–10, Nov. 2018, doi: 10.1016/j.rser.2018.07.019.
[15] S. Hammes, J. Weninger, R. Pfluger, and W. Pohl, “Take the Right Seat: The Influence of Occupancy Schemes on Performance Indicators of Lighting in Open Plan Offices,” Energies, vol. 15, no. 9, Art. no. 9, May 2022, doi: 10.3390/en15093378.
[16] S. Hammes and J. Weninger, “Optimizing User Distributions in Open-Plan Offices for Communication and Their Implications for Energy Demand and Light Doses: A Living Lab Case Study,” Buildings, vol. 15, no. 19, p. 3458, Sep. 2025, doi: 10.3390/buildings15193458.
[17] Y. Cheng, C. Fang, J. Yuan, and L. Zhu, “Design and Application of a Smart Lighting System Based on Distributed Wireless Sensor Networks,” Appl. Sci., vol. 10, no. 23, p. 8545, Nov. 2020, doi: 10.3390/app10238545.
[18] H. Zou, Y. Zhou, J. Yang, and C. J. Spanos, “Device-free occupancy detection and crowd counting in smart buildings with WiFi-enabled IoT,” Energy Build., vol. 174, pp. 309–322, Sep. 2018, doi: 10.1016/j.enbuild.2018.06.040.
[19] A. K. Mikkilineni, J. Dong, T. Kuruganti, and D. Fugate, “A novel occupancy detection solution using low-power IR-FPA based wireless occupancy sensor,” Energy Build., vol. 192, pp. 63–74, Jun. 2019, doi: 10.1016/j.enbuild.2019.03.022.
[20] S. Petersen, T. H. Pedersen, K. U. Nielsen, and M. D. Knudsen, “Establishing an image-based ground truth for validation of sensor data-based room occupancy detection,” Energy Build., vol. 130, pp. 787–793, Oct. 2016, doi: 10.1016/j.enbuild.2016.09.009.
[21] W. Zhang, J. Li, P. W. Tien, and J. K. Calautit, “Vision-based occupancy detection in indoor environments: A comparison of standard RGB and thermal cameras,” J. Build. Eng., vol. 114, p. 114215, Nov. 2025, doi: 10.1016/j.jobe.2025.114215.
[22] A. J. Amiri, K. Mahmoodi, A. Tootchi, and A. Razban, “A review of occupancy detection techniques for HVAC control: Advances and practical challenges,” J. Build. Eng., vol. 113, p. 113962, Nov. 2025, doi: 10.1016/j.jobe.2025.113962.
[23] H.-C. Shih, “A robust occupancy detection and tracking algorithm for the automatic monitoring and commissioning of a building,” Energy Build., vol. 77, pp. 270–280, Jul. 2014, doi: 10.1016/j.enbuild.2014.03.069.
[24] V. A. Sindagi and V. M. Patel, “A survey of recent advances in CNN-based single image crowd counting and density estimation,” Pattern Recognit. Lett., vol. 107, pp. 3–16, May 2018, doi: 10.1016/j.patrec.2017.07.007.
[25] N. O’Mahony et al., “Deep Learning vs. Traditional Computer Vision,” in Advances in Computer Vision, vol. 943, K. Arai and S. Kapoor, Eds., in Advances in Intelligent Systems and Computing, vol. 943. , Cham: Springer International Publishing, 2020, pp. 128–144. doi: 10.1007/978-3-030-17795-9_10.
[26] W. Zhang, J. Calautit, P. W. Tien, Y. Wu, and S. Wei, “Deep learning models for vision-based occupancy detection in high occupancy buildings,” J. Build. Eng., vol. 98, p. 111355, Dec. 2024, doi: 10.1016/j.jobe.2024.111355.
[27] V. Chidurala and X. Li, “Detection of moving objects using thermal imaging sensors for occupancy estimation,” Internet Things, vol. 17, p. 100487, Mar. 2022, doi: 10.1016/j.iot.2021.100487.
[28] W. Song and J. Calautit, “Impact of occupancy behavior on building energy efficiency: What’s next in detection and monitoring technologies?,” Energy, vol. 8, p. 100350, Jul. 2025, doi: 10.1016/j.nxener.2025.100350.
[29] A. Paxton and R. Dale, “Frame-differencing methods for measuring bodily synchrony in conversation,” Behav. Res. Methods, vol. 45, no. 2, pp. 329–343, Jun. 2013, doi: 10.3758/s13428-012-0249-2.
[30] Z. Zivkovic, “Improved adaptive Gaussian mixture model for background subtraction,” in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Cambridge, UK: IEEE, 2004, pp. 28-31 Vol.2. doi: 10.1109/ICPR.2004.1333992.
[31] C. Stauffer and W. E. L. Grimson, “Adaptive background mixture models for real-time tracking,” in Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), Fort Collins, CO, USA: IEEE Comput. Soc, 1999, pp. 246–252. doi: 10.1109/CVPR.1999.784637.
[32] R. Szeliski, Computer Vision: Algorithms and Applications. in Texts in Computer Science. Cham: Springer International Publishing, 2022. doi: 10.1007/978-3-030-34372-9.
[33] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539.
[34] W. Liu et al., “SSD: Single Shot MultiBox Detector,” in Computer Vision – ECCV 2016, vol. 9905, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds., in Lecture Notes in Computer Science, vol. 9905. , Cham: Springer International Publishing, 2016, pp. 21–37. doi: 10.1007/978-3-319-46448-0_2.
[35] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” 2015, arXiv. doi: 10.48550/ARXIV.1506.02640.
[36] S. Hanchinamani, P. Tigadi, P. Puttanavvar, D. Patilkulkarni, and P. Patil, “Integrating Keyframe Extraction and MoveNet Thunder for Robust Video Pose Estimation and Classification,” in 2024 5th International Conference for Emerging Technology (INCET), Belgaum, India: IEEE, May 2024, pp. 1–6. doi: 10.1109/INCET61516.2024.10593236.
[37] E. P. Washabaugh, T. A. Shanmugam, R. Ranganathan, and C. Krishnan, “Comparing the accuracy of open-source pose estimation methods for measuring gait kinematics,” Gait Posture, vol. 97, pp. 188–195, Sep. 2022, doi: 10.1016/j.gaitpost.2022.08.008.
[38] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature Pyramid Networks for Object Detection,” 2016, arXiv. doi: 10.48550/ARXIV.1612.03144.
[39] B. Haberl, “Entwicklung von Python-Skripten zur Ableitung des Anwesenheitsstatus als zeitbasierte binäre Sequenzen aus multimodalen Bilddaten.” University of Innsbruck, Austria, Oct. 03, 2025.
[40] M. V. Bavaresco, S. D’Oca, E. Ghisi, and R. Lamberts, “Methods used in social sciences that suit energy research: A literature review on qualitative methods to assess the human dimension of energy use in buildings,” Energy Build., vol. 209, p. 109702, Feb. 2020, doi: 10.1016/j.enbuild.2019.109702.
[41] D. Maniccia, B. Rutledge, M. S. Rea, and W. Morrow, “Occupant Use of Manual Lighting Controls in Private Offices,” J. Illum. Eng. Soc., vol. 28, no. 2, pp. 42–56, Jul. 1999, doi: 10.1080/00994480.1999.10748274.
[42] J. A. Rodríguez-Rodríguez, E. López-Rubio, J. A. Ángel-Ruiz, and M. A. Molina-Cabello, “The Impact of Noise and Brightness on Object Detection Methods,” Sensors, vol. 24, no. 3, p. 821, Jan. 2024, doi: 10.3390/s24030821.
[43] N. Kalbo, Y. Mirsky, A. Shabtai, and Y. Elovici, “The Security of IP-Based Video Surveillance Systems,” Sensors, vol. 20, no. 17, p. 4806, Aug. 2020, doi: 10.3390/s20174806.
[44] A. Bhardwaj, S. Bharany, A. Osman Ibrahim, A. Almogren, A. Ur Rehman, and H. Hamam, “Unmasking vulnerabilities by a pioneering approach to securing smart IoT cameras through threat surface analysis and dynamic metrics,” Egypt. Inform. J., vol. 27, p. 100513, Sep. 2024, doi: 10.1016/j.eij.2024.100513.
[45] Austrian Research Promotion Agency (FFG), “BOREALIS (Self-learning building controls for a greener and healthier society),” FFG Projektdatenbank. Accessed: Jan. 27, 2026. [Online]. Available: https://projekte.ffg.at/projekt/5121377
Published
How to Cite
Conference Proceedings Volume
Section
License
Copyright (c) 2026 Sascha Hammes, Benjamin Haberl, Johannes Weninger, Philipp Zech
This work is licensed under a Creative Commons Attribution 4.0 International License.
Funding data
-
Österreichische Forschungsförderungsgesellschaft
Grant numbers 910225