Automatic Collection of Production Performance Data Based on Multi-Object Tracking Algorithms

Hyuna Lim, Seojeong Oh, Hyeongjun Son, Yosep Oh


Recently, digital transformation in manufacturing has been accelerating. It results in that the data collection technologies from the shop-floor is becoming important. These approaches focus primarily on obtaining specific manufacturing data using various sensors and communication technologies. In order to expand the channel of field data collection, this study proposes a method to automatically collect manufacturing data based on vision-based artificial intelligence. This is to analyze real-time image information with the object detection and tracking technologies and to obtain manufacturing data. The research team collects object motion information for each frame by applying YOLO (You Only Look Once) and DeepSORT as object detection and tracking algorithms. Thereafter, the motion information is converted into two pieces of manufacturing data (production performance and time) through post-processing. A dynamically moving factory model is created to obtain training data for deep learning. In addition, operating scenarios are proposed to reproduce the shop-floor situation in the real world. The operating scenario assumes a flow-shop consisting of six facilities. As a result of collecting manufacturing data according to the operating scenarios, the accuracy was 96.3%.


Full Text:



Bae, G., Bae, S., Jeong, T., Heo, J., Moon, S., Song, S., Lee, S., Lee, J., Bae, Y., Jung, J., Na, H., Park, Y., Shin, G., Wang, J., and Kim,. B., “Smart Factory Management & Technology,” Dreamdesign, 2019.

Bewley, A., Ge, Z., Ott, L., Famos, F., and Upcroft, B., “Simple Online and Realtime Tracking,” arXiv:1602.00763, 2016.

Choi, W., and Cho, S., “Development of Mask Inspection Systems Based on Deep Learning,” IE Magazine, Vol. 29, No. 1, pp. 42-44, 2022.

Jeong, J., Kim, J., Chi, S., Roh, M., and Biggs, H., “Solitary Work Detection of Heavy Equipment Using Computer Vision,” KSCE Journal of Civil and Environmental Engineering Research, Vol. 41, No. 4, pp. 441-447, 2021.

Cho, C., Kim, J., Lee, S., Kim, B., Seo, Y., Park, Y., and Kwak, H.., “Vision-based Inspection Systems for Defect Detection,” Proceedings of The Korean Society of Manufacturing Technology Engineers Conference, No. 12, pp 34-34, 2021.

Jung, B., Seo, S., Park, B., and Bae, S., “Microscopic Traffic Parameters Estimation from UAV Video Using Multiple Object Tracking of Deep Learning-based,” The Journal of The Korea Institute of Intelligent Transportation Systems, Vol. 20, No. 5, pp. 83-99, 2021.

Kim, J., and Jeong, J., “Smart Warehouse Management System Utilizing IoT-based Autonomous Mobile Robot for SME Manufacturing Factory,” The Journal of the Institute of Internet, Broadcasting and Communication, Vol. 18, No. 5, pp. 237-244, 2018.

KSX9101-1, Manufacturing Business Data Interchange Between Manufacturing Enterprise Business Systems - Part 1: Data Model, 2021.

Lee, B., Liew, L., Cheach, W., and Wang, Y., “Occlusion handling in videos object tracking: A survey,” IOP Conference Series Earth and Environmental Science, Vol. 18, 2014.

Moon, S., Lee, J., Nam, D., and Kim, H., “A comparative study on multi object tracking methods for sports video,” Proceedings of the Korea Information Processing Society Conference, Vol. 23, No. 2, pp. 653-654, 2016.

Park, J., Hong, J., and Kim, W., “A Study on Intuitive IoT Interface System using 3D Depth Camera,” The Journal of Society for e-Business Studies, Vol. 22, No. 2, pp. 137-152, 2017.

Park, J., Park, D., Hyun, D., Na, Y., and Lee, S., “Deep-Learning Based Real-time Fire Detection Using Object Tracking Algorithm,” Journal of the Korea Society of Computer and Information, Vol. 27, No. 1, pp. 1-8, 2022.

Park, E., “Multi-Object Tracking Using Joint Detection and Identification Network based MLFPN,” Hanyang University, MS Thesis, 2021.

Redmon, J., Divvala, S., Girshick, R., and Farhadi, A., “You Only Look Once: Unified, Real-Time Object Detection,” arXiv:1506.02640, 2016.

Wojke, N., Bewley, A., and Paulus, D., “Simple Online and Realtime Tracking with a Deep Association Metric,” arXiv:1703.07402, 2017.

Yang, S., Jung, I., Kang, D., and Baek, H., “Real-Time Multi-Object Tracking using Mixture of SORT and DeepSORT,” The Journal of Korean Institute of Information Technology, Vol. 19, No. 10, pp. 1-9, 2021.

Yun, J., An, H., and Choi, Y., “A Machine Learning Based Facility Error Pattern Extraction Framework for Smart Manufacturing,” The Journal of Society for e-Business Studies, Vol. 23, No. 2, pp. 97-110, 2018.


  • There are currently no refbacks.