OpenPose EfficientDet Feedback Supports Upper Limb Rehabilitation Robots in Clinical Motion Training with Environmental Cues in Clinics

Authors

  • Lydia Bennett Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia Author
  • Mathieu Rolland Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia Author
  • Serena Ferrari Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia Author
  • Federica Ferrara Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia Author

DOI:

https://doi.org/10.66525/ijictt99

Keywords:

Upper-Limb Rehabilitation, OpenPose, EfficientDet, Robotic Feedback

Abstract

Neurological disorders, particularly stroke, represent a leading cause of long-term disability worldwide, heavily impacting upper-limb motor function and diminishing the overall quality of life for millions of individuals. Traditional upper-limb rehabilitation relies extensively on manual therapy, which is highly labor-intensive, difficult to quantify, and restricted by the availability of specialized physical therapists. In response to these challenges, robotic rehabilitation has emerged as a promising adjunct to clinical practice. However, many current robotic systems lack real-time, context-aware visual feedback mechanisms that can accurately interpret both patient kinematics and interaction with therapeutic objects. This paper proposes a novel framework integrating the OpenPose human pose estimation algorithm and the EfficientDet object detection architecture to provide comprehensive motion and environmental feedback for upper-limb rehabilitation robots in clinical settings. By combining markerless skeletal tracking with high-efficiency object recognition, the proposed system enables robots to adaptively adjust task difficulty and provide corrective guidance based on the precise spatial relationship between the patient and the target objects. The methodology outlines the integration of these two deep learning pipelines, detailing the synchronization and data fusion processes required for low-latency clinical applications. Experimental validation in a controlled clinical environment demonstrates significant improvements in kinematic tracking accuracy, therapeutic object interaction detection, and overall system responsiveness. Furthermore, qualitative assessments suggest enhanced patient engagement and adherence due to the immediate, visually driven corrective feedback. The findings indicate that the unified OpenPose-EfficientDet framework offers a robust, scalable, and highly effective solution for advancing intelligent robotic rehabilitation therapies.

Downloads

Download data is not yet available.

References

1. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 27325-27335).

2. Lv, Q., Kong, W., Li, H., Zeng, J., Qiu, Z., Qu, D., ... & Pang, J. (2025). F1: A vision-language-action model bridging understanding and generation to actions. arXiv preprint arXiv:2509.06951.

3. Li, B., Zhang, R., Liang, H., Zhang, J., Zhang, J., Chen, X., ... & Wang, J. (2026). Interagent: Physics-based multi-agent command execution via diffusion on interaction graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15253-15265).

4. Wang, Siheng, et al. "DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection." arXiv preprint arXiv:2604.02753 (2026).

5. Li, X., Wang, P., Li, G., & Zhang, Y. (2023). Design of a novel self-test-on-chip interface ASIC for capacitive accelerometers. IEEE Transactions on Circuits and Systems I: Regular Papers, 70(7), 2834-2843.

6. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems.

7. Yu, D., Mahamadu, A. M., Chen, W., & Li, N. (2026). Understanding the challenges and future trends of immersive technology use in indoor pedestrian and evacuation dynamics: A systematic review. Journal of Building Engineering, 115703.

8. Kavraki, L. E., Švestka, P., Latombe, J.-C., & Overmars, M. H. (1996). Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Automation, 12(4), 566–580.

9. Wang, Y. Y., Lou, M., Wang, Y., Wu, W. G., & Yang, F. (2022). Stochastic failure analysis of reinforced thermoplastic pipes under axial loading and internal pressure. China Ocean Engineering, 36(4), 614-628.

10. Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., et al. (2023). PaLM-E: An embodied multimodal language model. In Proceedings of ICML.

11. Xiao, Q., & Ghaffari, M. (2026). LaWM: Least Action World Models for Long-Horizon Physical Consistency from Visual Observations. arXiv preprint arXiv:2605.08279.

12. Peng, Q., Bai, C., Zhang, G., Xu, B., Liu, X., Zheng, X., ... & Lu, C. (2025, October). NavigScene: Bridging local perception and global navigation for beyond-visual-range autonomous driving. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 4193-4202).

13. Wan, H., Cheng, J., Deng, Y., Wu, D., Chen, Y., Lin, Z., ... & Ji, X. Towards Physics Aware Embodied Control with Graph based Object-centric Learning. ACM Transactions on Cyber-Physical Systems.

14. Lv, Qi, et al. "Spatial-temporal graph diffusion policy with kinematic modeling for bimanual robotic manipulation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

15. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of CVPR.

16. Wan, H., Zhang, Y., Wang, J., Wu, D., Li, M., Chen, X., ... & Ji, X. (2025). Toward universal embodied planning in scalable heterogeneous field robots collaboration and control. Journal of Field Robotics, 42(5), 2318-2336.

17. Li, Q., Luo, H., Cheng, H., Deng, Y., Sun, W., Li, W., & Liu, Z. (2023). Incipient fault detection in power distribution system: A time-frequency embedded deep-learning-based approach. IEEE Transactions on Instrumentation and Measurement, 72, 1–14.

18. Wang, K., Hou, W., Hong, L., & Guo, J. (2025). Smart transparency: A user-centered approach to improving human–machine interaction in high-risk supervisory control tasks. Electronics, 14(3), 420.

19. Luo, H., Feng, Q., Sun, W., Li, W., Li, Q., Zhao, W., & Liu, Z. (2026). FD-Mamba with neural observer and frequency-enhanced update for incipient feeder fault detection. IEEE Internet of Things Journal, 13(10), 20761–20772.

20. Cheng, H., Luo, H., Liu, Z., Sun, W., Li, W., & Li, Q. (2025). Reinforcement learning based robust volt/var control in active distribution networks with imprecisely known delay. IEEE Transactions on Computational Social Systems, 1–12.

21. Wang, Y., Fang, Y., Wang, T., Feng, Y., Tan, Y., Zhang, S., ... & Xu, R. (2025). Dreamnav: A trajectory-based imaginative framework for zero-shot vision-and-language navigation. arXiv preprint arXiv:2509.11197.

22. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024). A conditional denoising diffusion probabilistic model for point cloud upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20786-20795).

23. Yang, Y., Wang, Q., Li, C., Hu, H., Wu, C., Jiang, Y., ... & Xu, B. (2025, May). Fewer may be better: Enhancing offline reinforcement learning with reduced dataset. In International Conference on Learning Representations (Vol. 2025, pp. 7076-7098).

24. Zhu, W., Zhang, Y., Jin, X., Zeng, W., & Zhang, L. (2025, October). Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 3565-3574).

25. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems.

26. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of ICLR.

27. Wang, H., Li, Z., Zhang, Y., He, Z., Jiang, L., Li, K., ... & Gu, D. (2026). ConsistNav: Closing the Action Consistency Gap in Zero-Shot Object Navigation with Semantic Executive Control. arXiv preprint arXiv:2605.09869.

28. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

29. Wang, C., Ke, L., Liu, X., Huang, D., Yao, J., & Chu, W. Explainable AI Analysis of Brake Control in CARLA Through Reference and Distilled Policies. Frontiers in Robotics and AI, 13, 1847808.

30. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Harley, T., Lillicrap, T., et al. (2016). Asynchronous methods for deep reinforcement learning. In Proceedings of ICML.

31. Wan, H., Chen, Y., Deng, Y., Wei, Z., Li, D., Lin, Z., ... & Ji, X. (2025, October). EmbodiedAgent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Control. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 12140-12146). IEEE.

32. Girshick, R. (2015). Fast R-CNN. In Proceedings of ICCV.

33. Li, Yanshu, et al. "Cama: Enhancing multimodal in-context learning with context-aware modulated attention." arXiv e-prints (2025): arXiv-2505.

34. Yang, Y., Hu, H., Li, W., Li, S., Yang, J., Zhao, Q., & Zhang, C. (2023, June). Flow to control: Offline reinforcement learning with lossless primitive discovery. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 9, pp. 10843-10851).

35. Yang, Z., Ji, W., & Wang, Z. (2024). Adaptive joint configuration optimization for collaborative inference in edge-cloud systems. Science China Information Sciences, 67(4), 149103.

36. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., et al. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.

37. Sui, M., Jiang, L., Lyu, T., Wang, H., Zhou, L., Chen, P., & Alhosain, A. (2024). Application of deep learning models based on efficientdet and openpose in user-oriented motion rehabilitation robot control. Journal of Intelligence Technology and Innovation, 2(3), 47-77.

38. Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., ... & Zhao, Q. (2021). Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 10299-10312.

39. Tang, A., Hiraoka, T., Hiraoka, N., Shi, F., Kawaharazuka, K., Kojima, K., ... & Inaba, M. (2024, May). Humanmimic: Learning natural locomotion and transitions for humanoid robot via wasserstein adversarial imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 13107-13114). IEEE.

40. Zhang, Luyan. "MCP: A Control-Theoretic Orchestration Framework for Synergistic Efficiency and Interpretability in Multimodal Large Language Models." arXiv preprint arXiv:2509.16597 (2025).

Downloads

Published

2026-06-01