In the present day, technology has advanced significantly and has played a crucial role in various aspects of human development, including medical fields. Even entertainment technologies such as video games can be transformed into tools for physical therapy. One such example is the ‘Computer Vision-Based Interactive Application to Improve Eye-hand Coordination Skill in Occupational Therapy,’ commonly known as ‘Funny CV.’
Funny CV is the Senior Project undertaken by the 17th batch of Bachelor of Science in Information and Communication Technology (ICT International Program) students. The team consists of Mr. Pantakorn Sathapanasakul, Mr. Akkarapong Sirilertsoontorn, and Mr. Donlawat Sasawattecha, under supervision of Assoc. Prof. Dr. Worapan Kusakunniran, Program Chair of Ph.D. in Computer Science and the Head of the Machine Vision and Information Transfer Research Lab (MVIT Lab), who has expertise in Computer Vision, Image, and Video Processing.
“The project started when the students encountered children facing physical challenges at a physical therapy clinic. These children, dealing with movement disabilities due to brain-related impairments, required continuous therapy throughout their lives to maintain their functional abilities. The students observed that they could apply their expertise in Computer Vision and Image and Video Processing to this context. The students were firstly tasked with exploring how technology could be integrated into physical therapy activities. These activities include muscle movement, both small and large muscle groups, and overall body movements. Finally, the students came up with the topic, ‘Eye-hands coordinate,’ said Assoc. Prof. Dr. Worapan.”
The term “Eyes-hands coordinate” is a form of therapy designed to enhance coordination between the eyes and hands. This skill is crucial for various activities such as writing and daily life movements. Current treatment methods often involve having children hold a pencil and draw lines following a given pattern, such as drawing within circles. However, this activity only captures a certain level of interest because children are limited to paper-based tasks and require stimulation from parents or physiotherapists. Moreover, these methods lack the ability to collect statistical data on the depth information of the rehabilitation process. Therefore, the objective of this project is to explore how we can transition from drawing on paper to creating video games as a means of therapy.
“In the developed video game, children are tasked with mimicking hand gestures, standing in front of a camera, and selecting levels to play. The game focuses on skeletal tracking of key points such as the head, hand, and wrist to monitor and control the hand movements. The gameplay involves various levels, each designed to engage the child in different ways, such as drawing shapes or avoiding obstacles. Importantly, the game utilizes a Webcam Controller to accurately capture and record the child’s movements, providing valuable data for further analysis.”
The integration of game development tools ensured that the game was not only beneficial for therapy but also entertaining and challenging for the children. The levels were designed to be diverse, requiring different hand movements, making the experience enjoyable while serving its therapeutic purpose.
“The different levels are designed to allow children to play in various scenarios. For example, dragging their fingers to make a character eat fruit in one level, or in higher levels, controlling a panda to avoid bamboo obstacles. There is a total of 6 levels, each increasing in difficulty, making the game enjoyable due to its visual and auditory elements, as well as the increasing challenge of earning more points. While the child is playing, we collect data on how steady their hands are, track scrolling movements (how many times it hits the edges), and deduct points accordingly. Thus, we obtain more information about the child’s movements compared to traditional paper-based activities.”
In addition to game development tools, another crucial aspect of the Funny CV is the use of a Webcam Controller as a tool to assist in monitoring and collecting learning data of children. This is because using a pen, mouse, or keyboard may not accurately reflect the child’s movements in real-time.
“The Webcam Controller involves using any single camera connected to the computer, such as a webcam, to capture video of the player. Once the video is obtained, it undergoes Image and Video Processing to extract the hand position data of the child. This extraction yields X and Y coordinates, for example, at a position of 10, 28. Subsequently, we track the hand movements in the next frame, and the next set of position data may change to 10, 39. Tracking data in this X, Y axis manner provides us with information on both distance and direction of movement. This data can be interpreted and translated into the movement data of a character in the video game. We use this movement data to simulate and align with the typical rehabilitation activities of the child.”
Finally, Assoc. Prof. Dr. Worapan concluded his thoughts on the study of Computer Vision and its future developments by stating, ‘Currently, technology offers a wide range of tools, including AI, Machine Learning, and Computer Vision, which have become instrumental in addressing problems across various disciplines. For instance, they are used in detecting abnormalities within our bodies, such as X-rays, CT scans, and Magnetic Resonance Imaging (MRI). Therefore, if we were to draw a comparison, Computer Vision is essentially anything that humans can assess by looking at images or videos. It can perform tasks such as medical imaging, where it can identify cancer through CT scans, or biometrics, where it recognizes individuals based on their faces. Reflecting back on the Funny CV project, we may discover that the application of Computer Vision is not limited to Webcam Games or Rehab Games; rather, it extends to other sciences. Any field that involves decision-making or interpretation through visual data, like images or videos, holds the potential for Computer Vision to contribute and automate processes. However, the key to success lies in having sufficient datasets for development and providing a learning framework for Machine Learning.'”
Funny CV also won Honorable Mention in Invention Awards in Information Technology and Communication Arts Category from ‘National Research Award, Fiscal Year 2024’, held by the National Research Council of Thailand (NRCT). The announcement was made on October 6, 2023.