CS 488 Senior Design Project - Team Baby Watchers

Team Members: Colten Davis, Aidan Nelson, Saxon McFarland

Abstract

Our team was given the task to help analyze infant movement analysis, specifically infants rolling over from their back to their stomach. Roll analysis plays a large part in making safety equipment for infants, as well as identifying possible neuro-muscular deficiencies. Existing motion capture data was collected, as well as raw video footage. Our goal was to select and configure currently available open-source software to aid in movement analysis with the hopes of eliminating the need for the motion capture data. Research papers on the subject have shown several distinct roll types, as described by specific limb movements that aid in the rolling over movement. OpenPose was chosen as the software to use. Testing revealed the intense amount of computational power needed for OpenPose to run smoothly. With limited hardware resources, running OpenPose via a Windows portable version was our only choice. The raw videos themselves proved challenging due to distance, angle, and number of individuals present. OpenPose was not able to effectively track movement, and the output data whether JSON keypoints or skeletal overlays, was not useful. After exhaustive testing and research, we concluded that we could not provide any analysis into infant roll movement. Existing tools do exist to solve this problem; however, they are proprietary, and the licenses alone can cost into the thousands of dollars

Approach

Given the sponsor’s goal, we immediately knew this wasn’t a tool we could code from scratch. Computer vision is a field by itself, and programs that can do it took larger teams and years to complete. Our solution had to be an open-source tool we could simply configure to accomplish our goal. Searching for available tools was as easy as running a few google searches. There are only a few open-source programs out there ( later we’d find out why ). OpenPose presented itself as the go-to program for movement analysis, as it came with lightweight tools, had a simple GUI output, and could run easily from the command line. A few other programs were looked at, but the amount of adaptation that would need to be done was too great a task for our team. We choose OpenPose and began the testing phase.

Testing

Initially we did not have any videos from our sponsor team; due to videos having infants and their parents in the videos, there was a concern over confidentiality. The decision was made to blur the faces of the infants, then send over the videos. Testing began on simple YouTube videos while we waited for infant videos. We used the portable Windows version, essentially a demo of OpenPose. There is a Python API, but the implementation was more complicated than anticipated, and we could not get it working. Testing revealed that OpenPose was much more hardware intensive than initially anticipated. Smooth video is considered 24 frames per second (FPS), and with out setups we were getting 10-15 fps. This low FPS causes very low accuracy for OpenPose, as does higher resolutions. Lowering resolution did not have an effect, as OpenPose was still struggling.

Once we did get actual video footage of infants, the problems compounded. The videos were taken with a wide-angle GoPro. The distance was several feet, at an elevated angle. Also, since the infant could not be left alone, at times there would one to two individuals in the frame. Given all these factors, OpenPose could not accurately track movement. Output data was noisy and deemed useless for any type of analysis.

Results

As mentioned, we were unable to deliver a product capable of meeting our sponsors needs. We believe several factors influenced this outcome. First, computer vision itself is a difficult task. When researching a product to use, it was discovered there were far more paid programs, than free ones. It was obvious that the sheer amount of funding needed to create a good product was staggering. We found tools that would meet our sponsors needs with a few clicks, but the licensing fees were thousands of dollars, required the use of proprietary APIs etc. OpenPose, while considered lightweight, is still hardware intense. Our hardware, consisting of laptops, made OpenPose run sluggish at best, and no configuration tinkering would help. Also, do to having to blur the infant faces in the videos, we did not get infant videos until sprint 3, which immediately put an intense time crunch on our team. Once we had the videos, running OpenPose only once was what we considered the nail in the coffin on our project. The videos had several problems: distance, angle of capture, and number of individuals in the frame at any one time. All these compounded to render OpenPose’s ability to accurately track the infant movement and output useful data impossible.

Screenshots/Videos

Sources

• Kinovea Program
• OpenPose GitHub
• OpenPose Guide
• OpenPose Python API Guide