Gesture Recognition

原创于 2019-10-11 15:37:24 发布 · 1.2k 阅读

CC 4.0 BY-SA版权

文章标签：

“Gesture Recognition” allows Media Input devices(such as cameras) through the use of algorithms to recognize shapes and their movements according to their postures while giving them a meaning to be processed on machines.
Nowadays exist several types of recognition techniques and algorithms, such as 3D Model Based Algorithms, where the plain sight of an object is detected as a collection of apex and lines inside a 3D Grid, while it’s location is generated using it’s relative position in real time.
The present technique is based on Hidden Markov model (HMM).

There are a vast gamma of other algorithms and models for gesture recognition, according to the user needs.

By using different platforms and languages, it is possible to seize potential of the mentioned algorithms such as Unity with C# and Python.

For implementing the present explanation, the following tools were used:

Unity Version: 2019.1.11f1
Visual Studio Community 2017 - Version: 15.9.14
Python 3.7.4

In this app, an unity model will perform an action according to the hand gestures by the use of a virtual server and OpenCV to detect how many fingers are being detected by a webcam, for this, the information will be sent through the use of Sockets to a predefined ‘port’.

Some previous Knowledge
User Datagram Protocol (UDP)
UDP is an alternative communications protocol to Transmission Control Protocol (TCP) used primarily for establishing connections between applications on the internet.
These Protocols TCP run on top of the Internet Protocol (IP).
It provides port numbers to differentiate user requests.

By running the command python script, the webcam becomes active with two open windows showing the changes detected by it, a small frame will be captured in a green box， where the edges of the hand will be detected and captured by OpenCV.

在这里插入图片描述

Zambrano,Jose.(2019, October 11), "Five Fingers Pattern"

The script will allow recognizing gestures by comparing the changes between the number of edges detected from the hand, resulting into a change of state and a message display according the changes detected as showed above:
在这里插入图片描述

Zambrano,Jose.(2019, October 11), "Change from Full hand to 3 fingers showing (Mano Cornuta Pattern)"![在这里插入图片描述](https://img-blog.csdnimg.cn/2019101114423749.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L01hY3dvbGZ6,size_16,color_FFFFFF,t_70)Zambrano,Jose.(2019, October 11), "Change from Full hand to a fist showing (Jumping command)"

Lastly, this information will be sent in real-time to Unity via localhost, therefore, allowing to send different instructions to the Application’s avatar.
在这里插入图片描述

Zambrano,Jose.(2019, October 11), "Change of state in Unity" ![在这里插入图片描述](https://img-blog.csdnimg.cn/20191011144500733.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L01hY3dvbGZ6,size_16,color_FFFFFF,t_70)Zambrano,José.(2019, October 11), "Jumping command sent from OpenCV to Unity using UDP"

Reference:

UDP explanation
https://www.techopedia.com/definition/13460/user-datagram-protocol-udp
Markov’s hidden models
http://papers.nips.cc/paper/1956-the-infinite-hidden-markov-model.pdf
Learning and 3D object Recognition System through plain sights
http://webpersonal.uma.es/~CTO/papers/ISTANET04.pdf
Project Tutorial
https://www.raywenderlich.com/u/raunaqness
Using OpenCV with Unity
https://docs.opencv.org/3.1.0/d4/d73/tutorial_py_contours_begin.html