Hand Recognition Signals from Python to Unity

By using a Hand Recognition algorithm which captures and draws edges of the hand, and count how many vertex can find, according to the number of vertex can determine the shape that is shown through the Camera.

By detecting the changes of state, this algorithm will send information or messages by IP to any other software that may need them, as shown above.

Change of state detection

 if(count_defects == 0 and 3 in last):
            last = []
            sock.sendto( ("Rock!").encode(), (UDP_IP, UDP_PORT) )
            print("_"*10, "Shitou", "_"*10)

Unity’s application will receive this information by the following code:

client = new UdpClient(port); 
IPEndPoint anyIP = new IPEndPoint(IPAddress.Parse("0.0.0.0"), port); 
byte[] data = client.Receive(ref anyIP); 
string text = Encoding.UTF8.GetString(data); 

For easier understanding of the shape the, image captured by the WebCam is blurred by a Gaussian effect as showed here:

# Capture frames from the camera
    ret, frame = capture.read()
    
    # Get hand data from the rectangle sub window   
    cv2.rectangle(frame,(100,100),(300,300),(0,255,0),0)
    crop_image = frame[100:500, 100:500]
    
    # Apply Gaussian blur
    blur = cv2.GaussianBlur(crop_image, (3,3), 0)
    
    # Change color-space from BGR -> HSV
    hsv = cv2.cvtColor(blur, cv2.COLOR_BGR2HSV)
     # Create a binary image with where white will be skin colors and rest is black
    mask2 = cv2.inRange(hsv, np.array([2,0,0]), np.array([20,255,255]))
    
    # Kernel for morphological transformation    
    kernel = np.ones((5,5))
    
    # Apply morphological transformations to filter out the background noise
    dilation = cv2.dilate(mask2, kernel, iterations = 1)
    erosion = cv2.erode(dilation, kernel, iterations = 1)    
       
    # Apply Gaussian Blur and Threshold
    filtered = cv2.GaussianBlur(erosion, (3,3), 0)
    ret,thresh = cv2.threshold(filtered, 127, 255, 0)

Once obtained the messages explaining the hand gestures captured, Unity will interact according to each programmed sign, which are, Hand, Rock and Scissors
Rock Gesture

Scissors Gesture

Paper Gesture

According to the output messages received by the IP, then the C# will translate it into a text for further management, as shown below:

 port = 5065; 
 print("UDP Initialized");
 receiveThread = new Thread(new ThreadStart(ReceiveData)); 
 receiveThread.IsBackground = true; 
 receiveThread.Start(); 
        

this way the information will be ready for the user to manage the hand variables as he sees it fit.