【Facial landmark detection】Detect eyes, nose, lips, and jaw with dlib, OpenCV, and Python

这篇文章是我们有关面部标志检测及其在计算机视觉和图像处理中的应用。我们使用检测到的面部标志来帮助我们标记和提取面部区域，包括：

• Mouth
• Right eyebrow
• Left eyebrow
• Right eye
• Left eye
• Nose
• Jaw

首先讨论与面部坐标相关的（x，y）坐标，以及如何将这些面部坐标映射到面部的特定区域。然后，我们将编写一些代码，这些代码可用于提取每个面部区域。我们将通过在一些示例图片上展示我们的方法的结果。到本博文结尾，您将对如何通过面部标志物（自动）提取面部区域有深刻的了解，并将这些知识应用于您自己的应用程序。

Facial landmark indexes for face regions（面部区域的面部标志索引）

在dlib中实现的面部界标检测器生成68个（x，y）坐标，这些坐标映射到特定的面部结构。通过在标记的iBUG 300-W数据集上训练预测器来获得这68个点映射。下面我们可以直观地看到这68个坐标分别映射到什么：

通过图像，我们可以看到可以通过简单的Python索引访问面部区域（假设使用Python零索引，也就是索引从0开始到67，上图中的索引从1开始到68）：

• The mouth can be accessed through points [48, 68]
• The right eyebrow through points [17, 22]
• The left eyebrow through points [22, 27]
• The right eye using [36, 42]
• The left eye with [42, 48]
• The nose using [27, 35]
• And the jaw via [0, 17]

这些映射在 imutils 库的 face_utils 中的 FACIAL_LANDMARKS_IDXS 字典中进行编码：

# define a dictionary that maps the indexes of the facial
# landmarks to specific face regions
FACIAL_LANDMARKS_IDXS = OrderedDict([
	("mouth", (48, 68)),
	("right_eyebrow", (17, 22)),
	("left_eyebrow", (22, 27)),
	("right_eye", (36, 42)),
	("left_eye", (42, 48)),
	("nose", (27, 35)),
	("jaw", (0, 17))
])

使用此字典，我们只需提供一个字符串作为键，就可以轻松地将索引提取到面部界标数组中并提取各种面部特征。

Visualizing facial landmarks with OpenCV and Python（使用OpenCV和Python可视化面部标志）

稍微困难一点的任务是可视化这些面部标志中的每一个并将结果覆盖在输入图像上。为此，我们需要已经包含在 imutils 库中的 visualize_facial_landmarks 函数：

def visualize_facial_landmarks(image, shape, colors=None, alpha=0.75):
	# create two copies of the input image -- one for the
	# overlay and one for the final output image
	overlay = image.copy() # 45
	output = image.copy() # 46
 
	# if the colors list is None, initialize it with a unique
	# color for each facial landmark region
	if colors is None: # 50
		colors = [(19, 199, 109), (79, 76, 240), (230, 159, 23),
			(168, 100, 168), (158, 163, 32),
			(163, 38, 32), (180, 42, 220)]

我们的 visualize_facial_landmarks 函数需要两个参数，然后是两个可选参数，每个参数均在下面详述：

• image : The image that we are going to draw our facial landmark visualizations on
• shape : The NumPy array that contains the 68 facial landmark coordinates that map to various facial parts.
• colors : A list of BGR tuples used to color-code each of the facial landmark regions.
• alpha : A parameter used to control the opacity of the overlay on the original image

第45和46行创建了我们输入图像的两个副本-我们将需要这些副本，以便可以在输出图像上绘制半透明的叠加层。

第50行进行检查以查看颜色列表是否为None，如果是，则使用BGR元组的预设列表对其进行初始化（请记住，OpenCV以BGR顺序而不是RGB的方式存储颜色/像素强度）。

现在，我们可以通过面部界标将每个面部区域可视化：

# loop over the facial landmark regions individually
	for (i, name) in enumerate(FACIAL_LANDMARKS_IDXS.keys()): # 56
		# grab the (x, y)-coordinates associated with the
		# face landmark
		(j, k) = FACIAL_LANDMARKS_IDXS[name]
		pts = shape[j:k]
 
		# check if are supposed to draw the jawline
		if name == "jaw": # 63
			# since the jawline is a non-enclosed facial region,
			# just draw lines between the (x, y)-coordinates
			for l in range(1, len(pts)):
				ptA = tuple(pts[l - 1])
				ptB = tuple(pts[l])
				cv2.line(overlay, ptA, ptB, colors[i], 2) # 69
 
		# otherwise, compute the convex hull of the facial
		# landmark coordinates points and display it
		else: # 73
			hull = cv2.convexHull(pts)
			cv2.drawContours(overlay, [hull], -1, colors[i], -1) # 75

在第56行，我们遍历了FACIAL_LANDMARKS_IDXS字典中的每个条目。对于这些区域中的每个区域，我们提取给定面部部分的索引，并从形状NumPy数组中获取（x，y）坐标。

第63-69行检查一下我们是否在绘制颌骨，如果是，我们只需在各个点之间循环，绘制一条将颌骨尖端连接在一起的线。否则，第73-75行处理计算点的凸包并在覆盖图上绘制该包。

最后一步是通过cv2.addWeighted函数创建一个透明的叠加层：

# apply the transparent overlay
	cv2.addWeighted(overlay, alpha, output, 1 - alpha, 0, output)
 
	# return the output image
	return output

在将 visualize_facial_landmarks 应用于图像和关联的面部地标之后，输出看起来类似于以下图像：

完整代码

# import the necessary packages
from collections import OrderedDict
import numpy as np
import cv2

# define a dictionary that maps the indexes of the facial
# landmarks to specific face regions
FACIAL_LANDMARKS_IDXS = OrderedDict([
	("mouth", (48, 68)),
	("right_eyebrow", (17, 22)),
	("left_eyebrow", (22, 27)),
	("right_eye", (36, 42)),
	("left_eye", (42, 48)),
	("nose", (27, 35)),
	("jaw", (0, 17))
])

def rect_to_bb(rect):
	# take a bounding predicted by dlib and convert it
	# to the format (x, y, w, h) as we would normally do
	# with OpenCV
	x = rect.left()
	y = rect.top()
	w = rect.right() - x
	h = rect.bottom() - y

	# return a tuple of (x, y, w, h)
	return (x, y, w, h)

def shape_to_np(shape, dtype="int"):
	# initialize the list of (x, y)-coordinates
	coords = np.zeros((68, 2), dtype=dtype)

	# loop over the 68 facial landmarks and convert them
	# to a 2-tuple of (x, y)-coordinates
	for i in range(0, 68):
		coords[i] = (shape.part(i).x, shape.part(i).y)

	# return the list of (x, y)-coordinates
	return coords

def visualize_facial_landmarks(image, shape, colors=None, alpha=0.75):
	# create two copies of the input image -- one for the
	# overlay and one for the final output image
	overlay = image.copy()
	output = image.copy()

	# if the colors list is None, initialize it with a unique
	# color for each facial landmark region
	if colors is None:
		colors = [(19, 199, 109), (79, 76, 240), (230, 159, 23),
			(168, 100, 168), (158, 163, 32),
			(163, 38, 32), (180, 42, 220)]

	# loop over the facial landmark regions individually
	for (i, name) in enumerate(FACIAL_LANDMARKS_IDXS.keys()):
		# grab the (x, y)-coordinates associated with the
		# face landmark
		(j, k) = FACIAL_LANDMARKS_IDXS[name]
		pts = shape[j:k]

		# check if are supposed to draw the jawline
		if name == "jaw":
			# since the jawline is a non-enclosed facial region,
			# just draw lines between the (x, y)-coordinates
			for l in range(1, len(pts)):
				ptA = tuple(pts[l - 1])
				ptB = tuple(pts[l])
				cv2.line(overlay, ptA, ptB, colors[i], 2)

		# otherwise, compute the convex hull of the facial
		# landmark coordinates points and display it
		else:
			hull = cv2.convexHull(pts)
			cv2.drawContours(overlay, [hull], -1, colors[i], -1)

	# apply the transparent overlay
	cv2.addWeighted(overlay, alpha, output, 1 - alpha, 0, output)

	# return the output image
	return output