Android eye detection and tracking with OpenCV

Android eye detection and tracking with OpenCV

• Posted on: 28.1.2013 
• By: Roman Hošek 
• With: 120 Comments

This tutorial is for older samples, if you are starting with the new ones (2.4.6) please look at this updated tutorial. Method and principles of matching are the same for both.
Finally I found some time to write promised tutorial of eye detection and template matching on Android. Since OpenCV for Android is getting better and better, some code snippets can be old and not the best possible way to solve the problem. If you found some improvements, please comment or contact me, I will edit this post and share it to others.

We take standard OpenCV example for face detections and extends it a little.
Android OpenCV SDK can be found here
If you arent’n familiar with Eclipse and OpenCV yet, please read basic setup of opencv4android first.
Import Face detection sample to Eclipse and clean/build the project to be sure is correctly imported and working.

As you can se on the video, there are some differences in GUI against the pure sample. There is a slider to easily change the matching method and button to recreate the eye template.
So at first we add those elements to the GUI.

1. Open FdActivity.java and change content of LoaderCallbackInterface method with snippet below. Code is pretty straightforward – instead of putting instance FdView as content view, we programmatically create new layout, add button and verticalseekbar (this class is included in downloadable project at the end of page) to layout and pass the whole layout as content view.

   [java]
   // Create and set View
   mView = new FdView(mAppContext);
   mView.setDetectorType(mDetectorType);
   mView.setMinFaceSize(0.2f);

   VerticalSeekBar VerticalseekBar = new VerticalSeekBar(
   getApplicationContext());
   VerticalseekBar.setMax(5);
   VerticalseekBar.setPadding(20, 20, 20, 20);
   RelativeLayout.LayoutParams vsek = new RelativeLayout.LayoutParams(
   RelativeLayout.LayoutParams.WRAP_CONTENT, 400);
   vsek.addRule(RelativeLayout.ALIGN_PARENT_RIGHT);
   // Dont forget to set the id, or aligment will not work
   VerticalseekBar.setId(1);
   VerticalseekBar
   .setOnSeekBarChangeListener(new OnSeekBarChangeListener() {

   public void onProgressChanged(SeekBar seekBar,
   int progress, boolean fromUser) {

   method = progress;
   switch (method) {
   case 0:
   matching_method.setText("TM_SQDIFF");
   break;
   case 1:
   matching_method.setText("TM_SQDIFF_NORMED");
   break;
   case 2:
   matching_method.setText("TM_CCOEFF");
   break;
   case 3:
   matching_method.setText("TM_CCOEFF_NORMED");
   break;
   case 4:
   matching_method.setText("TM_CCORR");
   break;
   case 5:
   matching_method.setText("TM_CCORR_NORMED");
   break;
   }

   }

   public void onStartTrackingTouch(SeekBar seekBar) {
   }

   public void onStopTrackingTouch(SeekBar seekBar) {
   }
   });

   matching_method = new TextView(getApplicationContext());
   matching_method.setText("TM_SQDIFF");
   matching_method.setTextColor(Color.YELLOW);
   RelativeLayout.LayoutParams matching_method_param = new RelativeLayout.LayoutParams(
   RelativeLayout.LayoutParams.WRAP_CONTENT,
   RelativeLayout.LayoutParams.WRAP_CONTENT);
   matching_method_param
   .addRule(RelativeLayout.ALIGN_PARENT_RIGHT);
   matching_method_param.addRule(RelativeLayout.BELOW,
   VerticalseekBar.getId());

   Button btn = new Button(getApplicationContext());
   btn.setText("Create template");
   RelativeLayout.LayoutParams btnp = new RelativeLayout.LayoutParams(
   RelativeLayout.LayoutParams.WRAP_CONTENT,
   RelativeLayout.LayoutParams.WRAP_CONTENT);
   btnp.addRule(RelativeLayout.ALIGN_PARENT_LEFT);
   btn.setId(2);
   // Listen for click
   btn.setOnClickListener(new OnClickListener() {
   public void onClick(View v) {
   mView.resetLearFramesCount();
   }
   });

   RelativeLayout frameLayout = new RelativeLayout(
   getApplicationContext());
   frameLayout.addView(mView, 0);
   frameLayout.addView(btn, btnp);

   frameLayout.addView(VerticalseekBar, vsek);
   frameLayout.addView(matching_method, matching_method_param);

   setContentView(frameLayout);
   [/java]

2. For testing purposes, edit SampleCvViewBase.java line 35 as shown below. This will change video source to the front camera

   [java]
   mCamera = new VideoCapture(Highgui.CV_CAP_ANDROID+1);
   [/java]

3. Now we are ready to do the main part. Open FdView.java

   Add some Mats variables for zooming, templates and computations, Classificators for right and left eye

   [java]
   class FdView extends SampleCvViewBase {
   private static final String TAG = "Sample::FdView";
   private Mat mRgba;
   private Mat mGray;
   // Mats for zoom
   private Mat mZoomCorner;
   private Mat mZoomWindow;
   private Mat mZoomWindow2;
   // Helper Mat
   private Mat mResult;
   // Mat for templates
   private Mat teplateR;
   private Mat teplateL;

   private File mCascadeFile;
   private CascadeClassifier mJavaDetector;

   // Classifiers for left-right eyes
   private CascadeClassifier mCascadeER;
   private CascadeClassifier mCascadeEL;

   private DetectionBasedTracker mNativeDetector;

   private static final Scalar FACE_RECT_COLOR = new Scalar(0, 255, 0, 255);

   public static final int JAVA_DETECTOR = 0;
   public static final int NATIVE_DETECTOR = 1;

   // Matching methods
   private static final int TM_SQDIFF = 0;
   private static final int TM_SQDIFF_NORMED = 1;
   private static final int TM_CCOEFF = 2;
   private static final int TM_CCOEFF_NORMED = 3;
   private static final int TM_CCORR = 4;
   private static final int TM_CCORR_NORMED = 5;

   private int mDetectorType = JAVA_DETECTOR;

   private float mRelativeFaceSize = 0;
   private int mAbsoluteFaceSize = 0;
   // counter of learning frames
   private int learn_frames = 0;
   // match value
   private double match_value;
   // rectangle used to extract eye region – ROI
   private Rect eyearea = new Rect();
   [/java]

   Now we need to load cascade classifier files for left and right eye – haarcascade_lefteye_2splits.xml distributed with OpenCV package (data folder) I used the same classifier for both eyes, because for right eye, haarcascade_lefteye_2splits.xml gives me better results, than haarcascade_righteye_2splits.xml . But you can try it with both – simply rewrite the filename.
   Dont forget to copy haarcascade_lefteye_2splits.xml to /raw directory in your Android project (if is not present, simply create it)

   [java]
   public FdView(Context context) {
   super(context);

   try {
   InputStream is = context.getResources().openRawResource(R.raw.lbpcascade_frontalface);
   File cascadeDir = context.getDir("cascade", Context.MODE_PRIVATE);
   mCascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml");
   FileOutputStream os = new FileOutputStream(mCascadeFile);

   byte[] buffer = new byte[4096];
   int bytesRead;
   while ((bytesRead = is.read(buffer)) != -1) {
   os.write(buffer, 0, bytesRead);
   }
   is.close();
   os.close();

   // ——————————— load left eye classificator ———————————–
   InputStream iser = context.getResources().openRawResource(R.raw.haarcascade_lefteye_2splits);
   File cascadeDirER = context.getDir("cascadeER", Context.MODE_PRIVATE);
   File cascadeFileER = new File(cascadeDirER, "haarcascade_eye_right.xml");
   FileOutputStream oser = new FileOutputStream(cascadeFileER);

   byte[] bufferER = new byte[4096];
   int bytesReadER;
   while ((bytesReadER = iser.read(bufferER)) != -1) {
   oser.write(bufferER, 0, bytesReadER);
   }
   iser.close();
   oser.close();
   //—————————————————————————————————-

   // ——————————— load right eye classificator ————————————
   InputStream isel = context.getResources().openRawResource(R.raw.haarcascade_lefteye_2splits);
   File cascadeDirEL = context.getDir("cascadeEL", Context.MODE_PRIVATE);
   File cascadeFileEL = new File(cascadeDirEL, "haarcascade_eye_left.xml");
   FileOutputStream osel = new FileOutputStream(cascadeFileEL);

   byte[] bufferEL = new byte[4096];
   int bytesReadEL;
   while ((bytesReadEL = isel.read(bufferEL)) != -1) {
   osel.write(bufferEL, 0, bytesReadEL);
   }
   isel.close();
   osel.close();

   // ——————————————————————————————————

   mJavaDetector = new CascadeClassifier(mCascadeFile.getAbsolutePath());
   mCascadeER = new CascadeClassifier(cascadeFileER.getAbsolutePath());
   mCascadeEL = new CascadeClassifier(cascadeFileER.getAbsolutePath());
   if (mJavaDetector.empty()|| mCascadeER.empty() || mCascadeEL.empty()) {
   Log.e(TAG, "Failed to load cascade classifier");
   mJavaDetector = null;
   mCascadeER=null;
   mCascadeEL=null;
   } else
   Log.i(TAG, "Loaded cascade classifier from " + mCascadeFile.getAbsolutePath());

   mNativeDetector = new DetectionBasedTracker(mCascadeFile.getAbsolutePath(), 0);

   cascadeDir.delete();
   cascadeFileER.delete();
   cascadeDirER.delete();
   cascadeFileEL.delete();
   cascadeDirEL.delete();

   } catch (IOException e) {
   e.printStackTrace();
   Log.e(TAG, "Failed to load cascade. Exception thrown: " + e);
   }
   }
   [/java]

   Now edit the image processing:

   [java]
   @Override
   protected Bitmap processFrame(VideoCapture capture) {
   …
   MatOfRect faces = new MatOfRect();

   if (mDetectorType == JAVA_DETECTOR)
   {
   if (mJavaDetector != null)
   mJavaDetector.detectMultiScale(mGray, faces, 1.1, 2, 2 // TODO: objdetect.CV_HAAR_SCALE_IMAGE
   , new Size(mAbsoluteFaceSize, mAbsoluteFaceSize), new Size());

   // Prepare zoom mats
   if (mZoomCorner == null || mZoomWindow == null)
   CreateAuxiliaryMats();

   Rect[] facesArray = faces.toArray();
   // Iterate through all detected faces
   for (int i = 0; i < facesArray.length; i++){ Rect r = facesArray[i]; Core.rectangle(mGray, r.tl(), r.br(), new Scalar(0, 255, 0, 255), 3); // Draw rectangle around the face Core.rectangle(mRgba, r.tl(), r.br(), new Scalar(0, 255, 0, 255), 3); [/java]

   Now we detect face, right, nothing new, its face detection sample:) What about eyes? As face is found, it reduces our ROI (region of interest – where we will finding eyes) to face rectangle only. From face anatomy we can exclude the bottom part of face with mouth and some top part with forehead and hair. I could be computed relatively to the face size. See picture below – original image -> detected face -> eye area -> area splitted area for right, left eye. It saves computing power.
   

   [java]
   // compute the eye area
   eyearea = new Rect(r.x +r.width/8,(int)(r.y + (r.height/4.5)),r.width – 2*r.width/8,(int)( r.height/3.0));
   // split it
   Rect eyearea_right = new Rect(r.x +r.width/16,(int)(r.y + (r.height/4.5)),(r.width – 2*r.width/16)/2,(int)( r.height/3.0));
   Rect eyearea_left = new Rect(r.x +r.width/16 +(r.width – 2*r.width/16)/2,(int)(r.y + (r.height/4.5)),(r.width – 2*r.width/16)/2,(int)( r.height/3.0));
   // draw the area – mGray is working grayscale mat, if you want to see area in rgb preview, change mGray to mRgba
   Core.rectangle(mGray,eyearea_left.tl(),eyearea_left.br() , new Scalar(255,0, 0, 255), 2);
   Core.rectangle(mGray,eyearea_right.tl(),eyearea_right.br() , new Scalar(255, 0, 0, 255), 2);
   [/java]

   Count 5 first frames for learning – get_template function get classifier, area to detect, and desired size of new template.

   [java]
   if(learn_frames<5){
   teplateR = get_template(mCascadeER,eyearea_right,24);
   teplateL = get_template(mCascadeEL,eyearea_left,24);
   learn_frames++;
   }else{
   // Learning finished, use the new templates for template matching
   match_value = match_eye(eyearea_right,teplateR,FdActivity.method); //Or hardcode method you needs eg TM_SQDIFF_NORMED
   match_value = match_eye(eyearea_left,teplateL,FdActivity.method); //Or hardcode method you needs eg TM_SQDIFF_NORMED
   }
   // cut eye areas and put them to zoom windows
   Imgproc.resize(mRgba.submat(eyearea_left), mZoomWindow2, mZoomWindow2.size());
   Imgproc.resize(mRgba.submat(eyearea_right), mZoomWindow, mZoomWindow.size());

   }
   …
   [/java]

   Zoom method:

   [java]
   private void CreateAuxiliaryMats() {
   if (mGray.empty())
   return;

   int rows = mGray.rows();
   int cols = mGray.cols();

   if (mZoomWindow == null){
   mZoomWindow = mRgba.submat(rows / 2 + rows / 10 ,rows , cols / 2 + cols / 10, cols );
   mZoomWindow2 = mRgba.submat(0, rows / 2 – rows / 10 , cols / 2 + cols / 10, cols );
   }

   }
   [/java]

   Matching the template
   area – region of interest
   mTemplate – template of eye, created by get_template
   type – type of matching method
   returns matching score of template and desired area

   [java]
   private double match_eye(Rect area, Mat mTemplate,int type){
   Point matchLoc;
   Mat mROI = mGray.submat(area);
   int result_cols = mROI.cols() – mTemplate.cols() + 1;
   int result_rows = mROI.rows() – mTemplate.rows() + 1;
   //Check for bad template size
   if(mTemplate.cols()==0 ||mTemplate.rows()==0){
   return 0.0;
   }
   mResult = new Mat(result_cols, result_rows, CvType.CV_8U);

   switch (type){
   case TM_SQDIFF:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_SQDIFF) ;
   break;
   case TM_SQDIFF_NORMED:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_SQDIFF_NORMED) ;
   break;
   case TM_CCOEFF:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_CCOEFF) ;
   break;
   case TM_CCOEFF_NORMED:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_CCOEFF_NORMED) ;
   break;
   case TM_CCORR:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_CCORR) ;
   break;
   case TM_CCORR_NORMED:
   Imgproc.matchTemplate(mROI, mTemplate, mResult, Imgproc.TM_CCORR_NORMED) ;
   break;
   }

   Core.MinMaxLocResult mmres = Core.minMaxLoc(mResult);
   // there is difference in matching methods – best match is max/min value
   if(type == TM_SQDIFF || type == TM_SQDIFF_NORMED)
   { matchLoc = mmres.minLoc; }
   else
   { matchLoc = mmres.maxLoc; }

   Point matchLoc_tx = new Point(matchLoc.x+area.x,matchLoc.y+area.y);
   Point matchLoc_ty = new Point(matchLoc.x + mTemplate.cols() + area.x , matchLoc.y + mTemplate.rows()+area.y );

   Core.rectangle(mRgba, matchLoc_tx,matchLoc_ty, new Scalar(255, 255, 0, 255));

   if(type == TM_SQDIFF || type == TM_SQDIFF_NORMED)
   { return mmres.maxVal; }
   else
   { return mmres.minVal; }

   }
   [/java]

   On following picture you can see all ROI areas and matching in progress – yellow rectangles.

   
   Get template – find eye in desired roi by haar classifier, if eye is found, reduce roi to the eye only and search for the darkness point – pupil. Create rectangle of desired size, centered in pupil – our new eye template.
   

   [java]
   private Mat get_template(CascadeClassifier clasificator, Rect area,int size){
   Mat template = new Mat();
   Mat mROI = mGray.submat(area);
   MatOfRect eyes = new MatOfRect();
   Point iris = new Point();
   Rect eye_template = new Rect();
   clasificator.detectMultiScale(mROI, eyes, 1.15, 2,Objdetect.CASCADE_FIND_BIGGEST_OBJECT|Objdetect.CASCADE_SCALE_IMAGE, new Size(30,30),new Size());

   Rect[] eyesArray = eyes.toArray();
   for (int i = 0; i < eyesArray.length; i++){
   Rect e = eyesArray[i];
   e.x = area.x + e.x;
   e.y = area.y + e.y;
   Rect eye_only_rectangle = new Rect((int)e.tl().x,(int)( e.tl().y + e.height*0.4),(int)e.width,(int)(e.height*0.6));
   // reduce ROI
   mROI = mGray.submat(eye_only_rectangle);
   Mat vyrez = mRgba.submat(eye_only_rectangle);
   // find the darkness point
   Core.MinMaxLocResult mmG = Core.minMaxLoc(mROI);
   // draw point to visualise pupil
   Core.circle(vyrez, mmG.minLoc,2, new Scalar(255, 255, 255, 255),2);
   iris.x = mmG.minLoc.x + eye_only_rectangle.x;
   iris.y = mmG.minLoc.y + eye_only_rectangle.y;
   eye_template = new Rect((int)iris.x-size/2,(int)iris.y-size/2 ,size,size);
   Core.rectangle(mRgba,eye_template.tl(),eye_template.br(),new Scalar(255, 0, 0, 255), 2);
   // copy area to template
   template = (mGray.submat(eye_template)).clone();
   return template;
   }
   return template;
   }
   [/java]

   Red rectangles are templates and white dots are pupils.

   
   

   Download Eclipse Project 
   
   

Android eye OpenCV template matching