Tech Overview: Facial Recognition

Facial Recognition Technologies

Today’s post is regarding facial recognition technologies. This is not a complete overview of facial recognition and identification technologies. This is just a high-level overview of some facial recognition methodologies. 

Image-based recognition 

An image is stored in computer memory as an array of numbers, usually from 0-255.¹ Color images, for example are often stored using an indexed palette, where each pixel represents a palette color.² 

Figure 1: Indexed color  (Image adapted from Wikipedia)³

Similarly, an integral image is often used in computer vision applications such as facial recognition, and it is a summed-area table where each square represents the sum of the pixel values above and to the left of it.⁴ For example, in the integral image is shown in Figure 2, the value of 101 in the third row of the second column in 2. represents the sum of all the values in the first three rows of the first and second column in 1.  (31+2+12+26+13+17 = 101).⁵

Figure 2: Integral Image example (Image adapted from Wikipedia)⁶

Image Recognition Algorithms

Algorithms for recognition are often divided into two main approaches:  geometric or photometric.⁷

Geometric Approach

Geometric approaches are feature-based; they perform recognition by identifying useful facial features.⁸ A feature-based approach matches nodal points within the image to a template database in order to perform identification.⁹  Nodal points can include relative positions of features like eyes and noses and/or the size and shape of such features.¹⁰

Geometric, feature-based recognition typically follows a series of steps¹¹

• Detection
• Alignment
• Feature Extraction
• Matching

Video-chat filters, such as those one might use to turn themselves into a cat, likely use a facial detection method known as the Viola-Jones framework.¹² 

Figure 3: Lawyer Cat¹³

The Viola-Jones framework performs real-time scans of image data in the form of integral images.¹⁴ The use of integral images allows for quick identification of areas of contrast in the form of four types of rectangular features used in the framework.¹⁵ The features are shown in Figure 4, below. These features often correspond to areas such as the nose bridge and eye sockets; the nose bridge is lighter than surrounding areas, and eye sockets are darker than surrounding areas.¹⁶ Alone, this would not be reliable indicators that a face is present.¹⁷ However, If there are enough matches in one area of the image the framework makes a reasonable conclusion that a face is present.¹⁸   Note that this method is most accurate for images where the person is directly facing the camera.¹⁹   

Figure 4: a) Features used by the Viola-Jones Framework; b) Application of Viola Jones Features in Facial recognition (Image Adapted from Wikipedia)²⁰

As a sidenote, to map a snapchat filter onto your face by an “active shape model,” which is a model of an “average face” that was generated through machine learning of a training set of possible thousands of images in which facial features were manually marked. ²¹ This “average face” then scaled and aligned with what has been detected as your face. ²²  

Photometric Approach

Photometric approaches operate directly on the pixel values and make matches based on the intensity values for the picture and the stored template.²³ This sort of approach for axial recognition requires a database which is normalized to a compressed face representation and attempts to recognize the face in its entirety.²⁴  

An example of a photometric approach includes eigenfaces.²⁵  Eigenfaces can be generated on a database of images of human faces.²⁶  By performing statistical analysis on these images, eigenfaces are generated.²⁷ Eigenfaces appear as a pattern of light and dark areas, as shown in Figure 5, below.²⁸  New faces can be projected onto  a set of eigenfaces, and how the new face differs from those eigenfaces is recorded.²⁹ To identify the new face, its recorded variations are compared against “gallery images”, recorded variations of facial images that have been processed using the same set of eigenfaces.³⁰  Note that while an eigenface approach would use features,  no structure of the face is created with links between facial features as would be the case in a geometric approach.³¹

Figure 5: Some Eigenfaces from AT&T Labs Cambridge³²

3D Facial recognition

3D facial recognition employing infrared or near-infrared emitters and receivers have been employed as a security measure in systems such as FaceID and Xbox Kinect.³³ These ID verification methods work by projecting structured light, basically an array of infrared dots, onto a person’s face.³⁴ Figure 6, below shows an example of structured light.³⁵

Figure 6: Structured Light³⁶

Where the dots contact an object they will be distorted in comparison to their expected location from the system’s initial calibration.³⁷   Figure 7, below, shows an example of how the Kinect interprets the feedback from the infrared reflection to determine the depth of objects in its field of view.³⁸

Figure 7: Creation of a Depth image using a Kinect³⁹

By interpreting the distortions, the systems are able to create a 3D map of an individual’s face for use in future ID verification.⁴⁰

References:

1. 1. David Capello, Color Mode, ASPERITE,  https://www.aseprite.org/docs/color-mode/; Indexed Color, WIKIPEDIA, https://en.wikipedia.org/wiki/Indexed_color.
   2. Id.
   3. Indexed Color, WIKIPEDIA, https://en.wikipedia.org/wiki/Indexed_color.
   4. Integral Image, MATHWORKS, https://www.mathworks.com/help/images/integral-image.html; Summed-Area Table, WIKIPEDIA,  https://en.wikipedia.org/wiki/Summed-area_table.
   5. Id. 
   6. Summed-Area Table, WIKIPEDIA, https://en.wikipedia.org/wiki/Summed-area_table.
   7. Facial Recognition System, WIKIPEDIA,   https://en.wikipedia.org/wiki/Facial_recognition_system;  
   8. Id.
   9. Rean Neil Luces, Template-based versus Feature-based Template Matching, MEDIUM, https://medium.datadriveninvestor.com/template-based-versus-feature-based-template-matching-e6e77b2a3b3a; Facial Recognition System, WIKIPEDIA, https://en.wikipedia.org/wiki/Facial_recognition_system. 
   10. Facial Recognition System, WIKIPEDIA,  https://en.wikipedia.org/wiki/Facial_recognition_system.
   11. Id.
   12. Vox, How Snapchat’s filters work, YOUTUBE (Jun. 28, 2016), https://www.youtube.com/watch?v=Pc2aJxnmzh0; Thomas Smith, Here Is the Exact Video Filter That Created Lawyer Cat; DEBUGGER, https://debugger.medium.com/i-figured-out-the-exact-filter-that-created-lawyer-cat-fe1c3e8319c7.
   13. 394th District Court of Texas – Live Stream, Kitten Zoom Filter Mishap, YOUTUBE (Feb 9, 2021), https://www.youtube.com/watch?v=KxlPGPupdd8.
   14. Viola-Jones Object Detection Framework, WIKIPEDIA, https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework; Summed-Area Table, WIKIPEDIA, https://en.wikipedia.org/wiki/Summed-area_table.
   15. Vox, How Snapchat’s filters work, YOUTUBE (Jun. 28, 2016), https://www.youtube.com/watch?v=Pc2aJxnmzh0; Viola-Jones Object Detection Framework, WIKIPEDIA, https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework. 
   16. Id. 
   17. Id. 
   18. Id. 
   19. Id. 
   20. Viola-Jones Object Detection Framework, WIKIPEDIA, https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework.
   21. Vox, How Snapchat’s filters work, YOUTUBE (Jun. 28, 2016), https://www.youtube.com/watch?v=Pc2aJxnmzh0; U.S. Patent Publication No. 2015/0221118A1  (published Aug. 6, 2015)(Elena Shaburova, applicant). 
   22.  Id.
   23. Facial Recognition System, WIKIPEDIA, https://en.wikipedia.org/wiki/Facial_recognition_system.
   24. Id. 
   25. Id. 
   26. Maryna Longnickel, Eigenfaces, MEDIUM, https://medium.datadriveninvestor.com/template-based-versus-feature-based-template-matching-e6e77b2a3b3a; Eigenface, WIKIPEDIA,https://en.wikipedia.org/wiki/Eigenface.
   27. Id.
   28. Id. 
   29. Id. 
   30. Id.  
   31. Facial Recognition System, WIKIPEDIA, https://en.wikipedia.org/wiki/Facial_recognition_system.
   32. Id.
   33. Facial Recognition System, WIKIPEDIA,  https://en.wikipedia.org/wiki/Facial_recognition_system.
   34. Kinect, WIKIPEDIA, https://en.wikipedia.org/wiki/Kinect.
   35. Structured Light, WIKIPEDIA, https://en.wikipedia.org/wiki/Structured_light.
   36. Id. 
   37. Id.
   38. Kinect, WIKIPEDIA, https://en.wikipedia.org/wiki/Kinect.
   39. Id.
   40. Structured Light, WIKIPEDIA, https://en.wikipedia.org/wiki/Structured_light; Kinect, WIKIPEDIA, https://en.wikipedia.org/wiki/Kinect.