Digital life recorder implementing enhanced facial recognition subsystem for acquiring face glossary data

• US 8,005,272 B2
• Filed: 12/31/2008
• Issued: 08/23/2011
• Est. Priority Date: 01/03/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Stevens, Mark B., Wilson, John David, Grim, Clifton E. III, Marzke, Rex Edward
• Primary Examiner(s): DWIVEDI, MAHESH H
• Application Number:
  US12/347,182
• Publication Number:
  US 20090174787A1
• Time in Patent Office: 965 Days
• Field of Search: 382/115, 382/118
• US Class Current: 382/118
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

Identifying individual facial images that are broadcast to enable optimized indexing and storage of facial information. Frames of data including faces are continually captured from a stream of incoming data. The facial frame data is extracted and processed into individual facial images. The individual facial images may be compared to existing facial image data in a database or cache to determine the identity of a facial image. The individual facial images may also be compared to facial images and metadata describing the facial images that are broadcast from external recording subsystems. The individual facial images stored to the glossary may be indexed based on the metadata received in the broadcast from an external recording subsystem or by metadata received from the continuous face frame capture.

Claims

1. A computer implemented method for identifying an individual in an image, the computer implemented method comprising:

capturing audio data and video data by a digital life recorder comprising a plurality of cameras positioned on a user, a plurality of microphones positioned on the user, a set of headphones positioned on the user, and a display device, wherein the display device is a mobile device and wherein the video data includes a continuous stream of images;

extracting, from the continuous stream of images, data that includes a set of facial frames;

responsive to capturing the data that includes the set of facial frames, automatically identifying, by a processing unit of the digital life recorder, an individual face in the set of facial frames based on metadata associated with the set of facial frames to form an identification, wherein the metadata associated with the set of facial frames includes a first time a facial frame in the set of facial frames including the individual face was captured by a camera in the plurality of cameras;

indexing the individual face identified in a glossary based on the metadata associated with the set of facial frames;

responsive to identifying the individual face, displaying the individual face and the identification of the individual face on the display device of the digital life recorder and requesting confirmation of the identification of the individual face from the user;

extracting a set of voice commands spoken by the user from the audio data, wherein extracting the set of voice commands spoken by the user from the audio data comprises;

recognizing a voice as that of the user and filtering the set of voice commands from the audio data captured;

identifying a second time a first voice command in the set of voice commands was spoken by the user;

executing the first voice command to identify an individual face in the set of facial frames by matching the second time the first voice command was spoken by the user with the first time the facial frame in the set of facial frames was captured by the camera, wherein the first voice command includes an identification from the user of the individual face in the set of facial frames;

executing a second voice command in the set of voice commands to control the capturing of the audio data and the video data by the plurality of cameras and the plurality of microphones of the digital life recorder positioned on the user;

obtaining feedback from the user about the capturing of the audio data and the video data by the digital life recorder using the set of headphones; and

using the feedback obtained to further control the capturing of the audio data and the video data by the digital life recorder.

2. The computer implemented method of claim 1, wherein automatically identifying the individual face in the set of facial frames comprises:

processing the set of facial frames to extract a set of single face images and metadata describing the set of single face images; and

comparing each face image in the set of single face images to a facial image data broadcast from an external recording subsystem.

3. The computer implemented method of claim 2 further comprising:

responsive to comparing each face image in the set of single face images to a facial image data broadcast from an external recording subsystem, determining an existence of a match;

responsive to the existence of the match, determining if an immediate identification is required;

responsive to an immediate identification being required, transmitting identification information to a monitor; and

responsive to an immediate identification not being required, transmitting identification information to a data processed queue.

4. The computer implemented method of claim 3 further comprising:

responsive to determining that a match does not exist, comparing the set of facial frames that have been processed to metadata in the glossary.

5. The computer implemented method of claim 1 further comprising:

responsive to the processing unit of the digital life recorder being unable to identify the individual face in the set of facial frames, displaying the individual face on the display device of the digital life recorder and requesting an identification of the individual face through the display device.

6. The computer implemented method of claim 1 further comprising:

detecting a connectivity of an external recording subsystem;

determining if identification information exists in the external recording subsystem; and

responsive to an existence of identification information, receiving facial and metadata information from the external recording subsystem.

17. The computer implemented method of claim 1, wherein the identification from the user of the individual face in the set of facial frames includes a name of a person pictured in the set of facial frames.

7. A computer program product comprising:

a computer readable storage medium tangibly embodying executable program instructions configured to identify an individual in an image;

first program instructions configured to capture audio data and video data by a digital life recorder comprising a plurality of cameras positioned on a user, a plurality of microphones positioned on the user, a set of headphones positioned on the user, and a display device, wherein the display device is a mobile device and wherein the video data includes a continuous stream of images;

second program instructions configured to extract from the continuous stream of images, data that includes a set of facial frames;

third program instructions configured to automatically identify, responsive to capturing the data that includes the set of facial frames, an individual face in the set of facial frames based on metadata associated with the set of facial frames to form an identification, wherein the metadata associated with the set of facial frames includes a first time a facial frame in the set of facial frames including the individual face was captured by a camera in the plurality of cameras;

fourth program instructions configured to index the individual face in a glossary based on the metadata associated with the set of facial frames;

fifth program instructions configured to display, responsive to identifying the individual face, the individual face and the identification of the individual face on the display device of the digital life recorder and request confirmation of the identification of the individual face from the user;

sixth program instructions configured to extract a set of voice commands spoken by the user from the audio data, wherein the sixth program instructions comprise;

program instructions configured to recognize a voice as that of the user and filtering the set of voice commands from the audio data captured;

seventh program instructions configured to identify a second time a first voice command in the set of voice commands was spoken by the user;

eighth program instructions configured to execute the first voice command to identify an individual face in the set of facial frames by matching the second time the first voice command was spoken by the user with the first time the facial frame in the set of facial frames was captured by the camera, wherein the first voice command includes an identification from the user of the individual face in the set of facial frames;

ninth program instructions configured to execute a second voice command in the set of voice commands to control the capture of the audio data and the video data by the plurality of cameras and the plurality of microphones of the digital life recorder positioned on the user;

tenth program instructions configured to obtain feedback from the user about the capturing of the audio data and the video data by the digital life recorder using the set of headphones; and

eleventh program instructions configured to use the feedback obtained to further control the capturing of the audio data and the video data by the digital life recorder, wherein the first through eleventh program instructions are stored on the computer readable storage medium.

8. The computer program product of claim 7, wherein the third program instructions comprise:

program instructions configured to process the set of facial frames to extract a set of single face images and metadata describing the set of single face images; and

program instructions configured to compare each face image in the set of single face images to a facial image data broadcast from an external recording subsystem.

9. The computer program product of claim 8 further comprising:

twelfth program instructions configured to determine, responsive to comparing each face image in the set of single face images to a facial image data broadcast from an external recording subsystem, an existence of a match;

thirteenth program instructions configured to determine, responsive to the existence of the match, if an immediate identification is required;

fourteenth program instructions that, responsive to an immediate identification being required, transmit identification information to a monitor; and

fifteenth program instructions configured to transmit, responsive to an immediate identification not being required, identification information to a data processed queue, wherein the twelfth through fifteenth program instructions are stored on the computer readable storage medium.

10. The computer program product of claim 9 further comprising:

sixteenth program instructions configured to compare, responsive to determining that a match does not exist, the set of facial frames that have been processed to metadata in the glossary, wherein the sixteenth program instructions are stored on the computer readable storage medium.

11. The computer program product of claim 7 further comprising:

seventeenth program instructions configured to compare, responsive to being unable to identify the individual face in the set of facial frames, the individual face on the display device of the digital life recorder and request an identification of the individual face through the display device, wherein the seventeenth program instructions are stored on the computer readable storage medium.

12. An apparatus comprising:

a bus system;

a memory connected to the bus system, wherein the memory includes a computer usable program code; and

a processing unit connected to the bus system, wherein the processing unit is configured to execute the computer usable program code to;

capture audio data and video data by a digital life recorder comprising a plurality of cameras positioned on a user, a plurality of microphones positioned on the user, a set of headphones positioned on the user, and a display device, wherein the display device is a mobile device and wherein the video data includes a continuous stream of images;

extract, from the continuous stream of images, data that includes a set of facial frames;

responsive to capturing the data that includes the set of facial frames, automatically identify an individual face in the set of facial frames based on metadata associated with the set of facial frames to form an identification, wherein the metadata associated with the set of facial frames includes a first time a facial frame in the set of facial frames including the individual face was captured by a camera in the plurality of cameras;

index the individual face in a glossary based on the metadata associated with the set of facial frames;

responsive to identifying the individual face, display the individual face and the identification of the individual face on the display device of the digital life recorder and request confirmation of the identification of the individual face from the user;

extract a set of voice commands spoken by the user from the audio data, wherein in executing the computer usable program code to extract the set of voice commands spoken by the user from the audio data the processing unit is further configured to execute the computer usable program code to;

recognize a voice as that of the user and filtering the set of voice commands from the audio data captured;

identify a second time a first voice command in the set of voice commands was spoken by the user;

execute the first voice command to identify an individual face in the set of facial frames by matching the second time the first voice command was spoken by the user with the first time the facial frame in the set of facial frames was captured by the camera, wherein the first voice command includes an identification from the user of the individual face in the set of facial frames;

execute a second voice command in the set of voice commands to control the capture of the audio data and the video data by the plurality of cameras and the plurality of microphones of the digital life recorder positioned on the user;

obtain feedback from the user about the capturing of the audio data and the video data by the digital life recorder using the set of headphones; and

use the feedback obtained to further control the capturing of the audio data and the video data by the digital life recorder.

13. The apparatus of claim 12, wherein in executing the computer usable program code to automatically identify the individual face in the set of facial frames, the processing unit is further configured to execute the computer usable program code to:

process the set of facial frames to extract a set of single face images and metadata describing the set of single face images; and

compare each face image in the set of single face images to facial image data broadcast from an external recording subsystem.

14. The apparatus of claim 13, wherein the processing unit is further configured to execute the computer usable program code to:

responsive to comparing each face image in the set of single face images to a facial image data broadcast from an external recording subsystem, determine an existence of a match;

responsive to the existence of the match, determining if an immediate identification is required;

responsive to an immediate identification being required, transmitting identification information to a monitor; and

responsive to an immediate identification not being required, transmitting identification information to a data processed queue.

15. The apparatus of claim 14, wherein the processing unit is further configured to execute the computer usable program code to:

responsive to determining that a match does not exist, compare the set of facial frames that have been processed to metadata in the glossary.

16. The apparatus of claim 13, wherein the processing unit is further configured to execute the computer usable program code to:

detect a connectivity of an external recording subsystem;

determine if identification information exists in the external recording subsystem; and

responsive to an existence of identification information, receive facial and metadata information from the external recording subsystem.