FACIAL DETECTION, RECOGNITION AND BOOKMARKING IN VIDEOS

• US 20120213490A1
• Filed: 08/29/2011
• Published: 08/23/2012
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Steiner, Matthew S.
• Granted Patent: US 9,251,854 B2
• Time in Patent Office: Days
• US Class Current: 386/241
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for facial bookmarking in videos. In one aspect, methods include receiving a digital video comprising video data, processing the video data to detect features indicative of a human face in the digital video, determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video, determining first timestamp data corresponding to the first frame, determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video, determining second timestamp data corresponding to the second frame, generating an identifier corresponding to the human face, generating a data set including the identifier, the first timestamp data and the second timestamp data, and appending the data set to the video data to provide annotated video data.

Claims

1. A system comprising:

a data processing apparatus; and

a computer storage medium encoded with a computer program, the program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data and the second timestamp data; and

appending the data set to the video data to provide annotated video data.

2. The system of claim 1, the operations further comprising:

processing the video data to associate the features indicative of the human face to a particular person; and

determining identity data associated with the particular person, wherein the identifier is generated based on the identity data.

3. The system of claim 1, wherein processing the video data to detect features indicative of a human face comprises processing one or more frames of the digital video as an image, wherein each image is processed using facial detection techniques.

4. The system of claim 1, wherein the operations further comprise displaying an annotated video based on the annotated video data, the annotated video comprising a bookmark corresponding to the data set.

5. The system of claim 4, wherein the operations further comprise:

receiving user input based on the bookmark; and

advancing a presented frame in the digital video to the first frame in response to the user input.

6. The system of claim 1, wherein the operations further comprise:

comparing a first position of the features indicative of the human face in the first frame to a second position of features indicative of a second human face in the second frame; and

determining, based on the first position and the second position, that the features indicative of the second human face correspond to the features indicative of the human face.

7. The system of claim 1, wherein the operations further comprise:

generating a facial model of the human face; and

comparing the facial model to known facial models.

8. The system of claim 7, wherein generating the facial model of the human face comprises generating a plurality of facial templates, each facial template corresponding to a frame in the video data in which the human face is detected, the facial model comprising the plurality of facial templates.

9. The system of claim 7, wherein each of the known facial models corresponds to a user of a social networking service.

10. The system of claim 7, wherein the operations further comprise updating a known facial model based on one or more of a plurality of facial templates of the facial model.

11. The system of claim 7, wherein the known facial models are each generated after receiving the digital video.

12. The system of claim 7, wherein the known facial models are deleted from computer-readable memory after comparing.

13. The system of claim 7, wherein the known facial models are accessed from a persistent storage device that electronically stores a database of known facial models.

14. The system of claim 7, wherein comparing the facial models comprises comparing each facial template of the facial model to facial templates of each of the known facial models.

15. The system of claim 7, wherein the operations further comprise:

generating a confidence score between the facial model and a known facial model based on comparing the facial model to known facial models;

comparing the confidence score to a threshold confidence score; and

indicating that the facial model corresponds to the known facial model when the confidence score is greater than the threshold confidence score.

16. The system of claim 7, wherein the operations further comprise:

determining identity data corresponding to the known facial model;

associating the facial model with the identity data; and

appending the identity data to the data set.

17. The system of claim 16, wherein the operations further comprise storing the facial model as a known facial model.

18. The system of claim 1, wherein the operations further comprise receiving user input, the user input indicating an identity associated with the human face, wherein the identifier corresponds to the identity.

19. A computer-readable medium coupled to one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data and the second timestamp data; and

appending the data set to the video data to provide annotated video data.

20. A computer-implemented method, comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data and the second timestamp data; and

appending the data set to the video data to provide annotated video data.