Facial detection, recognition and bookmarking in videos

• US 9,251,854 B2
• Filed: 08/29/2011
• Issued: 02/02/2016
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Steiner, Matthew S.
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): Dunn, Mishawn
• Application Number:
  US13/219,820
• Publication Number:
  US 20120213490A1
• Time in Patent Office: 1,618 Days
• Field of Search: 386/239, 386/241, 386/248, 382/103, 382/118, 382/213, 382/218, 382/305
• US Class Current: 1/1
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for facial bookmarking in videos. In one aspect, methods include receiving a digital video comprising video data, processing the video data to detect features indicative of a human face in the digital video, determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video, determining first timestamp data corresponding to the first frame, determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video, determining second timestamp data corresponding to the second frame, generating an identifier corresponding to the human face, generating a data set including the identifier, the first timestamp data and the second timestamp data, and appending the data set to the video data to provide annotated video data.

Claims

1. A system comprising:

a data processing apparatus; and

a computer storage medium encoded with a computer program, the program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data, and the second timestamp data;

appending the data set to the video data to provide annotated video data; and

displaying an indicator that indicates a location of the human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the human face, and wherein the indicator includes the identifier of the human face.

2. The system of claim 1,

wherein the first timestamp data indicates a first appearance of the human face in the digital video, and wherein the second timestamp data indicates a last appearance of the human face in the digital video.

3. The system of claim 1, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a bookmark; and

storing the first timestamp data and the second timestamp data in the bookmark.

4. The system of claim 1, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising displaying an annotated video based on the annotated video data, the annotated video comprising a bookmark corresponding to the data set.

5. The system of claim 4, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

receiving user input based on the bookmark; and

advancing a presented frame in the digital video to the first frame in response to the user input.

6. The system of claim 1, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

comparing a first position of the features indicative of the human face in the first frame to a second position of features indicative of a second human face in the second frame; and

determining, based on the first position and the second position, that the features indicative of the second human face correspond to the features indicative of the human face.

7. The system of claim 1, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a facial model of the human face; and

comparing the facial model to known facial models.

8. The system of claim 7, wherein generating the facial model of the human face comprises generating a plurality of facial templates, each facial template corresponding to a frame in the video data in which the human face is detected, the facial model comprising the plurality of facial templates.

9. The system of claim 7, wherein each of the known facial models corresponds to a user of a social networking service.

10. The system of claim 7, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising updating a known facial model based on one or more of a plurality of facial templates of the facial model.

11. The system of claim 7, wherein the known facial models are each generated after receiving the digital video.

12. The system of claim 7, wherein the known facial models are deleted from computer-readable memory after comparing.

13. The system of claim 7, wherein the known facial models are accessed from a persistent storage device that electronically stores a database of known facial models.

14. The system of claim 7, wherein comparing the facial models comprises comparing each facial template of the facial model to facial templates of each of the known facial models.

15. The system of claim 7, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a confidence score between the facial model and a known facial model based on comparing the facial model to known facial models;

comparing the confidence score to a threshold confidence score; and

indicating that the facial model corresponds to the known facial model when the confidence score is greater than the threshold confidence score.

16. The system of claim 7, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining identity data corresponding to the known facial model;

associating the facial model with the identity data; and

appending the identity data to the data set.

17. The system of claim 16, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising storing the facial model as a known facial model.

18. The system of claim 1, wherein the program further includes instructions that cause the data processing apparatus to perform operations comprising receiving user input, the user input indicating an identity associated with the human face, and wherein the identifier corresponds to the identity.

19. A non-transitory computer-readable medium coupled to one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data and the second timestamp data;

appending the data set to the video data to provide annotated video data; and

displaying an indicator that indicates a location of the human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the human face, and wherein the indicator includes the identifier of the human face.

20. A computer-implemented method comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier, the first timestamp data and the second timestamp data;

appending the data set to the video data to provide annotated video data; and

displaying an indicator that indicates a location of the human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the human face, and wherein the indicator includes an identifier of the human face.