Facial detection, recognition and bookmarking in videos

• US 9,984,729 B2
• Filed: 02/01/2016
• Issued: 05/29/2018
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Steiner, Matthew S.
• Primary Examiner(s): HUNTER, MISHAWN N
• Application Number:
  US15/011,884
• Publication Number:
  US 20160148651A1
• Time in Patent Office: 848 Days
• Field of Search: 386239, 386241, 386248
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for facial bookmarking in videos. In one aspect, methods include receiving a digital video comprising video data, processing the video data to detect features indicative of a human face in the digital video, determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video, determining first timestamp data corresponding to the first frame, determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video, determining second timestamp data corresponding to the second frame, generating an identifier corresponding to the human face, generating a data set including the identifier, the first timestamp data and the second timestamp data, and appending the data set to the video data to provide annotated video data.

Claims

1. A system comprising:

a data processing apparatus; and

a non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving a digital video comprising video data;

processing the video data to detect features indicative of a first human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the first human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

generating a first identifier corresponding to the first human face;

generating a data set comprising the first identifier and the first timestamp data;

determining that the first identifier corresponds to a first member of a social networking service;

linking the first identifier with a profile corresponding to the first member of the social networking service;

appending the data set to the video data to provide annotated video data; and

providing an indicator that indicates a presence of the first human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the first human face, the indicator includes the first identifier of the first human face, and selection of the indicator causes information about the profile corresponding to the first member of the social networking service to be displayed on a user device.

2. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

comparing a second human face to one or more profile images of one or more other members of the social networking service that are contacts of the first member of the social networking service; and

determining a second identifier for the second human face from the one or more profile images of the one or more other members of the social networking service.

3. The system of claim 2, wherein the one or more other members of the social networking service are connected in a social graph.

4. The system of claim 2, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining, from the video data, a second frame, in which the features indicative of the first human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

determining that the second timestamp data indicates a last appearance of the first human face in the digital video; and

appending the data set to the video data to provide the annotation that indicates the presence of the first human face in the digital video while the digital video is playing in the first frame, the second frame, and other frames that occur between the first frame and the second frame.

5. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

enabling at least one of any user of a social network and contacts of the first member to perform at least one of view and comment on the digital video.

6. The system of claim 4, wherein the first human face is not detected in at least one of the other frames based on determining that a back of a first human head faces a camera that records the digital video in at least one of the other frames and the first human face is identified as being present in the at least one of the other frames based on detection of at least one of a human anatomical feature, a clothing style, a clothing color, a skin tone, a hair style, and a hair color.

7. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining a privacy setting associated with the first member of the social networking service; and

based on the privacy setting, generating an anonymous bookmark for the first human face in the digital video that includes generic information about the first human face and excludes identifying information about the first member of the social networking service.

8. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a vector of image feature data from the first frame;

processing the vector of image feature data for characteristics, wherein the characteristics include at least one of a facial feature, a skin color, and a skin texture; and

comparing the characteristics to a second human face in a second frame to determine that the first human face in the first frame is different from the second human face in the second frame.

9. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a bookmark that identifies that the first identifier appears in the first frame;

providing the bookmark to a user; and

responsive to receiving a selection of the bookmark from the user, advancing a presented frame in the digital video to the first frame.

10. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

receiving an identification of the first human face from a user; and

wherein appending the data set to the video data to provide the annotation adding the identification of the first human face to the data set.

11. A non-transitory computer-readable medium coupled to one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier and the first timestamp data;

determining that the identifier corresponds to a first member of a social networking service;

linking the identifier with a profile corresponding to the first member of the social networking service;

appending the data set to the video data to provide annotated video data; and

providing an indicator that indicates a presence of the human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the human face, the indicator includes the identifier of the human face, and selection of the indicator causes information about the profile corresponding to the member of the social networking service to be displayed on a user device.

12. The computer-readable medium of claim 11, wherein the identifier corresponding to the human face is a name of the human face.

13. A computer-implemented method comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a first human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the first human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

generating a first identifier corresponding to the first human face;

generating a data set comprising the first identifier and the first timestamp data;

determining that the first identifier corresponds to a first member of a social networking service;

linking the first identifier with a profile corresponding to the first member of the social networking service;

appending the data set to the video data to provide annotated video data; and

providing an indicator that indicates a presence of the first human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the first human face, the indicator includes the first identifier of the first human face, and selection of the indicator causes information about the profile corresponding to the first member of the social networking service to be displayed on a user device.

14. The method of claim 13, further comprising:

comparing a second human face to one or more profile images of one or more other members of the social networking service that are contacts of the first member of the social networking service; and

determining a second identifier for the second human face from the one or more profile images of the one or more other members of the social networking service.

15. The method of claim 14, wherein the one or more other members of the social networking service are connected in a social graph.

16. The method of claim 14, further comprising:

determining, from the video data, a second frame, in which the features indicative of the first human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

determining that the second timestamp data indicates a last appearance of the first human face in the digital video; and

appending the data set to the video data to provide the annotation that indicates the presence of the first human face in the digital video while the digital video is playing in the first frame, the second frame, and other frames that occur between the first frame and the second frame.

17. The method of claim 13, further comprising:

enabling at least one of any user of a social network and contacts of the first member to perform at least one of view and comment on the digital video.

18. The method of claim 13, further comprising:

determining a privacy setting associated with the first member of the social networking service; and

based on the privacy setting, generating an anonymous bookmark for the first human face in the digital video that includes generic information about the first human face and excludes identifying information about the first member of the social networking service.

19. The method of claim 13, further comprising:

generating a vector of image feature data from the first frame;

processing the vector of image feature data for characteristics, wherein the characteristics include at least one of a facial feature, a skin color, and a skin texture; and

comparing the characteristics to a second human face in a second frame to determine that the first human face in the first frame is different from the second human face in the second frame.

20. The method of claim 13, further comprising:

generating a bookmark that identifies that the first identifier appears in the first frame;

providing the bookmark to a user; and

responsive to receiving a selection of the bookmark from the user, advancing a presented frame in the digital video to the first frame.