FACIAL DETECTION, RECOGNITION AND BOOKMARKING IN VIDEOS

• US 20160148651A1
• Filed: 02/01/2016
• Published: 05/26/2016
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Steiner, Matthew S.
• Granted Patent: US 9,984,729 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for facial bookmarking in videos. In one aspect, methods include receiving a digital video comprising video data, processing the video data to detect features indicative of a human face in the digital video, determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video, determining first timestamp data corresponding to the first frame, determining, from the video data, a second frame, in which the features indicative of the human face are detected in the digital video, determining second timestamp data corresponding to the second frame, generating an identifier corresponding to the human face, generating a data set including the identifier, the first timestamp data and the second timestamp data, and appending the data set to the video data to provide annotated video data.

Claims

1. A system comprising:

a data processing apparatus; and

a computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving a digital video comprising video data;

processing the video data to detect features indicative of a first human face and a second human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the first human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

generating a first identifier corresponding to the first human face;

determining, from the video data, a second frame, in which the features indicative of the second human face are detected in the digital video;

determining second timestamp data corresponding to the second frame;

generating a second identifier corresponding to the second human face;

generating a data set comprising the first identifier, the second identifier, the first timestamp data, and the second timestamp data; and

appending the data set to the video data to provide annotated video data.

2. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining that the first identifier corresponds to a first member of a social networking service;

comparing the second human face to one or more profile images of one or more other members of the social networking service that are contacts of the first member of the social networking service; and

determining the second identifier from the one or more profile images of the one or more other members of the social networking service.

3. The system of claim 2, wherein the members of the social networking service are connected in a social graph.

4. The system of claim 2, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining that the first identifier corresponds to the first member of the social networking service; and

linking the first identifier with a profile corresponding to the first member of the social networking service such that, upon selection of the link, information about the profile is displayed on a user device.

5. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining that the first identifier corresponds to a first member of a social networking service; and

enabling at least one of any user of a social network and contacts of the first member to perform at least one of view and comment on the digital video.

6. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining that the first timestamp data indicates a first appearance of the first human face in the digital video;

determining that the second timestamp data indicates a last appearance of the first human face in the digital video; and

providing an annotation that indicates a location of the first human face in the digital video while the digital video is playing the first frame, the second frame, and other frames that occur between the first frame and the second frame, wherein the first human face is undetectable in at least one of the other frames.

7. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

determining a privacy setting associated with a first member of a social networking service; and

based on the privacy setting, generating an anonymous bookmark for the first human face in the digital video that includes generic information about the first human face instead of identifying information about the first member of the social networking service.

8. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a vector of image feature data from the first frame;

processing the vector of image feature data for characteristics, wherein the characteristics include at least one of a facial feature, a skin color, and a skin texture; and

comparing the characteristics to the second human face in the second frame to determine that the first human face in the first frame is different from the second human face in the second frame.

9. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

generating a bookmark that identifies that the first identifier appears in the first frame; and

providing the bookmark to a user;

responsive to receiving a selection of the bookmark from the user, advancing a presented frame in the digital video to the first frame.

10. The system of claim 1, wherein the computer program further includes instructions that cause the data processing apparatus to perform operations comprising:

receiving an identification of the first human face from a user; and

wherein appending the data set to the video data to provide annotated video data includes adding the identification of the first human face to the data set.

11. A non-transitory computer-readable medium coupled to one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a digital video comprising video data;

processing the video data to detect features indicative of a human face in the digital video;

determining, from the video data, a first frame, in which the features indicative of the human face are detected in the digital video;

determining first timestamp data corresponding to the first frame;

generating an identifier corresponding to the human face;

generating a data set comprising the identifier and the first timestamp data;

determining that the identifier corresponds to a first member of a social networking service;

linking the first identifier with a profile corresponding to the first member of the social networking service;

appending the data set to the video data to provide annotated video data; and

providing an indicator that indicates a location of the human face in the digital video while the digital video is playing, wherein the indicator is an annotation associated with the human face, the indicator includes the identifier of the human face, and selection of the indicator causes information about the profile corresponding to the first member of the social networking service to be displayed on a user device.

12. The computer-readable medium of claim 11, wherein the identifier corresponding to the human face is a name of the human face.