Method and system for automated annotation of persons in video content

• US 8,213,689 B2
• Filed: 07/14/2008
• Issued: 07/03/2012
• Est. Priority Date: 07/14/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yagnik, Jay, Zhao, Ming
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US12/172,939
• Publication Number:
  US 20100008547A1
• Time in Patent Office: 1,450 Days
• Field of Search: 382/118, 382/209, 382/218, 382/225, 382/278
• US Class Current: 382/118
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

Methods and systems for automated annotation of persons in video content are disclosed. In one embodiment, a method of identifying faces in a video includes the stages of: generating face tracks from input video streams; selecting key face images for each face track; clustering the face tracks to generate face clusters; creating face models from the face clusters; and correlating face models with a face model database. In another embodiment, a system for identifying faces in a video includes a face model database having face entries with face models and corresponding names, and a video face identifier module. In yet another embodiment, the system for identifying faces in a video can also have a face model generator.

Claims

1. A computer-implemented method of identifying faces in a video, comprising:

(a) generating one or more face tracks from at least one input video stream;

(b) selecting one or more key face images for each of the one or more face tracks;

(c) clustering the one or more face tracks to generate one or more face clusters, wherein each face cluster is associated with at least one of said key face images;

(d) creating one or more face models from respective ones of the one or more face clusters, including deriving the one or more face models directly from features of the one or more key face images associated with the respective face cluster; and

(e) correlating at least one of said face models with a face model database.

2. The computer-implemented method of claim 1, wherein generating one or more face tracks comprises:

(i) detecting a face in the at least one input video stream; and

(ii) tracking the face in the at least one input video stream.

3. The computer-implemented method of claim 2, wherein the detecting and the tracking is repeated at intervals for the duration of the input video stream.

4. The computer-implemented method of claim 1, further comprising:

(f) annotating at least one output video stream using data from the face model database.

5. The computer-implemented method of claim 4, further comprising:

(i) annotating at least one face track in the at least one output video stream.

6. The computer-implemented method of claim 1, wherein the correlating includes using a majority voting algorithm to correlate the one or more face clusters to the face model database.

7. The computer-implemented method of claim 1, wherein the correlating includes using a probabilistic voting algorithm to correlate the one or more face clusters to the face model database.

8. A system for identifying faces in a video, comprising:

(a) a face model database having face entries that include face models and corresponding names; and

(b) a video face identifier module, comprising;

(i) a face detection module that detects faces in an input video stream, generating one or more detected faces;

(ii) a face tracking module that tracks at least one of the one or more detected faces and generates one or more face tracks;

(iii) an intra-track face clustering module that generates one or more intra-track face clusters using the one or more face tracks;

(iv) an inter-track face clustering module that generates one or more inter-track clusters using the one or more intra-track face clusters;

(v) a detected face model generator module that generates one or more detected face models for the one or more detected faces using the one or more inter-track face clusters, wherein the one or more face models are derived directly from features of one or more selected face images associated with the face cluster; and

(vi) a model comparison module that compares the one or more detected face models with said face entries.

9. The system of claim 8, further comprising:

(c) a face model generator, wherein the face model generator comprises;

(i) a name generating module that generates a name list;

(ii) an image searching module that locates one or more images having at least one face associated with one or more corresponding names in the name list;

(iii) a face detection module that detects one or more target faces in the one or more images;

(iv) a face model generation module that generates one or more face models for at least one of the one or more target faces;

(v) a collection module that pair-wise stores the one or more face models and the one or more corresponding names; and

(vi) a consistency learning module wherein one or more representative face models are selected from the one or more face models, and wherein the one or more representative face models are stored in the face model database.

10. The system of claim 9, wherein the name list is generated based on names in documents accessible at one or more remote locations.

11. The system of claim 10, wherein documents include news archives.

12. The system of claim 8, further comprises:

(d) a face model update module that updates the face model database based on said detected faces.

13. The system of claim 8, wherein the face model database includes one or more entries comprising one or more face images one or more corresponding text labels wherein each entry represents one entry.

14. The system of claim 13, wherein the face model database further includes one or more entries directly input by the user, comprising one or more face images and one or more corresponding text labels.

15. The system of claim 8, wherein the face tracking module includes a facial feature based face tracker.

16. The system of claim 8, wherein the intra-track clustering includes removing noise clusters.

17. The system of claim 8, wherein the inter-track clustering includes removing noise clusters.

18. The system of claim 8, wherein the face detection module is invoked periodically and the face tracking module is invoked for faces detected by the face detection module.

19. The system of claim 8, further comprising:

(e) a video annotation module that annotates the input video stream producing an annotated output video stream.

20. The system of claim 8, further comprising:

(f) a query module that associates a user query with one or more entries in the face model database.