Creating a narrative description of media content and applications thereof

• US 8,477,994 B1
• Filed: 02/26/2009
• Issued: 07/02/2013
• Est. Priority Date: 02/26/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Noshadi, Hyduke
• Primary Examiner(s): LE, BRIAN Q
• Application Number:
  US12/393,787
• Time in Patent Office: 1,587 Days
• Field of Search: 382/103, 382/170, 382/175, 382/180, 382/216, 382/284, 382/305, 382/306, 345/838, 345/433, 348/231.2, 358/1.18, 358/335, 358/452, 358/537, 707/2, 707/6, 707/7, 707/104.1, 707/200
• US Class Current: 382/103
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 1/00167 Processing or editing H04N1/00196 - H04N1/00201 take precedence
• CPC Class Codes: H04N 1/00196 Creation of a photo-montage, e.g. photoalbum H04N1/0019 takes precedence
• CPC Class Codes: H04N 2201/3261 of multimedia information, e.g. a sound signal
• CPC Class Codes: H04N 2201/3266 of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information

Abstract

This invention relates to creating a narrative description of media content. In an embodiment, a computer-implemented method describes content of a group of images. The group of images includes a first image and a second image. A first object in the first image is recognized to determine a first content data. A second object in the second image is recognized to determine a second content data. Finally, a narrative description of the group of images is determined according to a parameterized template and the first and second content data.

Claims

1. A method for describing content of a group of images, the group of images including a first image and a second image, comprising:

(a) recognizing a first object in the first image to determine a first content data that identifies the first object;

(b) recognizing a second object in the second image to determine a second content data that identifies the second object; and

(c) generating, by a computer, a narrative description of the group of images according to a parameterized template that specifies how to construct the narrative description from the first and second content data,

wherein each of the steps is performed on the computer.

2. The method of claim 1, wherein the recognizing (a) comprises recognizing a face in the first image to determine the first content data, wherein the first content data corresponds to an identity of the face.

3. The method of claim 1, wherein the recognizing (a) comprises recognizing a landmark in the first image to determine the first content data, wherein the first content data corresponds to the landmark.

4. The method of claim 1, wherein the recognizing (a) comprises extracting a metadata from the first image, and

wherein the determining (c) comprises determining the narrative description according to the metadata.

5. The method of claim 4, wherein the extracting the metadata comprises extracting a time when the first image was exposed.

6. The method of claim 4, wherein the extracting the metadata comprises extracting a location where the first image was exposed.

7. The method of claim 4, wherein the extracting the metadata comprises reading an EXIF header of the first image.

8. The method of claim 1, wherein the determining (c) comprises:

determining a value based on the first content data and the second content data; and

inserting the value into a parameter location in the template.

9. The method of claim 1, wherein the determining (c) comprises:

generating a plurality of sentences based on the template; and

assembling the sentences to form the narrative description.

10. The method of claim 1, wherein the group of images are at least a portion of a video and the first and second images are frames from the video.

11. The method of claim 1, wherein the narrative description describes content of the first and second images.

12. A system for describing media content, comprising:

a media database including a memory that stores a first image and a second image belonging to a group of images;

an object recognition module that recognizes a first object in the first image to determine a first content data that identifies the first object and recognizes a second object in the second image to determine a second content data that identifies the second object; and

a narrative compiler module that generates, by a computer, a narrative description of the group of images according to a parameterized template that specifies how to construct the narrative description from the first and second content data.

13. The system of claim 12, wherein the first object is a face and the first content data corresponds to an identity of the face.

14. The system of claim 12, wherein the first object is a landmark and the first content data corresponds to the landmark.

15. The system of claim 12, further comprising:

a metadata extractor module that extracts a metadata from the first image,

wherein the narrative compiler module determines the narrative description according to the metadata.

16. The system of claim 15, wherein the metadata comprises a time when the first image was exposed.

17. The system of claim 15, wherein the metadata includes a location where the first image was exposed.

18. The system of claim 15, wherein the metadata extractor module reads an EXIF header of the first image to extract the metadata.

19. The system of claim 12, wherein the narrative compiler module comprises:

a data formatter module that determines a value based on the first content data and the second content data; and

a sentence generator module that inserts the value into a parameter location of the template.

20. The system of claim 12, wherein the narrative compiler module comprises:

a sentence generator module that determines a plurality of sentences based on the template; and

a storyline assembler module that assembles the plurality of sentences to form the narrative description.

21. The system of claim 12, wherein the group of images are at least a portion of a video and the first and second images are frames from the video.

22. The system of claim 12, wherein the narrative description describes content of the first and second images.