Method for creating view-based representations from multimedia collections

• US 11,170,037 B2
• Filed: 06/11/2015
• Issued: 11/09/2021
• Est. Priority Date: 06/11/2014
• Status: Active Grant

• Current Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Original Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Inventors: Wood, Mark D., Das, Madirakshi, Stubler, Peter O.
• Primary Examiner(s): Shah, Vaishali
• Application Number:
  US14/736,917
• Publication Number:
  US 20150363409A1
• Time in Patent Office: 2,343 Days
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/538 Presentation of query results
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

A system that is capable of generating a multiplicity of representations from a set of multimedia objects, each with a potentially different form, is disclosed. In addition, a system that is capable of generating representations that include thematic groupings in addition to the traditional purely sequential groupings is disclosed. Analysis of metadata to generate derived metadata creates a more comprehensive repository of information about media assets. Using the abundance of metadata, embodiments of the methods disclosed teach uniquely grouping and prioritizing media assets into a storyboard, which is in turn mapped onto view-based representations based on a selected output modality.

Claims

1. A method, comprising:

storing multimedia assets on a computer storage device, wherein the multimedia assets comprise still digital images and each still digital image comprises recorded metadata and user-provided metadata, wherein the recorded metadata is generated by a camera at a time the multimedia assets are captured by the camera, and wherein the recorded metadata is associated with the multimedia assets at the time the multimedia assets are captured by the camera;

extrapolating derived metadata from the recorded metadata and the user-provided metadata, by a computer processor;

organizing the multimedia assets into one or more groupings, based on the recorded metadata and the user-provided metadata, by the computer processor;

detecting faces in the multimedia assets, wherein each detected face is associated with an identity, by the computer processor;

determining people importance scores for the multimedia assets based on the respective identities of the detected faces in each multimedia asset, wherein the people importance scores are directly proportional to a frequency that the respective identities of the detected faces appear throughout all of the multimedia assets, by the computer processor;

prioritizing the one or more groupings, by the computer processor;

prioritizing the multimedia assets in each of the one or more groupings based on the people importance scores, by the computer processor;

receiving a selection of an output modality, wherein the output modality represents a consumer photo product that incorporates one or more of the multimedia assets;

selecting one or more of the multimedia assets from a most prioritized grouping of the one or more groupings, based on the prioritization of the multimedia assets, for incorporation into the output modality; and

generating and displaying on a display a view-based representation of the output modality incorporating the selected one or more multimedia assets.

2. The method of claim 1, further comprising saving on the computer storage device the prioritization of the multimedia assets independently of a selected output modality.

3. The method of claim 1, further comprising generating and displaying the view-based representation of the multimedia assets composited pursuant to emphasis scores assigned to the multimedia assets, wherein the emphasis scores are directly proportional to the amount of space each respective multimedia asset is allotted in the view-based representation.

4. The method of claim 1, wherein grouping the multimedia assets is performed according to a technique selected from the group consisting of:

chronology, frequent itemset mining, location clustering, object detection, object recognition, event detection, and event recognition.

5. The method of claim 4, further comprising ordering the multimedia assets into a hierarchical arrangement based on the grouping of the multimedia assets, by the computer processor.

6. The method of claim 1, further comprising grouping the multimedia assets that are semantically equivalent, based on the recorded metadata and the derived metadata.

7. The method of claim 6, wherein grouping the multimedia assets that are semantically equivalent is based on one or more of the following classes of grouping data:

temporal, location, and visual similarity.

8. The method of claim 7, wherein the visual similarity class of grouping data is determined by comparing block-based color histograms.

9. The method of claim 5, wherein the hierarchical arrangement is configured to arrange the multimedia assets in sequential order, based on the time each multimedia asset was captured by the camera.

10. The method of claim 1, wherein a graphic user interface is provided on the display comprising the view-based representation.

11. The method of claim 10, wherein the graphic user interface indicates a size of any groupings of the multimedia assets in the view-based representation.

12. The method of claim 1, wherein the output modality is selected from the group consisting of:

a multimedia presentation, an interactive multimedia presentation, a photobook, a printed collage, a virtual collage, or a digital slide show.

13. The method of claim 1, further comprising identifying a plurality of thematic sets, wherein each of the thematic sets is determined from the derived metadata and wherein each thematic set comprises a subset of the multimedia assets, by a computer processor.

14. The method of claim 1, wherein the derived metadata comprises one or more of the following metadata types:

a frequency of sharing the multimedia asset with others, a number of sharing recipients, and a number of social media interactions with the multimedia asset.

15. A method, comprising:

storing multimedia assets on a computer storage device, wherein the multimedia assets comprise still digital images;

storing recorded metadata and user-provided metadata on the computer storage device, wherein the recorded metadata and the user-provided metadata are associated with the still digital images, and wherein the recorded metadata is generated by a camera at a time the still digital images are captured by the camera;

extrapolating derived metadata from the recorded metadata and the user-provided metadata, by a computer processor;

organizing the multimedia assets into one or more groupings based on the recorded metadata and the user-provided metadata, by the computer processor;

detecting faces in the multimedia assets, wherein each detected face is associated with an identity, by the computer processor;

determining using the processor to determine people importance scores for the multimedia assets based on the respective identities of the detected faces in each multimedia asset, wherein the people importance scores are directly proportional to a frequency that the respective identities of the detected faces appear throughout all of the multimedia assets, by the computer processor;

prioritizing first the multimedia assets in each grouping based on the people importance scores, by the computer processor;

prioritizing second the one or more groupings, by the computer processor;

generating a storyboard, wherein the storyboard comprises a hierarchical arrangement of the one or more groupings;

mapping the storyboard as a view-based representation according to a selected output modality, wherein the selected output modality represents a consumer photo product that incorporates one or more of the multimedia assets in accordance with the storyboard; and

rendering on a display the view-based representation.

16. The method of claim 15, wherein the user-provided metadata comprises metadata provided by a user via an interface located on an image capture device or via an image editing application.

17. The method of claim 15, wherein extrapolating, by the computer processor, derived metadata comprises applying one or more of the techniques chosen from the group consisting of:

temporal event clustering, geographic naming, scene classification, materials class extraction, low-level feature extraction, content-based image retrieval, face detection, face recognition, facial clustering, semantic event clustering, image value indexing, and video key frame extraction.

18. The method of claim 15, wherein the user-provided metadata comprises metadata indicating a multimedia asset preference recorded by one or more users via a social networking application.

19. The method of claim 15, wherein the multimedia assets are organized into the one or more groupings according to a chronological paradigm.

20. The method of claim 15, wherein the multimedia assets are organized into the one or more groupings according to a thematic paradigm.

21. The method of claim 15, wherein the multimedia assets in the one or more groupings according to a hybrid chronological-thematic paradigm.

22. The method of claim 15, wherein the one or more groupings comprise sequential segments and parallel segments.

23. The method of claim 15, wherein the multimedia assets are organized into the one or more groupings by similar events depicted in the multimedia assets and according to a chronological paradigm of the events depicted in the multimedia assets.

24. The method of claim 15, wherein the output modality is selected from the group consisting of hardcopy print media and softcopy digital presentation.

25. The method of claim 15, further comprising first prioritizing the multimedia assets in each grouping, as a function of the recorded metadata, the user-provided metadata, and the derived metadata, by the processor.

26. The method of claim 15, wherein organizing the multimedia assets into the one or more groupings comprises using a frequent itemset mining algorithm configured to create thematic groupings of the multimedia assets, by the computer processor;

wherein the thematic groupings are defined by a pre-established feature category and a feature descriptor, the feature descriptor being a subordinate classification of the feature category; and

wherein the multimedia assets are first prioritized according to an asset priority score for the multimedia assets in the thematic groupings, the priority score being a function of the multimedia assets'"'"' fulfillment of the feature descriptor.