Method and system for segmenting and identifying events in images using spoken annotations

• US 20030059112A1
• Filed: 06/01/2001
• Published: 03/27/2003
• Est. Priority Date: 06/01/2001
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Eastman Kodak Company
• Inventors: Loui, Alexander C., Stent, Amanda J.
• Granted Patent: US 6,810,146 B2
• Time in Patent Office: Days
• US Class Current: 382/176
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A method for automatically organizing digitized photographic images into events based on spoken annotations comprises the steps of: providing natural-language text based on spoken annotations corresponding to at least some of the photographic images; extracting predetermined information from the natural-language text that characterizes the annotations of the images; segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and identifying each event by assembling the categories of information into event descriptions. The invention further comprises the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner, such as in a photographic album.

Claims

1. A method for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said method comprising the steps of:

providing natural-language text based on spoken annotations corresponding to a plurality of frames of photographic images;

extracting predetermined information from the natural-language text that characterizes the annotations of the images;

segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and

identifying each event by assembling the categories of information into event descriptions.

2. The method as claimed in claim 1 further comprising the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner.

3. The method as claimed in claim 2 wherein the event descriptions are selected and arranged in a suitable manner as a photographic album.

4. The method as claimed in claim 1 wherein the step of providing natural-language text comprises the steps of:

acquiring spoken annotations; and

converting the spoken annotations to natural-language text.

5. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of:

segmenting the natural-language text into words and sentences; and

applying a plurality of dictionaries or gazetteers to the words and sentences to classify important words signifying possible events.

6. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of:

segmenting the natural-language text into words and sentences;

identifying elements of numerical expression that may help to define events; and

identifying expressions signifying at least one of date, time, money and percentage that may further define events.

7. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of:

segmenting the natural-language text into words and sentences; and

identifying references to people, location, events and objects of interest in relation to possible events.

8. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of:

segmenting the natural-language text into words and sentences; and

identifying noun and verb phrases that may relate to possible events.

9. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the step of extracting the natural-language text according to an XML specification.

10. The method as claimed in claim 1 wherein the step of segmenting the images into events comprises the steps of:

assigning a strength value for the certain categories of information which are indicative of a boundary between events;

computing the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to a current event; and

allocating the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

11. The method as claimed in claim 10 wherein the steps of computing the evidence and allocating the frame are taken with regard to an adjacent frame of the current frame.

12. The method as claimed in claim 10 wherein the steps of computing the evidence and allocating the frame are taken with regard to a non-adjacent frame of the current frame, and wherein the allocation of the intervening frames are made on the basis of the current frame.

13. The method as claimed in claim 12 wherein the steps of computing the evidence and allocating the frame are taken with regard to a frame that is separated by one frame from the current frame.

14. In a method wherein image frames are segmented into events by examining annotation data for the presence of certain categories of information which are indicative of a boundary between events, the improvement wherein the method of segmentation comprises the steps of:

assigning a strength value for the certain categories of information which are indicative of a boundary between events;

computing the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to a current event; and

allocating the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

15. The method as claimed in claim 14 wherein the steps of computing the evidence and allocating the frame are taken with regard to an adjacent frame of the current frame.

16. The method as claimed in claim 14 wherein the steps of computing the evidence and allocating the frame are taken with regard to a non-adjacent frame of the current frame, and wherein the allocation of the intervening frames are made on the basis of the current frame.

17. The method as claimed in claim 16 wherein the steps of computing the evidence and allocating the frame are taken with regard to a frame that is separated by one frame from the current frame.

18. In a method for automatically organizing digitized consumer images into events based on natural-language text corresponding to spoken annotations, wherein predetermined information characterizing the annotations of the images is automatically extracted from the natural-language text by use of a gazetteer, the improvement wherein the gazetteer comprises a collection of indices including commonly-used proper names, place names including typical tourist spots and celebration places, currency names, function or stop words, irregular verb forms, regular verbs, college and university names and typical events.

19. A computer program product for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

providing natural-language text based on spoken annotations corresponding to at least some of the photographic images;

extracting predetermined information from the natural-language text that characterizes the annotations of the images;

segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and

identifying each event by assembling the categories of information into event descriptions.

20. The computer program product as claimed in claim 19 further comprising the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner.

21. A system for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said system comprising:

an input for receiving natural-language text based on spoken annotations corresponding to a plurality of frames of photographic images;

an event extraction stage for extracting predetermined information from the natural-language text that characterizes the annotations of the images;

an event segmentation stage for segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and

an event identification stage for identifying each event by assembling the categories of information into event descriptions.

22. The system as claimed in claim 21 further comprising an event summarization stage for summarizing each event by selecting and arranging the event descriptions in a suitable manner.

23. The system as claimed in claim 21 wherein the event extraction stage comprises a natural language processor.

24. The system as claimed in claim 23 wherein the natural language processor comprises a plurality of finite state machines.

25. The system as claimed in claim 21 wherein the event segmentation stage comprises a processor that:

assigns a strength value for the certain categories of information which are indicative of a boundary between events;

computes the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to a current event; and

allocates the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

26. The system as claimed in claim 25 wherein the processor computes the evidence and allocates the frame with regard to an adjacent frame of the current frame.

27. The system as claimed in claim 26 wherein the processor computes the evidence and allocates the frame with regard to a non-adjacent frame of the current frame, and wherein the allocation of the intervening frames are made on the basis of the current frame.