Method for automatic extraction of semantically significant events from video

• US 6,721,454 B1
• Filed: 07/01/1999
• Issued: 04/13/2004
• Est. Priority Date: 10/09/1998
• Status: Expired due to Term

• Current Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Qian, Richard, Haering, Niels C.
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US09/346,550
• Time in Patent Office: 1,748 Days
• Field of Search: 382/103, 382/107, 382/168, 382/170-171, 382/173, 382/181, 382/224, 382/226-227, 382/232-233, 382/236, 382/240, 382/243, 382/305, 348/169, 348/135, 348/142, 348/143, 348/152, 375/240.08, 707/101, 707/102, 707/104.1
• US Class Current: 382/224
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Semantically meaningful events are detected in video with a multi-level technique. Video sequences are visually analyzed at the first level of the technique to detect shot boundaries, to measure color and texture of the content, and to detect objects in the content. At the second level of the technique, objects are classified and the content in each shot is summarized. At the third level of the technique, an event inference module infers events on the basis of temporal and spatial phenomena disclosed in the shot summaries. The technique may be extended to additional domains by incorporating additional domain related processes at the upper levels of the technique which utilize data produced at the first level of the technique by processes which are domain independent.

Claims

1. A computerized method of inferring an event portrayed in a video sequence having semantic content that belongs to at least one known domain, said video sequence having one or more intervals each containing a plurality of sequential frames, said method comprising the steps of:

(a) a computer analyzing the content of said video over at least one interval of said one or more intervals;

(b) a computer summarizing said analysis over said at least one interval; and

(c) a computer inferring from said summary whether or not said event is portrayed in said video sequence based on criteria associated with said semantic content.

2. The method of claim 1 wherein said step of analyzing said content of said video sequence comprises the steps of:

(a) segmenting said video sequence into said one or more intervals where each of said intervals contains relatively homogeneous content among said plurality of sequential frames within said interval;

(b) detecting an object in said content; and

(c) measuring at least one of a color and a texture of said content.

3. The method of claim 2 wherein a color histogram evaluates the content of each of said plurality of sequential frames within said one or more intervals.

4. The method of claim 2 wherein said object is detected by comparing said content of a first frame with said content of a second frame.

5. The method of claim 3 further comprising the step of adjusting the content of said plurality of sequential frames within at least one interval of said one or more intervals for a global motion of said content within said at least one interval.

6. The method of claim 1 wherein said step of summarizing said analysis comprises characterizing said content with at least one of a spatial descriptor, a temporal descriptor and an object descriptor.

7. The method of claim 1 wherein said event is inferred from at least one of a spatial descriptor, a temporal descriptor and an object descriptor in said summary.

8. A computerized method of inferring an event portrayed in a video sequence having semantic content that belongs to at least one known domain, said method comprising the steps of:

(a) a computer decomposing said video into at least one plurality of frames having relatively homogenous content;

(b) a computer detecting an object in said content;

(c) a computer classifying said object;

(d) a computer characterizing said content of said at least one plurality of frames by at least one of a spatial descriptor, a temporal descriptor, and an object descriptor; and

(e) a computer inferring from said characterization of said content whether or not said event is portrayed in said video sequence based on criteria associated with said semantic content.

9. The method of claim 8 wherein said object is detected by comparing said content of a first frame with said content of a second frame of said at least one plurality of frames.

10. The method of claim 9 wherein comparing said content includes a step of adjusting said content for a global motion of said content between said first frame and said second frame.

11. The method of claim 8 wherein at least one of a texture and a color measure of said content is used to classify said object.

12. The method of claim 8 wherein said at least one plurality of frames are is detected by comparing a plurality of histograms of said content.

13. A computerized method of inferring an event portrayed in a video sequence having semantic content that belongs to at least one known domain, said video sequence having one or more intervals each containing a plurality of sequential frames, said method comprising the steps of:

(a) a computer decomposing said video into said one or more intervals where each of said intervals has relatively homogenous content;

(b) a computer detecting an object moving independent of a global motion of said content;

(c) a computer measuring the position of said object in an initial frame and a subsequent frame of said one or more intervals;

(d) a computer measuring the size of said object;

(e) a computer measuring at least one of a color and a texture of said content;

(f) a computer classifying said object from at least one of said at least one of a color and texture measure, said position measure, and said size measure;

(g) a computer summarizing said content over said one or more intervals by characterizing at least one of said classification of said object in said initial frame and said subsequent frame; and

(h) a computer inferring from said summary whether or not said event is portrayed in said video sequence based on criteria associated with said semantic content.

14. The method of claim 13 wherein said object is detected by comparing said content of a first frame with said content of a second frame, said content being adjusted for said global motion of said content.

15. The method of claim 13 wherein a boundary of said shot is detected by comparing a first histogram representing said content of a first frame of said video with a second histogram representing said content of a second frame.