Method and system for object-oriented motion-based video description

• US 6,665,423 B1
• Filed: 01/27/2000
• Issued: 12/16/2003
• Est. Priority Date: 01/27/2000
• Status: Expired due to Fees

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Tekalp, A. Murat, Mehrotra, Rajiv
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): SUKHAPHADHANA, CHRISTOPHER T
• Application Number:
  US09/492,635
• Time in Patent Office: 1,419 Days
• Field of Search: 382/107, 382/236, 382/103, 348/135, 348/143, 348/154, 348/155
• US Class Current: 382/107
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/543 using regions

Abstract

An object-oriented method for describing the content of a video sequence comprises the steps of (a) establishing a temporal object-based segment for an object of interest; (b) describing the temporal object-based segment by describing one or more semantic motions of the object within its temporal object-based segment; and (c) describing the temporal object-based segment by describing one or more semantic interactions of the object with one or more other objects within its temporal object-based segment. The semantic motions of the object may be further described in terms of the properties of elementary coherent motions within the semantic motion. Additionally, the semantic interactions of the object may be further described in terms of the properties of the elementary spatio-temporal relationships among the interacting objects.

Claims

1. An object-oriented method for describing the motion content of a video sequence, said method comprising the steps of:

(a) identifying a primary object of interest and an interacting object of interest, and detecting one or more object-based segments corresponding to both objects;

(b) describing the object-based segment by one or more interaction units that provide a semantic description of meaningful interactions between the objects; and

(c) describing each interaction unit as a composition of interaction-based segments, wherein each interaction-based segment comprises an elementary reaction unit that describes the primary object'"'"'s interaction with the interacting object, wherein the description of the primary object'"'"'s interaction with the interacting object includes a parametric motion representation of the relative motion of the interacting objects including the dominant motion of the interacting object with respect to the centroid of the primary object.

2. The method as claimed in claim 1 wherein each description provided by step (c) includes a beginning and ending identifier for each elementary reaction unit.

3. The method as claimed in claim 1 wherein each description provided by step (c) includes identification of the type of interaction.

4. The method as claimed in claim 3 wherein the type of interaction is selected from the group including spatial coexistence, spatial contact and spatial occlusion.

5. The method as claimed in claim 1 wherein each description provided by step (c) includes description of the temporal overlapping of the interacting objects.

6. A method for automatically detecting elementary motion units in order to describe the content of a video sequence, said method comprising the steps of:

(a) identifying an object of interest and detecting an object-based segment corresponding to the object;

(b) estimating the dominant motion of the object in frames of the video sequence;

(c) computing a dissimilarity value representing dominant motion of the object between frames of the video sequence; and

(d) establishing elementary motion units corresponding to significant changes in the dissimilarity value.

7. The method as claimed in claim 6 wherein the step (b) of estimating the dominant motion of the object includes estimation of the affine motion parameters of the dominant motion.

8. The method as claimed in claim 7 wherein the step (c) of computing a dissimilarity value representing dominant motion of the object between frames includes computing the dissimilarity of dominant motion by comparing the affine models of the object in adjacent frames.

9. A method for automatically detecting elementary motion units in order to describe the content of a video sequence, said method comprising the steps of:

(a) identifying an object of interest and detecting an object-based segment corresponding to the object;

(b) providing a dense motion field in each frame representing motion of the object between frames within the object-based segment;

(c) estimating an affine parameter set from the dense motion field in order to represent the estimated dominant motions of the object;

(d) computing a dissimilarity value representing dominant motions of the object between adjacent frames;

(e) identifying local maxima in the dissimilarity value; and

establishing elementary motion units corresponding to significant changes in the local maxima.

10. A method for automatically detecting elementary reaction units in order to describe the content of a video sequence, said method comprising the steps of:

(a) identifying a primary object of interest and an interacting object of interest, and detecting object-based segments corresponding to both objects;

(b) establishing the spatio-temporal relationship between the primary object and the interacting object from any one or more of the group including (i) coexistence wherein the two objects appear together without any other type of spatio-temporal relationship between them, (ii) spatial contact wherein the boundaries of the two objects overlap and (iii) occlusion wherein one object occludes the other object; and

(c) establishing elementary reaction units descriptive of the spatio-temporal relationship established in step (b).

11. The method as claimed in claim 10 wherein establishing the spatio-temporal relationship in step (b) comprises:

(a) determining if there is overlap between both object-based segments in order to establish coexisting objects;

(b) determining if there is a common boundary between the two coexisting objects in order to establish spatially contacting objects; and

(c) determining if there is frame-to-frame change in the relative sizes of a pair of contacting objects in order to establish occluding objects.

12. A method for automatically detecting elementary reaction units in order to describe the content of a video sequence, said method comprising the steps of:

(a) identifying a primary object of interest and an interacting object of interest, and detecting object-based segments corresponding to both objects;

(b) identifying overlapping regions of the primary and interacting objects and partitioning the regions into elementary reaction units where the relative motion either remains unchanged between frames or coherent without abrupt and dramatic changes;

(c) computing a parametric motion model for the motion of the interacting object with reference to the primary object;

(d) computing the motion trajectory of the interacting object with reference to the primary object; and

(e) providing description of the elementary reaction units in terms of the parametric motion model and the motion trajectory established in steps (c) and (d).