Method and device for describing video contents

• US 7,199,798 B1
• Filed: 01/20/2000
• Issued: 04/03/2007
• Est. Priority Date: 01/26/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Maeda, Junji, Echigo, Tomio, Tomita, Alberto, Kurokawa, Masato
• Primary Examiner(s): YANG, RYAN R
• Application Number:
  US09/890,168
• Time in Patent Office: 2,630 Days
• Field of Search: 345/632, 345/81, 345/719, 345721-723, 345/473, 345/474, 463/43, 725/115, 725/143, 348/169, 348/157, 348/586, 348/598, 348/600, 382/103, 375/240.1, 375/240.8, 375/240.16, 715/723, 715/726, 352/39
• US Class Current: 345/473
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

An object of the present invention is to provide a description method for efficiently representing contents of motion picture with a small data volume. The organization of the present invention (1) represents a trajectory of how each object has moved over time by using reference plane representing position information of each object, (2) sets a description unit based on a type of action of each object by using changes in shape of each object, (3) has actions of each object represented as each behavioral section, and (4) comprises a description facility capable of reading and interpreting definition of an object dependent on video contents, definition of classes of actions, and definition of interpretation of a scene by interaction of plural objects.

Claims

1. A description means for contents of a motion picture, said means comprising:

(a) means for setting a reference plane by using a zone description and a camera model;

(b) means for describing each of a plurality of objects on said motion picture by position on said reference plane and a predefined type of actions, said describing means providing a description based on changes in the shape of an object wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural re-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns; and

(c) means for describing each of a plurality of scenes in said motion picture by using said means for describing each of a plurality of objects in respective ones of said scenes.

9. The description means of claim 1 wherein the reference plane comprises information on object positions independently of a camera movement.

10. The description means of claim 1 wherein the reference plane represents a ground for determining positions of objects in relation to an actual direction of motion of the object.

2. A search means for contents of a motion picture, said means comprising of:

(a) means for setting a reference plane by using a zone description and a camera model;

(b) means for describing each of a plurality of objects on said motion picture by position on said reference plane and a predefined type of actions, said describing means providing a description based on changes in the shape of an object wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns;

(c) means for describing each of a plurality of scenes in said motion picture by using said means for describing each of a plurality of objects in respective ones of said scenes; and

(d) means for searching the motion picture by using said means for describing each object or said means for describing each scene.

11. The search means of claim 2 wherein the reference plane comprises information on object positions independently of a camera movement.

12. The search means of claim 2 wherein the reference plane represents a ground for determining positions of objects in relation to an actual direction of motion of the object.

3. A description method for a motion picture, said method comprising the steps of:

(a) determining a reference plane by using a zone description and a camera model, wherein the reference plane represents information of objects and their positions included in said motion picture;

(b) representing changes over time of each of said object positions on said reference plane as a trajectory;

(c) setting a description unit based on a predefined type of actions of each of said objects by using changes in shape of each object so as to assign actions of each object as respective behavioral sections, wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns; and

(d) defining each of a plurality of scenes in said motion picture by a plurality of said objects.

13. The description method of claim 3 wherein the reference plane further represents motion of objects independently of camera motion.

14. The description method of claim 3 wherein the reference plane further represents a ground for determining position and movement of objects in relation to an actual direction of motion of each of respective ones of the objects that is independent of a camera motion.

4. A search method for a motion picture, said method comprising the steps of:

(a) setting a reference plane by using a zone description and a camera model, wherein the reference plane represents information of objects and their positions included in said motion picture;

(b) representing changes over time of each of said object positions on said reference plane as a trajectory;

(c) setting a description unit based on a predefined type of actions of each of said objects by using changes in shape of each of said objects so as to assign actions of each object as respective behavioral sections, wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according actions of the object there are consecutive silhouettes of plural re-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns;

(d) defining each of a plurality of scenes in said motion picture by a plurality of said objects; and

(e) searching for a specific one of said scenes by using said actions of each object for said one scene.

15. The search method of claim 4 wherein setting the reference plane further comprises setting a ground for determining position and movement of objects in relation to an actual direction of motion of an individual one of the objects that is independent of a camera motion.

5. A description method for a motion picture, said method comprising the steps of:

(a) determining a reference plane from said motion picture by using a zone description and a camera model;

(b) cutting a region map for a region presented in said motion picture, identifying an object to obtain trajectory ID for an individual one of a plurality of objects in said motion picture, classifying action to obtain an action ID, and obtaining a camera parameter from said motion picture;

(c) creating a description of actions by each of said objects from said region map, said object trajectory ID, said action ID and said camera parameter, said description creating step also providing a description based on changes in the shape of an object wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns; and

(d) creating a description of scenes in said motion picture by using said description of actions by each of said objects.

16. The description method of claim 5 wherein determining the reference plane further comprises setting a ground for determining position and movement of objects in relation to an actual direction of motion of an individual one of the objects that is independent of a camera motion.

6. A search method for a motion picture, said method comprising the steps of:

(a) determining a reference plane from said motion picture by using a zone description and a camera model;

(b) cutting a region map for a region presented in said motion picture, identifying an object to obtain trajectory ID for an individual one of a plurality of objects in said motion picture, classifying action to obtain an action ID and obtaining a camera parameter from said motion picture;

(c) creating a description of actions by each of said objects from said region map, said object trajectory ID, said action ID and said camera parameter, said description creating step also providing a description based on changes in the shape of an object wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns;

(d) creating a description of scenes in said motion picture by using said description of actions by each of said objects; and

(e) searching for a specific one of said scenes by using said description of actions by each object or said description of scenes.

17. The search method of claim 6 wherein determining the reference plane further comprises setting a ground for determining position and movement of respective ones of said objects in relation to an actual direction of motion of an individual one of the objects that is independent of a camera motion.

7. A computer readable storage medium which has recorded a program, the program directing a computer to execute the steps of:

recording management data for searching a motion picture;

recording data of a description of actions by each of a plurality of objects defined by data of their respective positions on a reference plane and predefined types of actions, the reference plane being set by using a zone description and a camera model, wherein recorded data includes a description based on changes in the shape of an object, and the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns; and

recording data of a description of scenes defined by said data of description of actions.

8. A computer readable storage medium which has recorded a program, said program directing a computer to execute the steps of:

(a) determining a reference plane set by using a zone description and a camera model, the reference plane representing information of objects and their positions included in a motion picture;

(b) representing changes over time of each of each of said object positions on said reference plane as a trajectory;

(c) setting a description unit based on predefined types of actions of each of said objects by using changes in shape of each object so as to assign actions of each object as a respective behavioral section, wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns; and

(d) defining each of a plurality of scenes in said motion picture by plural ones of said objects.

18. The computer readable storage medium of claim 8 wherein the reference plane further represents a ground for determining actual object position and movement included in the motion picture that is independent of a camera motion tracking the object position and movement.

19. A computer readable storage medium which has recorded a program, said program directing a computer to execute the steps of:

(a) setting a reference plane by using a zone description and a camera model, the reference plane representing information of object positions included in a motion picture;

(b) representing changes over time of each object position on said reference plane as a trajectory;

(c) setting a description unit based on predefined types of actions of each object by using changes in shape of each object so as to assign actions of each object as a respective behavioral section, wherein the shape of the object is determined by the silhouette of the object, changes in the silhouette show specific changes according to actions of the object, there are consecutive silhouettes of plural pre-instructed action patterns, and input movement patterns are identified by comparison with the pre-instructed patterns;

(d) defining each of a plurality of scenes of said motion picture by plural ones of said objects; and

(e) searching a specific one of said scenes by using said actions of each object for said one scene.