Management of properties for hyperlinked video

• US 6,642,940 B1
• Filed: 03/03/2000
• Issued: 11/04/2003
• Est. Priority Date: 03/03/2000
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Dakss, Jonathan, Vasconcelos, Nuno, Chalom, Edmond, Bove, V. Michael Jr.
• Primary Examiner(s): Cabeca, John
• Assistant Examiner(s): Vu, Kieu D.
• Application Number:
  US09/518,588
• Time in Patent Office: 1,341 Days
• Field of Search: 345/716, 345/719, 345/723, 345/760, 345/721, 345/726, 345/764, 345/846, 345/856, 345/859, 345/854, 707/501.1, 382/203, 382/209, 382/190, 382/199, 382/180, 382/173, 715/500.1
• US Class Current: 715/723
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858

Abstract

The process of identifying and associating information with objects in a hyperlinked video sequence is automated by creating an accessible list of object information, including semantic representations, which updates with the identification of new objects. Because objects appear in more than one shot in many video sequences, the invention makes guesses about the identification of objects in a newly segmented sequence. If it guesses the object correctly, the author is relieved of the need to manually search a database of object information to make the association.

Claims

1. A method of classifying and assigning properties to unclassified objects in an unclassified image frame, the method comprising steps of:

a. locating unclassified objects in the unclassified image frame;

b. finding at least one basis image frame containing identified objects including at least some of the objects in the unclassified image frame, the identified objects having properties associated therewith, the properties including at least one action associated with selection of an object; and

c. classifying the unclassified objects by locating corresponding identified objects in the at least one basis image frame, and assigning the classified objects the properties associated with the identified objects corresponding thereto.

2. The method of claim 1 further comprising the steps of:

a. modeling at least some of the identified objects of the at least one basis image frame in terms of probability density functions with respect to at least one feature thereof; and

b. modeling the unclassified objects in terms of probability density functions with respect to at least one feature thereof, unclassified objects being identified by locating corresponding identified objects based on the probability density functions.

3. The method of claim 2 wherein each image frame comprises an array of pixels, the objects in a frame comprising regions within the pixel array, the pixels having at least one feature parameter associated therewith, each feature of an object corresponding to at least one feature parameter associated with the pixels representing the object.

4. The method of claim 2 wherein the at least one feature is selected from the group consisting of color, texture, motion, and position.

5. The method of claim 4 further comprising the step of organizing a database of the at least one feature of the identified objects, the at least one feature having a plurality of feature parameters associated therewith, the database being organized by:

a. defining, for each feature parameter, a series of data bins corresponding to selected values of the feature parameter; and

b. associating, with each data bin, objects having feature parameters with values that accord with the values corresponding to the data bin.

6. The method of claim 5 wherein the located objects also have a plurality of feature parameters associated with the at least one feature thereof, the identifying step comprising:

a. obtaining, for the located objects, values for the feature parameters;

b. identifying, in the database, data bins corresponding to the located-object feature parameters;

c. selecting identified objects associated with (i) the data bin corresponding to the located-object feature parameters and (ii) additional data bins having a selected proximity thereto; and

d. comparing the probability density functions of the located objects in the unclassified image frame with the selected objects.

7. The method of claim 6 wherein the at least one feature comprises color, the plurality of feature parameters associated with color comprising chrominance and luminance.

8. The method of claim 2 wherein:

a. the at least one basis image frame comprises a series of video frames organized into a sequence of shots and including at least one training frame in which objects have been manually identified; and

b. the probability density functions of the identified objects include, for each identified object, (i) a series of third-order probability density functions each derived from a training frame in which the object appeared, (ii) a series of second-order probability density functions each derived from the third-order probability density functions within a single shot, and (iii) a first-order probability density function derived from the second-order probability density functions.

9. The method of claim 8 wherein:

a. the unclassified image frame is one of a sequence of unclassified video frames constituting a shot, objects being located in the sequence of unclassified frames by manual identification in a plurality of selected training frames within the shot, followed by statistical comparison of the indicated objects with the frames in the shot that were not selected as training frames; and

b. the probability density functions of the located objects include, for each located object, (i) a series of third-order probability density functions each derived from one of the training frames in which the object appeared, and (ii) a series of second-order probability density functions each derived from the third-level probability density functions;

and further comprising the steps of;

c. comparing the second-order probability density functions of the located objects in the unclassified image frame with the first-order probability density functions of the identified objects from the basis video frames to identify the located objects.

10. The method of claim 1 further comprising the steps of:

a. organizing an occurrence database of the identified objects, the occurrence database tracking co-occurrence of objects within any single frame; and

b. if the step of identifying unclassified objects by locating corresponding identified objects based returns, for a given unclassified object, a plurality of potentially matching identified objects, consulting the occurrence database to identify any of the potentially matching identified objects that have co-occurred in a frame with the unclassified object to thereby identify the unclassified object.

11. The method of claim 1 wherein:

a. the located objects have a plurality of feature parameters associated therewith;

b. the identified objects in the at least one basis image frame have a plurality of feature parameters associated therewith;

and further comprising the steps of;

c. organizing a database of the feature parameters of the identified objects, the database being organized by;

i. defining, for each feature parameter, a series of data bins corresponding to selected values of the feature parameter; and

ii. associating, with each data bin, objects having feature parameters with values that accord with the values corresponding to the data bin;

d. obtaining, for the located objects, values for the feature parameters associated therewith; and

e. identifying the data bins corresponding to the located-object feature parameters; and

the identifying step comprising selecting, as identified objects potentially corresponding to the located objects, identified objects associated with (i) the data bin corresponding to the located-object feature parameters and (ii) additional data bins having a selected proximity thereto.

12. The method of claim 1 wherein at least some of the actions are hyperlinks.

13. Apparatus for classifying and assigning properties to unclassified objects in an unclassified image frame, the apparatus comprising:

a. an analysis module for locating unclassified objects in the unclassified frame; and

b. computer storage for storing at least one basis frame containing identified objects and including at least some of the objects in the unclassified image frame, the identified objects having properties associated therewith, the properties including at least one action associated with selection of an object;

the analysis module being configured to classify the unclassified objects by (i) locating corresponding identified objects in the stored basis frames and (ii) assigning to the classified objects the properties associated with the identified objects corresponding thereto.

14. The apparatus of claim 13 wherein the analysis module is further configured to:

a. model at least some of the identified objects of the at least one basis image frame in terms of probability density functions with respect to at least one feature thereof; and

b. model the unclassified objects in terms of probability density functions with respect to at least one feature thereof,

the analysis module identifying unclassified objects by locating corresponding identified objects based on the probability density functions.

15. The apparatus of claim 14 wherein each image frame comprises an array of pixels and further comprising a memory for storing the pixels, the objects in a frame comprising regions within the pixel array, the pixels having at least one feature parameter associated therewith, each feature of an object corresponding to at least one feature parameter associated with the pixels representing the object.

16. The apparatus of claim 15 wherein the at least one feature parameter is selected from the group consisting of color, texture, motion, and position.

17. The apparatus of claim 16 further comprising a database for storing the at least one feature of the identified objects, the at least one feature having a plurality of feature parameters associated therewith, the database comprising:

a. for each feature parameter, a series of data bins corresponding to selected values of the feature parameter; and

b. means for associating, with each data bin, objects having feature parameters with values that accord with the values corresponding to the data bin.

18. The apparatus of claim 17 wherein the located objects also have a plurality of feature parameters associated with the at least one feature thereof, the analysis module being configured to:

a. derive, for the located objects, values for the feature parameters;

b. identify, in the database, data bins corresponding to the located-object feature parameters;

c. select identified objects associated with (i) the data bin corresponding to the located-object feature parameters and (ii) additional data bins having a selected proximity thereto; and

d. compare the probability density functions of the located objects in the unclassified image frame with the selected objects.

19. The apparatus of claim 18 wherein the at least one feature comprises color, the plurality of feature parameters associated with color comprising chrominance and luminance.

20. The apparatus of claim 15 further comprising means for receiving a manual indication from a user, and further wherein:

a. the at least one basis frame comprises a series of video frames organized into a sequence of shots and including at least one training frame in which objects have been manually identified by operation of the manual-indication means; and

b. the probability density functions of the identified objects include, for each identified object, (i) a series of third-order probability density functions each derived from a training frame in which the object appeared, (ii) a series of second-order probability density functions each derived from the third-order probability density functions within a single shot, and (iii) a first-order probability density function derived from the second-order probability density functions.

21. The apparatus of claim 20 wherein:

a. the unclassified image frame is one of a sequence of video frames constituting a shot, objects being located in the sequence of unclassified video frames by operation of the manual-indication means in a plurality of selected training frames within the shot, followed by statistical comparison of the indicated objects with the frames in the shot that were not selected as training frames; and

b. the probability density functions of the located objects include, for each located object, (i) a series of third-order probability density functions each derived from one of the training frames in which the object appeared, and (ii) a series of second-order probability density functions each derived from the third-level probability density functions;

the analysis module being further configured to compare the second-order probability density functions of the located objects in the unclassified video frame with the first-order probability density functions of the identified objects from the basis video frames to identify the located objects.

22. The apparatus of claim 13 further comprising an occurrence database of the identified objects, the occurrence database tracking co-occurrence of objects within any single frame, the analysis module being configured to locate, in the occurrence database, potentially matching identified objects that have co-occurred in a frame with the unclassified object to thereby identify the unclassified object.

23. The apparatus of claim 13 wherein:

a. the located objects have a plurality of feature parameters associated therewith;

b. the identified objects in the basis video frames have a plurality of feature parameters associated therewith;

and further comprising a database of the feature parameters of the identified objects, the database defining, for each feature parameter, a series of data bins corresponding to selected values of the feature parameter and associating, with each data bin;

objects having feature parameters with values that accord with the values corresponding to the data bin, the analysis module being further configured to

c. obtain, for the located objects, values for the feature parameters associated therewith;

d. identify the data bins corresponding to the located-object feature parameters; and

e. select, as identified objects potentially corresponding to the located objects, identified objects associated with (i) the data bin corresponding to the located-object feature parameters and (ii) additional data bins having a selected proximity thereto.

24. The apparatus of claim 13 wherein at least some of the actions are hyperlinks.