Apparatus and method for context-based indexing and retrieval of image sequences

• US 6,643,387 B1
• Filed: 01/24/2000
• Issued: 11/04/2003
• Est. Priority Date: 01/28/1999
• Status: Expired due to Term

• Current Assignee: MediaTek, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Sodagar, Iraj, Sethuraman, Sriram, Chalom, Edmond
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): SUKHAPHADHANA, CHRISTOPHER T
• Application Number:
  US09/490,085
• Time in Patent Office: 1,380 Days
• Field of Search: 348/699, 382/173, 382/107, 382/103, 382/224, 382/236
• US Class Current: 382/107
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

An apparatus and method for implementing object motion segmentation and object trajectory segmentation for an image sequence. Specifically, block-based motion vectors for a pair of adjacent frames are used to derive optical flow, e.g., affine, motion parameters. Such optical flow motion parameters are employed to determine key objects where their motion and trajectory within a sequence of frames are calculated and stored. Such object motion information is used to improve or offer image processing functions such as context-based indexing of the input image sequence by using motion-based information.

Claims

1. A method for performing object motion segmentation for an image sequence having a plurality of frames, said method comprising the steps of:

a) determining at least one connected region within one of said frames of said image sequence;

b) applying block-based motion vectors corresponding to blocks within said at least one connected region to generate optical flow motion parameters; and

c) using said generated optical flow motion parameters for identifying a key object.

2. The method of claim 1, wherein said applying step b) applies block-based motion vectors of at least three blocks of said at least one connected region to generate said optical flow motion parameters.

3. The method of claim 1, further comprising the step of:

d) determining if an identified key object found in step c) is of sufficient size.

4. The method of claim 1, further comprising the step of:

d) determining if any two identified key objects found in step c) are to be merged into a single key object.

5. The method of claim 4, wherein said determining step d) comprises the step of:

d1) applying block-based motion vectors from at least three blocks from said any two identified key objects to generate a new set of optical flow motion parameters.

6. The method of claim 1, further comprising the steps of:

d) labeling an identified key object found in step c); and

e) tracking said labeled key object in a current frame by using motion vector information from said current frame to correlate a region in said current frame with a labeled key object of a previous frame.

7. The method of claim 1, further comprising the steps of:

d) indexing the image sequence in accordance with said optical flow motion parameters of any identified key objects from step c).

8. The method of claim 7, wherein said indexing step d) indexes the image sequence in accordance with said optical flow motion parameters of any identified key objects in conjunction with a spatial information associated with said any identified key objects.

9. The method of claim 1, further comprising the steps of:

d) splitting said identified key object into a plurality of sub-objects; and

e) applying block-based motion vectors from at least three blocks from each of said sub-objects to generate a new set of optical flow motion parameters for each of said sub-objects.

12. The method of claim 1, wherein said optical flow motion parameters are affine motion parameters.

10. A method for performing motion segmentation and trajectory segmentation for an image sequence having a plurality of frames, said method comprising the steps of:

a) applying optical flow object motion segmentation to obtain motion information for identifying at least one key object for a frame of the image sequence; and

b) applying optical flow object trajectory segmentation to obtain trajectory information for said at least one key object for a frame interval greater than two frames of the image sequence.

11. The method of claim 10, wherein said applying step a) comprises the steps of:

a1) determining at least one connected region within one of said frame of said image sequence;

a2) applying block-based motion vectors to generate optical flow motion parameters; and

a3) using said generated optical flow motion parameters for searching one or more key objects.

13. An apparatus for performing object motion segmentation for an image sequence having a plurality of frames, said apparatus comprising:

means for determining at least one connected region within one of said frame of said image sequence; and

an optical flow object segmenter for applying block-based motion vectors corresponding to blocks within said at least one connected region to generate optical flow motion parameters and for using said generated optical flow motion parameters for searching one or more key objects.

14. The apparatus of claim 13, wherein said optical flow object segmenter applies block-based motion vectors of at least three blocks of said at least one connected region to generate said optical flow motion parameters.

15. The apparatus of claim 13, wherein said optical flow object segmenter determines if an identified key object is of sufficient size.

16. The apparatus of claim 13, wherein said optical flow object segmenter determines if any two identified key objects are to be merged into a single key object by applying block-based motion vectors from at least three blocks from said any two identified key objects to generate said optical flow motion parameters.

17. An apparatus for performing motion segmentation and trajectory segmentation for an image sequence having a plurality of frames, said apparatus comprising:

an optical flow object motion segmenter for obtaining motion information for at least one key object for a frame of the image sequence; and

an optical flow object trajectory segmenter for obtaining trajectory information for said at least one key object for a frame interval greater than two frames of the image sequence.

18. The apparatus of claim 17, wherein said optical flow object motion segmenter comprises:

means for determining at least one connected region within one of said frame of said image sequence; and

a segmenter for applying block-based motion vectors to generate optical flow motion parameters and for using said generated optical flow motion parameters for searching one or more key objects.

19. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps comprising of:

a) determining at least one connected region within one of said frame of said image sequence;

b) applying block-based motion vectors corresponding to blocks within said at least one connected region to generate optical flow motion parameters; and

c) using said generated optical flow motion parameters for identifying a key object.

20. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps comprising of:

a) applying optical flow object motion segmentation to obtain motion information for identifying at least one key object for a frame of the image sequence; and

b) applying optical flow object trajectory segmentation to obtain trajectory information for said at least one key object for a frame interval greater than two frames of the image sequence.