Apparatus and method for generating object-labeled image in video sequence

• US 7,024,020 B2
• Filed: 09/04/2001
• Issued: 04/04/2006
• Est. Priority Date: 01/20/2001
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Chang-yeong, Lee, Seong-deok, Moon, Young-su, Kim, Sang-kyun, Kim, Ji-yeon, Park, Doo-sik
• Primary Examiner(s): Werner, Brian
• Assistant Examiner(s): LAVIN, CHRISTOPHER L
• Application Number:
  US09/944,600
• Publication Number:
  US 20020097893A1
• Time in Patent Office: 1,673 Days
• Field of Search: 345/967, 382/103, 382/155, 382/159, 382/164, 382/165, 382/170, 382/173, 382/180, 382/181, 382/203, 382/209, 382/215, 382/217, 382/218, 382/224, 382/236, 382/234, 707/104.1, 348169-172
• US Class Current: 382/103
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

An apparatus and method for generating object-labeled images based on query images in a video sequence are provided. A video sequence is divided into a plurality of shots, each of which consists of a set of frames having a similar scene, and an initial object region is extracted from each of the shots by determining whether an object image exists in key frames of the shots. Based on the initial object region extracted from each of the key frames, object regions are tracked in all frames of the shots. Then, the object regions are labeled to generate object-labeled images. Therefore, the object-labeled image generating apparatus and method can be applied regardless of the degree of motion of an object and time required to extract query objects is reduced.

Claims

1. A method for receiving a video sequence including query objects to be extracted and generating object-labeled images based on the query objects, the method comprising the steps of:

(a) dividing the video sequence into one or more shots, each of which is a set of frames having a similar scene, and selecting one or more key frames from each of the shots;

(b) determining whether there exists an object similar to each of the query objects in each of the key frames and extracting the similar objects as corresponding query object based initial object regions from each of the key frames;

(c) for each query object, tracking object regions in all frames of only shots determined to have a respective similar object in a key frame based on the corresponding query object based initial object regions; and

(d) labeling the object regions tracked in each of the frames based on information on the corresponding query objects.

2. The method of claim 1, wherein step (b) comprises:

generating query object based shot mask images in all key frames of the shots by setting pixels of the query object based initial object regions extracted from each of the key frames as a first value and setting the remaining pixels of each of the key frames as a second value.

3. The method of claim 2, wherein step (c) comprises:

(c1) tracking the object regions in all frames of each of the shots based on the corresponding query image based shot mask images and video feature values of the corresponding query objects; and

(c2) generating query object based frame mask images in all frames of each of the shots by setting pixels of the object regions tracked in each of the frames as a first value and setting the remaining pixels of each of the key frames as a second value.

4. The method of claim 3, wherein, in step (d), each of the object regions is labeled in each frame with a unique number set to the corresponding query image or coordinate information of the corresponding query image in each frame.

5. An apparatus for receiving a video sequence including query objects to be extracted and generating object-labeled images based on the query objects, the apparatus comprising:

a shot and key frame setting unit for dividing the video sequence into one or more shots, each of which is a set of frames having a similar scene, and selecting one or more key frames from each of the shots;

an initial object region extractor for determining whether there exists an object similar to each of the query objects in each of the key frames and extracting the similar objects as corresponding query object based initial object regions from each of the key frames;

an object region tracker for tracking, for each query object, object regions in all frames of only shots determined to have a respective similar object in a key frame based on the corresponding query image object based initial object regions; and

an object-labeled image generator for labeling the object regions tracked in each of the frames based on information on the corresponding query objects.

6. The apparatus of claim 5, wherein the initial object region extractor generates query object based shot mask images in all key frames of each of the shots by setting pixels of the query object based initial object regions extracted from each of the key frames as a first value and setting the remaining pixels of each of the key frames as a second value.

7. The apparatus of claim 6, wherein the object region tracker tracks the object regions in all frames of each of the shots based on the corresponding query image based shot mask images and video feature values of the corresponding query objects, and generates query object based frame mask images in all frames of each of the shots by setting pixels of the object regions tracked in each of the frames as a first value and setting the remaining pixels of each of the key frames as a second value.

8. The apparatus of claim 5, wherein the object-labeled image generator labels each of the object regions in each frame with a unique number set to the corresponding query image or coordinate information of the corresponding query image in each frame.

9. A computer readable medium having embodied thereon a computer program for receiving a video sequence including query objects to be extracted and generating object-labeled images based on the query objects, wherein generating object-labeled images comprises the steps of:

(a) dividing the video sequence into one or more shots, each of which is a set of frames having a similar scene, and selecting one or more key frames from each of the shots;

(b) determining whether there exists an object similar to each of the query objects in each of the key frames and extracting the similar objects as corresponding query object based initial object regions from each of the key frames;

(c) for each query object, tracking object regions in all frames of only shots determined to have a respective similar object in a key frame based on the corresponding query object based initial object regions; and

(d) labeling the object regions tracked in each of the frames based on information on the corresponding query objects.