Video query system and method

• US 6,182,069 B1
• Filed: 01/02/1998
• Issued: 01/30/2001
• Est. Priority Date: 11/09/1992
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Petkovic, Dragutin, Niblack, Carlton Wayne
• Primary Examiner(s): HO, RUAY L
• Application Number:
  US09/002,340
• Time in Patent Office: 1,124 Days
• Field of Search: 707/1-10, 707/100-104, 707/200-206, 707/501, 707/516, 701/200, 706/45, 706/46, 706/50, 706/53, 706/55, 706/59, 706/61, 364/474.24, 364/474.07, 379/202, 386/96, 386/97
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Search of video images in a database is enhanced by specification of at least one object and a background in stored frames and in queries of the database. Video information is stored in the database by means of representative frames that include at least one object and a background. A query of the database is executed by determining whether a representative frame in the database having at least one object and a background is similar to a search image having at least one object and a background.

Claims

1. A method for retrieving video sequences from a database that contains representative frames, each representative frame representing video information, the method comprising:

receiving a search image and segmenting the search image into segments that include at least one object and a background;

executing a query of the database by determining whether a representative frame in the database that has at least one object and a background that is similar to the search image; and

returning a set of representative frames ranked in an order according to a degree of similarity to the search image.

2. The method of claim 1, wherein the search image specifies values for a plurality of image characteristics for the at least one object and the background.

3. The method of claim 2, wherein at least one image characteristic is motion.

4. The method of claim 2, wherein the plurality of image characteristics includes color, texture, and motion.

5. The method of claim 1, further including:

selecting a representative frame of the set of representative frames;

retrieving the video sequence that is represented by the representative frame from the database; and

displaying the video sequence.

6. The method of claim 5, wherein the representative frame is a key frame.

7. The method of claim 6, wherein the representative frame is a synthesized frame.

8. A method of storing video information in a database, comprising:

receiving a video sequence comprising a plurality of frames;

providing a representative frame representing the video sequence;

segmenting the representative frame into a plurality of segments that include at least one object and a background;

constructing a frame data representation of the representative frame that includes respective data representations of the at least one object and the background; and

storing the frame data representation in the database.

9. The method of claim 8, further including:

associating the frame data representation with the representative frame; and

storing the representative frame in the database.

10. The method of claim 9, further including:

associating the representative frame with the video sequence; and

storing the video sequence in the database.

11. The method of claim 8, wherein constructing includes computing values of the image characteristics of the object and the background, the respective data representations including the values.

12. The method of claim 11, wherein the image characteristics include motion.

13. The method of claim 11, wherein the image characteristics include color, texture, and motion.

14. The method of claim 8, wherein the representative frame is a key frame.

15. The method of claim 8, wherein the representative frame is a synthesized frame.