Image retrieving method and apparatuses therefor

• US 20020012520A1
• Filed: 09/20/2001
• Published: 01/31/2002
• Est. Priority Date: 05/16/1997
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Miyatake, Takafumi, Nagasaka, Akio
• Granted Patent: US 6,587,637 B2
• Time in Patent Office: Days
• US Class Current: 386/46
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

By sequentially inputting images for each frame, sequentially extracting features from the inputted frame images, converting the features sequentially extracted into a feature series corresponding to the inputted frame image series, compressing the feature series in the direction of the time axis, storing the compressed feature series in the storage, sequentially extracting features separately from the images to be retrieved for each inputted frame, sequentially comparing the features of the images to be retrieved for each frame with the stored compressed feature series, storing the progress state of this comparison, updating the stored progress state of the comparison on the basis of a comparison result with the frame features of the succeeding images to be retrieved, and retrieving image scenes matching with the updated progress state from the images to be retrieved on the basis of the comparison result between the updated progress state and the features of the images to be retrieved for each frame, the present invention can retrieve video images on the air or video images in the data base at high speed and enables self organization of video to be classified and arranged on the basis of the identity of partial images of video.

Claims

1. A signal series retrieving method in an information processing system including time sequential signal input means, a time sequential signal process controller, and a storage comprising:

a step of sequentially inputting time sequential signals;

a step of sequentially extracting features in each predetermined period of said inputted time sequential signals;

a step of converting said features sequentially extracted into a feature series corresponding to said inputted predetermined period series;

a step of compressing said feature series in the direction of the time axis;

a step of storing said compressed feature series in said storage;

a step of sequentially extracting features from said time sequential signals to be retrieved in each predetermined period of said inputted time sequential signals;

a step of sequentially comparing said features of said time sequential signals to be retrieved in each predetermined period with said stored compressed feature series;

a step of storing a progress state of said comparison; and

a step of retrieving a signal series matching with said progress state from said time sequential signals to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said time sequential signals to be retrieved in each predetermined period.

2. An image retrieving method in an information processing system including image input means, an image process controller, and a storage comprising:

a step of sequentially inputting images for each frame;

a step of sequentially extracting features from said inputted frame images;

a step of converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

a step of compressing said feature series in the direction of the time axis;

a step of storing said compressed feature series in said storage;

a step of sequentially extracting features from said images to be retrieved for each said inputted frame;

a step of sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

a step of storing a progress state of said comparison; and

a step of retrieving image scenes matching with said progress state from said images to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said images to be retrieved for each frame.

4. An image retrieving method according to claim 2, wherein as said feature, a statistic of brightness or color is used.

6. An image retrieving method according to claim 2, wherein said compression of said feature series in the direction of the time axis is executed on the assumption that when the difference between the feature of a frame image and the feature of the next frame image is within a predetermined tolerance, the features are the same.

8. An image retrieving method according to claim 2, wherein it is assumed that with respect to a match with said progress state of said feature series, when features with more than a predetermined length match with each other, a comparison result match occurs.

11. An image retrieving method according to claim 2, further comprising:

when a match with said progress state of said comparison occurs in a plurality of locations, the frame image series in said plurality of locations is stored so as to be accessed as a related set.

13. An image retrieving method according to claim 2, further comprising:

when a match with said progress state of said comparison occurs in a plurality of locations, sequential programs on the air are classified on the basis of the frame image series in said plurality of locations.

15. An image retrieving method according to claim 2, further comprising:

when a match with said progress state of said comparison occurs in a plurality of locations, a specific image on the air is detected on the basis of the frame image series matched in said plurality of locations and the time length thereof.

3. An image retrieving method in an information processing system including image input means, an image process controller, and a storage comprising:

a step of sequentially inputting images for each frame;

a step of sequentially extracting features from said inputted frame images;

a step of converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

a step of compressing said feature series in the direction of the time axis;

a step of storing said compressed feature series in said storage;

a step of sequentially extracting features from said images to be retrieved for each said inputted frame;

a step of sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

a step of storing a progress state of said comparison;

a step of updating said stored progress state of said comparison on the basis of a comparison result with said frame features of said succeeding images to be retrieved; and

a step of retrieving image scenes matching with said updated progress state from said images to be retrieved on the basis of said comparison result between said updated progress state and said features of said images to be retrieved for each frame.

5. An image retrieving method according to claim 3, wherein as said feature, a statistic of brightness or color is used.

7. An image retrieving method according to claim 3, wherein said compression of said feature series in the direction of the time axis is executed on the assumption that when the difference between the feature of a frame image and the feature of the next frame image is within a predetermined tolerance, the features are the same.

9. An image retrieving method according to claim 3, wherein it is assumed that with respect to a match with said updated progress state of said feature series, when features with more than a predetermined length match with each other, a comparison result match occurs.

10. An image retrieving method according to claim 3, wherein with respect to storage and update of said progress state of said comparison, the number of the top frame in which a comparison match may occur is provisionally recorded, and when the comparison match continues, the frame number to be compared is updated, and when the possibility of comparison match is lost, said frame number to be compared is deleted.

12. An image retrieving method according to claim 3, further comprising:

when a match with said updated progress state occurs in a plurality of locations, the frame image series in said plurality of locations is stored so as to be accessed as a related set.

14. An image retrieving method according to claim 3, further comprising:

when a match with said updated progress state occurs in a plurality of locations, sequential programs on the air are classified on the basis of the frame image series in said plurality of locations.

16. An image retrieving method according to claim 3, further comprising:

when a match with said updated progress state occurs in a plurality of locations, a specific image on the air is detected on the basis of the frame image series matched in said plurality of locations and the time length thereof.

17. A signal series retrieving system in an information processor including time sequential signal input means, a time sequential signal process controller, and a storage comprising:

means for sequentially inputting time sequential signals;

means for sequentially extracting features in each predetermined period of said inputted time sequential signals;

means for converting said features sequentially extracted into a feature series corresponding to said inputted predetermined period series;

means for compressing said feature series in the direction of the time axis;

means for storing said compressed feature series in said storage;

means for sequentially extracting features from said time sequential signals to be retrieved in each predetermined period of said inputted time sequential signals;

means for sequentially comparing said features of said time sequential signals to be retrieved in each predetermined period with said stored compressed feature series;

means for storing a progress state of said comparison; and

means for retrieving a signal series matching with said progress state from said time sequential signals to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said time sequential signals to be retrieved in each predetermined period.

18. A signal series retrieving system in an information processor including image input means, an image process controller, and a storage comprising:

means for sequentially inputting images for each frame;

means for sequentially extracting features from said inputted frame images;

means for converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

means for compressing said feature series in the direction of the time axis;

means for storing said compressed feature series in said storage;

means for sequentially extracting features from said images to be retrieved for each said inputted frame;

means for sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

means for storing a progress state of said comparison; and

means for retrieving image scenes matching with said progress state from said images to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said images to be retrieved for each frame.

19. A signal series retrieving system in an information processor including image input means, an image process controller, and a storage comprising:

means for sequentially inputting images for each frame;

means for sequentially extracting features from said inputted frame images;

means for converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

means for compressing said feature series in the direction of the time axis;

means for storing said compressed feature series in said storage;

means for sequentially extracting features from said images to be retrieved for each said inputted frame;

means for sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

means for storing a progress state of said comparison;

means for updating said stored progress state of said comparison on the basis of a comparison result with said frame features of said succeeding images to be retrieved; and

means for retrieving image scenes matching with said updated progress state from said images to be retrieved on the basis of said comparison result between said updated progress state and said features of said images to be retrieved for each frame.

20. A program storage enabling execution of a process by an information processor including time sequential signal input means, a time sequential signal process controller, and a storage comprising:

a storage medium storing a program including the following processes which can be read by said information processor;

a process of sequentially inputting time sequential signals;

a process of sequentially extracting features in each predetermined period of said inputted time sequential signals;

a process of converting said features sequentially extracted into a feature series corresponding to said inputted predetermined period series;

a process of compressing said feature series in the direction of the time axis;

a process of storing said compressed feature series in said storage;

a process of sequentially extracting features from said time sequential signals to be retrieved in each predetermined period of said inputted time sequential signals;

a process of sequentially comparing said features of said time sequential signals to be retrieved in each predetermined period with said stored compressed feature series;

a process of storing a progress state of said comparison; and

a process of retrieving a signal series matching with said progress state from said time sequential signals to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said time sequential signals to be retrieved in each predetermined period.

21. A program storage enabling execution of a process by an information processor including image input means, an image process controller, and a storage comprising:

a storage medium storing a program including the following processes which can be read by said information processor;

a process of sequentially inputting images for each frame;

a process of sequentially extracting features from said inputted frame images;

a process of converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

a process of compressing said feature series in the direction of the time axis;

a process of storing said compressed feature series in said storage;

a process of sequentially extracting features from said images to be retrieved for each said inputted frame;

a process of sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

a process of storing a progress state of said comparison; and

a process of retrieving image scenes matching with said progress state from said images to be retrieved on the basis of said comparison result between said stored progress state of said comparison and said features of said images to be retrieved for each frame.

22. A program storage enabling execution of a process by an information processor including image input means, an image process controller, and a storage comprising:

a storage medium storing a program including the following processes which can be read by said information processor;

a process of sequentially inputting images for each frame;

a process of sequentially extracting features from said inputted frame images;

a process of converting said features sequentially extracted into a feature series corresponding to said inputted frame image series;

a process of compressing said feature series in the direction of the time axis;

a process of storing said compressed feature series in said storage;

a process of sequentially extracting features from said images to be retrieved for each said inputted frame;

a process of sequentially comparing said features of said images to be retrieved for each frame with said stored compressed feature series;

a process of storing a progress state of said comparison;

a process of updating said stored progress state of said comparison on the basis of a comparison result with said frame features of said succeeding images to be retrieved; and

a process of retrieving image scenes matching with said updated progress state from said images to be retrieved on the basis of said comparison result between said updated progress state and said features of said images to be retrieved for each frame.

23. An information processor, comprising:

a display;

a memory having a process program and a data retaining area;

control means for performing an image input process, an image retrieval process, and a display process of a retrieved image on said display according to said process program;

means for storing a moving picture image inputted by said image input process in said memory in frame-wise;

means for retrieving a moving picture image segment 2 which is regarded as the same as a moving picture image segment 1 with a predetermined length from said memory by the frame series of said stored moving picture image when a frame is newly inputted by said image retrieval process and associating said moving picture image segments 1 and 2; and

means for displaying said associated moving picture image segments 1 and 2 in distinction from other moving picture image segments when a moving picture image inputted by said display process of a retrieved image on said display is simply displayed on said display.

24. A video camera, comprising:

a camera for inputting an image;

an input processor of said image;

a storage for storing an image inputted from said camera;

a output processor for reproducing and outputting an image stored in said storage;

a feature extractor for extracting the feature of an inputted image for each frame;

a memory area for tabling and retaining said extracted feature;

a process of comparing the feature of an inputted image with the feature on the table; and

a process of associating frames having the feature agreeing with a predetermined comparison condition as similar images.