Image retrieval system and image retrieval method

• US 6,665,442 B2
• Filed: 02/02/2001
• Issued: 12/16/2003
• Est. Priority Date: 09/27/1999
• Status: Expired

• Current Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Yamada, Yoshihisa, Nishikawa, Hirofumi, Asai, Kohtaro, Sekiguchi, Shunichi, Isu, Yoshimi
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Bayat, Ali
• Application Number:
  US09/773,570
• Publication Number:
  US 20010004739A1
• Time in Patent Office: 1,047 Days
• Field of Search: 382/232, 382/224, 382/217, 382/305, 382/173, 382/236, 382/190, 382/225, 382/226, 382/227, 382/228, 382/229, 382/230, 382/231, 382/192, 382/194, 382/195, 382/196, 707/104.1, 707/2, 707/3, 707/6, 707/100, 707/102, 348/231.99, 348/700, 348/231.2, 348/231.3, 348/231.4, 348/231.5, 348/231.6, 375/240, 345/719, 345/720, 345/721, 345/722, 345/723, 345/724, 345/725, 345/726
• US Class Current: 382/224
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 9/8042 involving data reduction
• CPC Class Codes: H04N 9/8047 using transform coding
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

When a retrieval condition of an attribute list is input from a user interface unit to a retrieval processing unit, the attribute list stored in an attribute list storing unit is retrieved in the retrieval processing unit. Thereafter, attribute information conforming to the retrieval condition is output to and displayed on a displaying unit. Thereafter, when a retrieval condition of the similarity retrieval is input from the user interface unit to the retrieval processing unit, image data stored in the image information storing unit is retrieved in the retrieval processing unit, and specific image data relating to a characteristic descriptor set conforming to the retrieval condition is selected in the retrieval processing unit. Thereafter, the specific image data is output to and displayed on the displaying unit.

Claims

1. An image retrieval system, comprising:

a characteristic descriptor producing unit for extracting a plurality of image characteristic values from pieces of input image data and producing a characteristic descriptor for each piece of input image data;

an image information storing unit for storing the characteristic descriptors produced in the characteristic descriptor producing unit while holding the correspondence of each characteristic descriptor to one piece of input image data;

an attribute list producing unit for producing an attribute list according to a piece of attribute information attached to each piece of input image data; and

an image retrieving unit for receiving a first retrieval condition relating to attribute information, searching the attribute list produced in the attribute list producing unit for one piece of attribute information conforming to the first retrieval condition, outputting the piece of attribute information conforming to the first retrieval condition, receiving a second retrieval condition relating to a characteristic descriptor, searching the image information storing unit for one piece of image data conforming to the second retrieval condition, and outputting the piece of image data conforming to the second retrieval condition.

2. An image retrieval system according to claim 1, wherein the attribute list is produced according to a syntax, which defines a data structure of the attribute list, in the attribute list producing unit, the attribute list is analyzed according to the syntax of the attribute list in the image retrieving unit, and the piece of attribute information conforming to the first retrieval condition is retrieved in the image retrieving unit.

3. An image retrieval system according to claim 1, wherein the characteristic descriptors are produced according to a syntax, which defines a data structure of each characteristic descriptor, in the characteristic descriptor producing unit, the characteristic descriptors are analyzed according to the syntax of the characteristic descriptors in the image retrieving unit, and the piece of image data conforming to the second retrieval condition is retrieved in the image retrieving unit.

4. An image retrieval system according to claim 1, wherein one image characteristic value is extracted in the characteristic descriptor producing unit for each frame, and one characteristic descriptor is produced in the characteristic descriptor producing unit for each video segment composed of a group of frames.

5. An image retrieval system according to claim 4, wherein each piece of input picture data received in the characteristic descriptor producing unit denotes compressed video data which composes one or more intra-frames and one or more inter-frames,

both an average value and a standard deviation are produced as one characteristic descriptor of the intra-frames of the video segment in the characteristic descriptor producing unit by extracting an average matrix of pixel values in a prescribed coding area of one intra-frame for each intra-frame of the video segment, calculating a sum of the average matrices of all intra-frames included in the video segment and calculating both the average value of the average matrices and the standard deviation of the average matrices from the sum and the number of intra-frames in the video segment, and

one characteristic descriptor of the inter-frames of the video segment is produced in the characteristic descriptor producing unit by extracting a matrix of motion vectors of pixels of the prescribed coding areas of one inter-frame for each inter-frame of the video segment, calculating an average of the motion vectors of each inter-frame of the video segment as a motion vector average, classifying zero run lengths, which are obtained according to a threshold processing for the motion vectors of one inter-frame, into a plurality of classified types of zero run lengths for each inter-frame of the video segment, calculating an average of the motion vector averages and a plurality of classified types of average zero run lengths in the video segment according to the number of inter-frames in the video segment and setting the average of the motion vector averages and the classified types of average zero run lengths as the characteristic descriptor of the inter-frames of the video segment.

6. An image retrieval system according to claim 1, wherein each piece of input picture data received in the characteristic descriptor producing unit denotes non-compressed video data, the characteristic values are extracted from the pieces of non-compressed video data in the characteristic descriptor producing unit to produce the characteristic descriptors, and the characteristic descriptors and pieces of compressed video data, which are obtained by compressing the pieces of non-compressed video data according to a prescribed video compression method, are stored in the image information storing unit while the correspondence of each characteristic descriptor to one piece of compressed video data is held.

7. An image retrieval system according to claim 1, wherein a prescribed condition equation stored in advance is read out by the image retrieving unit in cases where the image retrieving unit searches the image information storing unit for the piece of image data conforming to the characteristic descriptor specified by the second retrieval condition to output the piece of image data, and it is judged according to the prescribed condition equation whether or not the piece of image data conforms to the characteristic descriptor specified by the second retrieval condition.

8. An image retrieval system according to claim 1, wherein each piece of input image data denotes a monitored image recorded by a monitoring camera.

9. An image retrieval system according to claim 1, wherein each piece of input image data denotes an image of a stored video mail.

10. An image retrieval system according to claim 1, wherein each piece of input image data denotes an image of a stored broadcast program.

11. An image retrieval system according to claim 1, wherein each piece of input image data denotes a video image recorded by a video camera.

12. An image retrieval method, comprising the steps of:

extracting a plurality of image characteristic values from pieces of input image data to produce a characteristic descriptor for each piece of input image data;

storing the produced characteristic descriptors while holding the correspondence of each characteristic descriptor to one piece of input image data;

producing an attribute list according to a piece of attribute information attached to each piece of input image data;

receiving a first retrieval condition relating to attribute information;

searching the attribute list for one piece of attribute information conforming to the first retrieval condition;

outputting the piece of attribute information conforming to the first retrieval condition;

receiving a second retrieval condition relating to a characteristic descriptor;

searching the pieces of stored image data for one piece of image data conforming to the second retrieval condition while referring to the piece of attribute information conforming to the first retrieval condition; and

outputting the piece of image data conforming to the second retrieval condition.

13. An image retrieval system, comprising:

a characteristic descriptor producing unit for extracting a plurality of image characteristic values from pieces of input image data and producing a characteristic descriptor for each piece of input image data;

a for-retrieval description data producing unit for describing each characteristic descriptor as a piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to a space-time structure of the pieces of input image data;

an image information storing unit for storing each piece of for-retrieval description data with the piece of input image data corresponding to the piece of for-retrieval description data;

a first retrieval processing unit;

a second retrieval processing unit; and

a user interface unit,

wherein a retrieval request received from a user through the user interface unit is sent from the second retrieval processing unit to the first retrieval processing unit as retrieval request information, a retrieval result sent from the first retrieval processing unit is received in the second retrieval processing unit, the second retrieval processing unit presents the retrieval result to the user through the user interface unit, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.

14. An image retrieval system according to claim 13, further comprising:

a characteristic descriptor reliability calculating unit for calculating a degree of reliability of each characteristic descriptor produced in the characteristic descriptor producing unit,

wherein the for-retrieval description data producing unit describes each characteristic descriptor and the degree of reliability of the characteristic descriptor as one piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to the space-time structure of the pieces of input image data, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors and the degrees of reliability of the characteristic descriptors, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors and the degrees of reliability of the characteristic descriptors to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.

15. An image retrieval system according to claim 14, wherein the necessity of the conformity judgment processing for each characteristic descriptor is estimated in the first retrieval processing unit according to the degree of reliability of the characteristic descriptor in case of the conformity judgment processing, and the conformity judgment processing for the characteristic descriptor is skipped in cases where it is judged that the conformity judgment processing for the characteristic descriptor is not necessary.

16. An image retrieval system according to claim 14, further comprising:

a characteristic descriptor retrieval order determining unit for determining a use order of the characteristic descriptors in the retrieval according to the degrees of reliability calculated in the characteristic descriptor reliability calculating unit, wherein the for-retrieval description data producing unit describes each characteristic descriptor, the degree of reliability of the characteristic descriptor and the use order in the retrieval as one piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to the space-time structure of the pieces of input image data, the pieces of for-retrieval description data of the pieces of input image data stored in the image information storing unit are analyzed in the first retrieval processing unit according to the retrieval request information sent from the second retrieval processing unit to extract the characteristic descriptors, the degrees of reliability of the characteristic descriptors and the use order in the retrieval, a conformity judgment processing is performed in the first retrieval processing unit according to the extracted characteristic descriptors, the degrees of reliability and the use order in the retrieval to obtain the retrieval result, and the retrieval result is sent from the first retrieval processing unit to the second retrieval processing unit to make the second retrieval processing unit present the retrieval result to the user through the user interface unit.

17. An image retrieval system according to claim 13, wherein the retrieval request information, in which a use order of the characteristic descriptors in the retrieval, is sent from the second retrieval processing unit to the first retrieval processing unit as the retrieval request sent from the user through the user interface unit, and a retrieval order of the characteristic descriptors is renewed in the first retrieval processing unit according to the use order in the retrieval which is included in the retrieval request information sent from the second retrieval processing unit to perform a retrieval processing corresponding to the retrieval request of the user.

18. An image retrieval system according to claim 16, wherein the number of characteristic descriptor candidates in the retrieval result obtained in the conformity judgment processing is determined according to the degrees of reliability of the characteristic descriptors in the first retrieval processing unit for each of a plurality of retrieval steps of which an applied order is determined according to the use order in the retrieval, and the conformity judgment processing is performed.

19. An image retrieval system according to claim 17, wherein the number of characteristic descriptor candidates in the retrieval result obtained in the conformity judgment processing is determined according to the degrees of reliability of the characteristic descriptors in the first retrieval processing unit for each of a plurality of retrieval steps of which an applied order is determined according to the use order in the retrieval, and the conformity judgment processing is performed.

20. An image retrieval system according to claim 13, further comprising:

a video data reproducing server; and

a video data decoding and reproducing unit, wherein each piece of input image data denotes a piece of video data, a piece of key image data representing each of pieces of video data specified in the retrieval is sent from the first retrieval processing unit to the second retrieval processing unit as the retrieval result,

each piece of key image data is received in the second retrieval processing unit as the retrieval result sent from the first retrieval processing unit to present the retrieval result to the user through the user interface unit,

a reproduction request indicating a specific key image, which is selected by the user from a plurality of key images indicated by the pieces of key image data, is sent from the user interface unit and is received in the video data reproducing server, a piece of specific video data represented by the specific key image is read out from the image information storing unit and is sent to the video data decoding and reproducing unit, and

the piece of specific video data sent from the video data reproducing server is received in the video data decoding and reproducing unit, is decoded and is received.

21. An image retrieval method, comprising the steps of:

extracting a plurality of image characteristic values from pieces of input image data to produce a characteristic descriptor for each piece of input image data;

describing each characteristic descriptor as a piece of for-retrieval description data while holding the correspondence of the piece of for-retrieval description data to a space-time structure of the pieces of input image data;

storing each piece of for-retrieval description data with the piece of input image data corresponding to the piece of for-retrieval description data;

analyzing the pieces of stored for-retrieval description data of the pieces of input image data according to a retrieval request sent from a user to extract the characteristic descriptors;

performing a conformity judgment processing according to the extracted characteristic descriptors to obtain a retrieval result; and

presenting the retrieval result to the user.

22. An image retrieval method according to claim 21, further comprising the steps of:

preparing pieces of video data as the pieces of input image data;

presenting a piece of key image data, which represents each of pieces of video data specified in the retrieval, to the user as the retrieval result;

receiving a reproduction request indicating a specific key image which is selected by the user from a plurality of key images indicated by the pieces of key image data;

reading out a piece of specific video data represented by the specific key image; and

decoding and reproducing the piece of specific video data.