Still picture search/retrieval method carried out on the basis of color information and system for carrying out the same

• US 5,652,881 A
• Filed: 11/23/1994
• Issued: 07/29/1997
• Est. Priority Date: 11/24/1993
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Takahashi, Masamitsu, Iwai, Noriyuki, Yanagi, Kunihiro
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): HOMERE, JEAN RAYMOND
• Application Number:
  US08/346,301
• Time in Patent Office: 979 Days
• Field of Search: 395/600, 395/615, 395/613, 395/603, 395/605, 358/403, 364/DIG. 1, 382/162, 382/165, 382/181, 382/305
• US Class Current: 382/162
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Still picture search/retrieval method and system for retrieving a still picture without omission by extracting representative color component from a location of the still picture which includes a thing for search. A search condition input processing module receives color information constituting a search condition via an input unit. A still picture dividing method determination processing module receives via the input unit a method for dividing the still picture into rectangular meshes of a same size. A divided mesh designation processing module receives designation of a mesh for search from those resulting from the division. Still picture information is read out from a database for dividing each still picture into meshes by a still picture division processing module. A representative color component extraction processing module extracts a color component having the highest frequency of histogram within the search-destined mesh. A search execution processing module evaluates the representative color component to determine an evaluation value. A still picture having the evaluation value exceeding a threshold value is retrieved as the search result. The retrieved still picture is displayed on a display unit through a searched still picture display processing module.

Claims

1. A still picture search/retrieval system for selectively searching a particular still picture of concern from a plurality of still pictures stored in a database on the basis of a color designated as a search condition and displaying said particular still picture retrieved as the result of the search, comprising:

input means for inputting color data-related information concerning a color designated as the search condition for performing said search and region-related information concerning a region of each of said still pictures designated for performing said search;

search means for selecting from each of said still pictures a search-destined region for which the search is to be actually performed on the basis of said region-related information, determining a representative value of the color data of each of said still pictures on the basis of the color data of said selected search-destined region and deciding whether or not said representative value satisfies conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists;

display means for displaying said hit still picture as the still picture retrieved as the result of said search;

said search means being provided availably with a plurality of search methods each for deciding whether or not said representative value satisfies said search conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists;

wherein a command indicating which one is to be selected from said plurality of search methods is inputted through said input means;

wherein said search means performs the still picture search in accordance with said selected search method;

said region-related information including region division information for dividing each of the search-designated still pictures into a plurality of regions and search-destined region designating information for designating the search-destined region from the regions resulting from said division;

still picture dividing means for dividing each of said still pictures into a plurality of regions in accordance with said region division information;

wherein said search means includes;

representative color component extracting means for determining a representative value of the color data for each of said still pictures on the basis of the color data of said region designated by said search-destined region designating information; and

search execution means for determining evaluation values for said still pictures, respectively, on the basis of said color data-related information and said representative values of the color data of said still pictures, respectively, to thereby output said still pictures in a descending order of said evaluation values;

wherein said display means displays said still pictures retrieved by said search execution means in a descending order of said evaluation values.

2. A still picture search/retrieval system according to claim 1,

said color data including color component data for a plurality of color components constituting said color;

and

said color component data being divided into a plurality of groups in dependence on sizes of said color component data;

wherein upon determining said representative value of said color data, histogram of the color data contained in said search-destined region is determined to thereby determine the color data of a highest frequency as said representative value; and

wherein upon determination of said histogram, the color data are regarded to be identical with one another when all of said plurality of color component data constituting each of said color data belong to a same one of said color component data groups.

3. A still picture search/retrieval system according to claim 1,

said color data including color component data for a plurality of color components constituting said color data;

said color component data being divided into a plurality of color component groups on a basis of said color component data per se;

said input means being inputted with fuzzy degrees representing degrees of similarity of the designated color to each of said color component groups; and

said search means determining the color component data corresponding to said representative value of said determined color data for deriving a fuzzy evaluation value of the representative value of said color data on the basis of the said fuzzy degrees for said color component data, to thereby retrieve a particular still picture on the basis of said fuzzy evaluation value.

4. A still picture search/retrieval system for selectively searching a particular still picture of concern from a plurality of still pictures stored in a database on the basis of a color designated as a search condition and displaying said particular still picture retrieved as the result of the search, comprising:

input means for inputting color data-related information concerning a color designated as the search condition for performing said search and region-related information concerning a region of each of said still pictures designated for performing said search;

search means for selecting from each of said still pictures a search-destined region for which the search is to be actually performed on the basis of said region-related information, determining a representative value of the color data of each of said still pictures on the basis of the color data of said selected search-destined region and deciding whether or not said representative value satisfies conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists;

display means for displaying said hit still picture as the still picture retrieved as the result of said search said color data including color component data for a plurality of color components constituting said color data;

said color component data being divided into a plurality of color component groups on a basis of said color component data per se;

said input means being inputted with fuzzy degrees representing degrees of similarity of the designated color to each of said color component groups;

said search means determining the color component data corresponding to said representative value of said determined color data for deriving a fuzzy evaluation value of the representative value of said color data on the basis of the said fuzzy degrees for said color component data, to thereby retrieve a particular still picture on the basis of said fuzzy evaluation value; and

retrieved still picture output means for outputting sequentially a plurality of still pictures, if retrieved, in a descending order of said fuzzy evaluation values;

wherein said display means displays said still pictures in a descending order of said fuzzy evaluation values.

5. A still picture search/retrieval method for selectively searching a particular still picture of concern from a plurality of still pictures stored in a database on the basis of a color designated as search condition and displaying said particular still picture retrieved through the search, said method comprising the steps of:

inputting color data-related information concerning a color designated as search condition for performing execution of said search and region-related information concerning a region of each of said still pictures designated for execution of said search;

selecting from each of said still pictures a search-destined region which is to be subjected to the search in accordance with said region-related information, determining a representative value of the color data of each of said still pictures on the basis of the color data of said selected search-destined region and deciding whether or not said representative value satisfies conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists; and

displaying said hit still picture as the still picture retrieved as the result of said search;

providing a plurality of search procedures each for deciding whether or not said representative value satisfies said conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists;

inputting a command indicating which one is to be selected from said plurality of search procedures;

wherein the still picture search is executed in accordance with said selected search procedure;

wherein said region-related information includes region division information for dividing each of the search-designated still pictures into a plurality of regions and search-destined region designating information for designating selectively the search-destined region from the regions resulting from said division;

dividing each of said still pictures into a plurality of regions in accordance with said region division information;

determining a representative value of the color data for each of said still pictures on the basis of the color data of said region designated by said search-destined region designating information;

determining evaluation values for said still pictures, respectively, on the basis of said color data-related information and said representative values of the color data of said still pictures, respectively, to thereby retrieve and output said still pictures in a descending order of said evaluation values; and

displaying said retrieved still pictures in a descending order of said evaluation values.

6. A still picture search/retrieval method according to claim 5,

wherein said color data include color component data for a plurality of color components constituting each color;

further comprising the steps of;

dividing said color component data into a plurality of groups in dependence on sizes of said color component data; and

determining histogram of the color data contained in said search-destined region upon determination of said representative value of said color data to thereby determine the color data of a highest frequency as said representative value;

wherein upon determination of said histogram, the color data are regarded to be identical with one another when all of said plurality of color component data constituting each of said color data belong to a same one of said color component data groups.

7. A still picture search/retrieval method according to claim 5,

wherein said color data include color component data for a plurality of color components constituting each color;

further comprising the steps of;

dividing said color component data into a plurality of color component groups on a basis of said color component data;

inputting fuzzy degrees representing degrees of similarity of the designated color to said color component groups, respectively; and

determining the color component data corresponding to said representative value of said determined color data for thereby deriving a fuzzy evaluation value of the representative value of said color data on the basis of the said fuzzy degrees for said color component data, to thereby retrieve a particular still picture on the basis of said fuzzy evaluation value.

8. A still picture search/retrieval method for selectively searching a particular still picture of concern from a plurality of still pictures stored in a database on the basis of a color designated as search condition and displaying said particular still picture retrieved through the search, said method comprising the steps of:

inputting color data-related information concerning a color designated as search condition for performing execution of said search and region-related information concerning a region of each of said still pictures designated for execution of said search;

selecting from each of said still pictures a search-destined region which is to be subjected to the search in accordance with said region-related information, determining a representative value of the color data of each of said still pictures on the basis of the color data of said selected search-destined region and deciding whether or not said representative value satisfies conditions determined on the basis of said color data-related information to thereby decide whether or not any hit still picture exists;

displaying said hit still picture as the still picture retrieved as the result of said search;

wherein said color data include color component data for a plurality of color components constituting each color;

dividing said color component data into a plurality of color component groups on a basis of said color component data;

inputting fuzzy degrees representing degrees of similarity of the designated color to said color component groups, respectively; and

determining the color component data corresponding to said representative value of said determined color data for thereby deriving a fuzzy evaluation value of the representative value of said color data on the basis of the said fuzzy degrees for said color component data, to thereby retrieve a particular still picture on the basis of said fuzzy evaluation value;

outputting sequentially a plurality of still pictures, if retrieved, in a descending order of said fuzzy evaluation values; and

displaying said still pictures in a descending order of said fuzzy evaluation values.

9. A still picture search/retrieval system for selectively searching a particular still picture of concern from a plurality of still pictures on the basis of a color designated as search condition and displaying said particular still picture retrieved as the result of the search,

comprising:

an input unit for inputting information concerning search-destined color data and a search-destined region on the basis of which the search is to be performed;

a display unit for displaying results of the search;

a CPU including a command analysis module for analyzing commands inputted through said input means to thereby distribute said command to relevant processings, respectively, a search condition input processing module for inputting said color data information and an evaluation method selecting command, a still picture dividing method determination processing module for inputting a method for dividing said still picture into rectangular meshes of a same size, a divided mesh designation processing module for designating a search-destined mesh containing a thing which constitutes a subject for the search electively from the meshes resulting from said division, an evaluation method determination processing module for determining the evaluation procedure for the search on the basis of said search condition, a still picture division processing module for dividing said still picture in accordance with said inputted division method, a representative color component extraction processing module for extracting a representative color component of said search-destined mesh, a search execution processing module for selecting a still picture as the result of said search by evaluating said representative color component, and a searched still picture display processing module for displaying the searched still picture on said display unit;

a memory including an area for storing color gradation division numbers, a table for storing representative gradation values for said color gradation divisions, respectively, a fuzzy flag area having a flag indicating whether or not a fuzzy influence is to be employed in executing the search for the purpose of enabling still pictures of colors approximating the designated color to be retrieved, a table for storing fuzzy grades of inputted color components, respectively, when said fuzzy inference is to be employed for the search, a table for storing inputted gradation values serving as the representative color components when said fuzzy inference is not employed in the search, a still picture division information storing table for storing divisors for dividing the still picture in the vertical (heightwise) direction and in the horizontal (widthwise) direction, respectively, an area for storing locations of the designated regions in terms of identifier numbers, respectively, a table for storing the gradation values at starting points and end points of several ranges into which the color component gradation is to be divided, an evaluation table for storing Cartesian products of fuzzy sets inputted, a still picture information table for storing file names of still pictures extracted from a database, a search-destined mesh picture information table for storing still picture data of the search-destined meshes cut out from said still picture data, a frequency table for storing values of histogram, an area for storing an evaluation value corresponding to the representative color component value read out from said evaluation table, an area for storing a threshold value serving as a reference value for the search and determined previously in said system, and a search result table for storing results of the search; and

a database storing said still pictures.

10. A still picture search/retrieval method for performing a search of a particular still picture on the basis of a color of concern from a plurality of still pictures in accordance with commands inputted through an input unit,

said method comprising:

(a) a step of inputting color information as search condition and a method of dividing the still picture into rectangular meshes of a same size;

(b) a step of performing search condition input processing on the basis of said color information;

(c) a still picture division method determination processing step for determining a still picture dividing method on the basis of said division method;

(d) a divided mesh designation processing step for designating a mesh containing a thing constituting a subject for search from the meshes resulting from said division in said step (a) on the basis of the result of said still picture division method determination processing step (c);

(e) an evaluation method determination processing step for determining an evaluation method for the search on the basis of the color information inputted as the search condition;

(f) a still picture division processing step for dividing picture information extracted from a database in accordance with the division method determined in said still picture division method determination processing step (c) and cutting out picture information of the search-destined mesh designated in said divided mesh designation processing step (d);

(g) a representative color component extraction processing step for extracting a representative color component of the original still picture from the picture information of the search-destined mesh cut out in said still picture division processing step (f);

(h) a search execution processing step for determining the value of said representative color component in accordance with the evaluation method determined in said evaluation method determination processing step (e) for thereby retrieving said representative color component as the result of the search when said value is not smaller than a threshold value determined previously; and

(i) a search result display processing step for displaying said retrieved still picture after execution of said still picture division processing step (f), said representative color component extraction processing step (g) and said search execution processing (h) for each of the still pictures.

11. A still picture search/retrieval method according to claim 10,

said search condition input processing step

(b) including the substeps of:

(b1) determining whether the fuzzy inference is to be adopted or not;

(b2) setting on said fuzzy flag when said fuzzy inference is determined to be adopted;

(b3) responding to the fuzzy flag set on for thereby storing in a fuzzy grade table fuzzy grade values indicating degrees of similarity of the representative gradation values of the color components (RGB), respectively, contained in the representative gradation table stored in said memory to said color of concern;

(b4) setting off said fuzzy flag when said fuzzy inference is determined not to be adopted; and

(b5) responding to the fuzzy flag set off for thereby inputting the color constituting the search condition in terms of the representative gradation value thereof and storing it in the representative color component (RGB) table;

said still picture division method input processing step (a) including the substeps of;

(a1) designating the method of dividing the search-destined still picture into the meshes in terms of number of divisions in the vertical and horizontal directions, respectively;

(a2) placing said number of divisions in the vertical direction and said number of divisions in the horizontal direction as inputted in a still picture division information table secured on said memory for storing said numbers of divisions;

said divided mesh designation processing step

(d) including the substeps of;

(d1) determining the number of divisions for the still picture on the basis of said still picture division information to thereby display on said display unit the still picture as divided; and

(d2) designating the search-destined mesh and storing the identifier number of said designated search-destined mesh in a search-destined mesh picture information table;

said evaluation method determination processing step (e) including the substep of;

(e1) performing an evaluation table generation processing when said fuzzy flag is set on;

said still picture division processing step (f) including the substeps of;

(f1) extracting the still picture information from the database to thereby store file names thereof in a file name of a still picture information file name table; and

(f2) cutting out picture information of the search-destined mesh from said still picture information on the basis of said still picture information and said search-destined mesh picture information for thereby storing the cut-out picture information in a search-destined mesh picture information table;

said representative color component extraction processing step (g) including the substep of;

(g1) generating a frequency table containing histograms of color components of individual pixels located within said search-destined meshes for divided color space regions, respectively;

said search execution processing step (h) including the substeps of;

(h1) responding to the fuzzy flag set on for thereby determining from said representative gradation table identifier numbers corresponding to each component of the representative color component values, respectively;

(h2) determining elements corresponding to said identifier numbers, respectively, from the evaluation table secured on said memory to thereby storing the determined elements in an evaluation table secured on said memory;

(h3) comparing the evaluation value as obtained with the threshold value determined previously;

(h4) responding to the result of said comparison indicating that said evaluation value is not smaller than the threshold value determined previously, for thereby storing the still picture identifier number and the file name of said still picture as relevant entries in the search result table secured on said memory by regarding said still picture as satisfying the search condition; and

(h5) responding to the fuzzy flag set off for checking whether the representative color component (RGB) value is same as the representative color component value of a still picture, to thereby store the still picture identifier number and the file name of said still picture in the search result table as the relevant entries by deciding that said still picture satisfies the search condition; and

said searched still picture display processing step (i) including the substeps of;

(i1) deciding whether or not the result of the search is obtained;

(i2) responding to absence of the search result for thereby displaying a corresponding message on said display unit;

(i3) responding to presence of the search result for thereby deciding whether or not a plurality of search results are obtained;

(i4) responding to the presence of a single search result for thereby displaying the still picture stored in said search result table in an original size;

(i5) responding to the presence of plural search results and the fuzzy flag set on for thereby sorting said plural search results in a descending order of the evaluation values contained in said search result table;

(i6) displaying the still pictures on said display unit in a reduced scale in the order in which said still pictures are stored in said search result table;

(i7) selecting one of the still pictures being displayed on said display unit; and

(i8) displaying the selected still picture on said display unit in the original size.