Multimedia information retrieval method, program, record medium and system

• US 7,167,823 B2
• Filed: 11/27/2002
• Issued: 01/23/2007
• Est. Priority Date: 11/30/2001
• Status: Expired due to Fees

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Masumoto, Daiki, Endo, Susumu, Shiitani, Syuuichi, Uehara, Yuusuke
• Primary Examiner(s): HARPER, V PAUL
• Application Number:
  US10/305,184
• Publication Number:
  US 20030103675A1
• Time in Patent Office: 1,518 Days
• Field of Search: None
• US Class Current: 704/7
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

Paired image information and text information correlated to each other are retrieved as information sets. Frequency information on words used in text is extracted from text information in a group of information sets, and text information features are extracted based on frequency information. Text features are used to lay out information sets in a virtual space such that similar pieces of text are located close to each other, and images are displayed at those positions. Further, important words are extracted from those words extracted from text information in a group of information sets, and those words are laid out in the virtual space in the same manner as with information sets and displayed as labels.

Claims

1. A multimedia information retrieval method comprising:

a word frequency extraction step using as information sets paired image information and text information correlated to each other and extracting constituent word frequency information from said text information within said information sets;

a text feature extraction step extracting text features, based on said word frequency information of individual information sets;

an information set classification and layout step classifying and laying out said information sets in a virtual space, based on said text features;

a label feature extraction step selecting labels from constituent words of said text information within each information set and extracting features of selected labels;

a label layout step placing labels at positions corresponding to information sets classified and laid out in the virtual space, based on said label features; and

an information display step displaying image information and labels of said information sets, placed in said virtual space, depending on the positions of the viewpoint.

2. The method as defined in claim 1, wherein said information set classification and layout step includes classifying and laying out said information sets on a two-dimensional plane at a predetermined position in a three-dimensional virtual space, based on said text features, and

wherein said label layout step includes placing said labels at the front of the two-dimensional plane in which the information sets are classified and laid out, based on said label features.

3. The method as defined in claim 1, wherein said text feature extraction step comprises:

a morpheme analysis step extracting predetermined parts of speech such as nouns, noun compounds and adjectives by morpheme analyses of text information and creating a word list comprised of words used and their frequencies of occurrence;

a matrix creation step creating a word-text matrix whose rows and columns correspond respectively to text information and words and in which word frequencies of occurrence are laid out as elements; and

a text feature conversion step expressing text information of said word-text matrix by document vectors having coordinate points determined by frequencies of occurrence in a word space which has words as axes, projecting said document vectors onto a low-dimensional space by singular value decomposition and using as text features document vectors representing positions in said low-dimensional space.

4. The method as defined in claim 3, wherein said matrix creation step comprises:

a weight assignment step assigning weight to elements in said word-text matrix, based on word frequencies of occurrence in each text.

5. The method as defined in claim 1, wherein said label feature extraction step comprises:

a label selection step figuring out the significance of each constituent word of text information within each of said information sets and selecting words to be used as labels, based on the significance figured out.

6. The method as defined in claim 5, wherein said label selection step comprises:

a morpheme analyses step extracting nouns and noun compounds by morpheme analyses of text information and creating a word list comprised of words used and their frequencies of occurrence;

a matrix creation step creating a word-text matrix whose rows and columns correspond respectively to text information and words and in which said word frequencies of occurrence are laid out as elements; and

a label feature conversion step expressing words of said word-text matrix by word vectors having coordinate points determined by frequencies of occurrence in a text space which has individual pieces of text information as axes, projecting said word vectors onto a low-dimensional space by singular value decomposition and using as label features word vectors representing positions in the low-dimensional space;

wherein a predetermined number of words are selected as labels in descending order of significance which is represented by lengths of said word vectors.

7. The method as defined in claim 5 or 6, wherein said label layout step includes displaying labels such that the higher the significance figured out by said label selection step, the more the labels are displayed toward the front of said virtual space;

and

wherein said information display step includes changing how labels appear and the size in which they appear, depending on the position of the viewpoint relative to said virtual space.

8. The method as defined in claim 7, wherein said information display step includes fixing the horizontal position of said label relative to image information regardless of a horizontal displacement of the viewpoint in said virtual space and changing how labels appear and the size in which they appear depending on a change in the position of the viewpoint in the direction of depth.

9. The method as defined in claim 1, further comprising an information collection step collecting from the Internet information sets comprised of paired image and text information correlated to each other.

10. The method as defined in claim 9, wherein said information collection step comprises;

a relation analysis step analyzing the relationship between image information and text information and determining the range of information to be collected as information sets, if the relationship between said image information and said text information is unclear.

11. A computer readable record medium having therein stored a program for retrieving multimedia information, said program allowing a computer to execute:

a word frequency extraction step using as information sets paired image information and text information correlated to each other and extracting constituent word frequency information from said text information within said information sets

a text feature extraction step extracting text features, based on said word frequency information of individual information sets;

an information set classification and layout step classifying and laying out said information sets in a virtual space, based on said text features;

a label feature extraction step selecting labels from constituent words of said text information within each information set and extracting features of selected labels;

a label layout step placing labels at positions corresponding to information sets classified and laid out in the virtual space, based on said label features; and

an information display step displaying image information and labels of said information sets, placed in said virtual space, depending on the positions of the viewpoint.

12. The program as defined in claim 11, wherein said information set classification and layout step includes classifying and laying out said information sets on a two-dimensional plane at a predetermined position in a three-dimensional virtual space, based on said text features, and

wherein said label layout step includes placing said labels at the front of the two-dimensional plane in which the information sets are classified and laid out, based on said label features.

13. The program as defined in claim 11, wherein said text feature extraction step comprises:

a morpheme analysis step extracting predetermined parts of speech such as nouns, noun compounds and adjectives by morpheme analyses of text information and creating a word list comprised of words used and their frequencies of occurrence;

a matrix creation step creating a word-text matrix whose rows and columns correspond respectively to text information and words and in which word frequencies of occurrence are laid out as elements; and

a text feature conversion step expressing text information of said word-text matrix by document vectors having coordinate points determined by frequencies of occurrence in a word space which has words as axes, projecting said document vectors onto a low-dimensional space by singular value decomposition and using as text features document vectors representing positions in said low-dimensional space.

14. The program as defined in claim 13, wherein said matrix creation step comprises:

a weight assignment step assigning weight to elements in said word-text matrix, based on word frequencies of occurrence in each text.

15. The program as defined in claim 11, wherein said label feature extraction step comprises:

a label selection step figuring out the significance of each constituent word of text information within each of said information sets and selecting words to be used as labels, based on the significance figured out.

16. The program as defined in claim 15, wherein said label selection step comprises:

a morpheme analyses step extracting nouns and noun compounds by morpheme analyses of text information and creating a word list comprised of words used and their frequencies of occurrence;

a matrix creation step creating a word-text matrix whose rows and columns correspond respectively to text information and words and in which said word frequencies of occurrence are laid out as elements; and

a label feature conversion step expressing words of said word-text matrix by word vectors having coordinate points determined by frequencies of occurrence in a text space which has individual pieces of text information as axes, projecting said word vectors onto a low-dimensional space by singular value decomposition and using as label features word vectors representing positions in the low-dimensional space;

wherein

a predetermined number of words are selected as labels in descending order of significance which is represented by lengths of said word vectors.

17. The program as defined in claim 15 or 16, wherein said label layout step includes displaying labels such that the higher the significance figured out by said label selection step, the more the labels are displayed toward the front of said virtual space;

and

wherein said information display step includes changing how labels appear and the size in which they appear, depending on the position of the viewpoint relative to said virtual space.

18. The program as defined in claim 17, wherein said information display step includes fixing the horizontal position of said label relative to image information regardless of a horizontal displacement of the viewpoint in said virtual space and changing how labels appear and the size in which they appear depending on a change in the position of the viewpoint in the direction of depth.

19. The program as defined in claim 11, further comprising:

an information collection step collecting from the Internet information sets comprised of paired image and text information correlated to each other.

20. The program as defined in claim 19, wherein said information collection step comprises:

a relation analysis step analyzing the relationship between image information and text information and determining the range of information to be collected as information sets, if the relationship between said image information and said text information is unclear.

21. A multimedia information retrieval system comprising:

a word frequency extraction unit using as information sets paired image information and text information correlated to each other and extracting constituent word frequency information from said text information within said information sets;

a text feature extraction unit extracting text features, based on said word frequency information of individual information sets;

an information set classification and layout unit classifying and laying out said information sets in a virtual space, based on said text features;

a label feature extraction unit selecting labels from constituent words of said text information within each information set and extracting features of selected labels;

a label layout unit placing labels at positions corresponding to information sets classified and laid out in the virtual space, based on said label features; and

an information display unit displaying image information and labels of said information sets, placed in said virtual space, depending on the positions of the viewpoint.