Information Processing Device, Information Processing Method and Program

• US 20130163860A1
• Filed: 08/02/2011
• Published: 06/27/2013
• Est. Priority Date: 08/11/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suzuki, Hirotaka, Ito, Masato
• Granted Patent: US 9,280,709 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

The present invention relates to an information processing device, an information processing method, and a program capable of easily adding an annotation to content.

A feature amount extracting unit 21 extracts an image feature amount of each frame of an image of learning content and extracts word frequency information regarding frequency of appearance of each word in a description text describing a content of the image of the learning content (for example, a text of a caption) as a text feature amount of the description text. A model learning unit 22 learns an annotation model, which is a multi-stream HMM, by using an annotation sequence for annotation, which is a multi-stream including the image feature amount of each frame and the text feature amount. The present invention may be applied when adding the annotation to the content such as a television broadcast program, for example.

Claims

1. An information processing device, comprising:

feature amount extracting means for extracting an image feature amount of each frame of an image of learning content and extracting word frequency information regarding frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text; and

model learning means for learning an annotation model, which is a multi-stream HMM (hidden Markov model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount.

2. The information processing device according to claim 1, wherein

the learning content includes a text of a caption, and

the description text is the text of the caption included in the learning content.

3. The information processing device according to claim 2, wherein

the feature amount extracting means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount.

4. The information processing device according to claim 2, further comprising:

annotation adding means for adding an annotation to target content to which the annotation is to be added by using the annotation model.

5. The information processing device according to claim 4, wherein

the feature amount extracting means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and

the annotation adding means

extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, and

selects a word with high frequency in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as the annotation to be added to the frame of interest.

6. The information processing device according to claim 2, further comprising:

frame searching means for searching a keyword frame from target content from which the keyword frame, which is a frame of which content coincides with a predetermined keyword, is to be searched by using the annotation model.

7. The information processing device according to claim 6, wherein

the feature amount extracting means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and

the frame searching means

extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model, and

selects, when a frequency of the predetermined keyword is high in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence, the frame of interest as the keyword frame.

8. The information processing device according to claim 2, further comprising:

display controlling means for displaying an annotation to be added to a frame of target content to which the annotation is to be added by using the annotation model.

9. The information processing device according to claim 8, wherein

the feature amount extracting means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and

the display controlling means

extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount,

obtains a state corresponding to each frame of the target content by obtaining a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

obtains the annotation to be added to the frame corresponding to the state based on the multinomial distribution observed in the state, and

displays the annotation to be added to the frame corresponding to the state for each state of the annotation model.

10. The information processing device according to claim 9, wherein

the display controlling means

obtains an inter-state distance from one state to another state of the annotation model based on state transition probability from the one state to the another state,

obtains a state coordinate, which is a coordinate of a position of the state on a model map, such that an error between a Euclidean distance from the one state to the another state on the model map, which is a map on which the states of the annotation model are arranged, and the inter-state distance becomes small,

displays the model map, on which the corresponding state is arranged, in a position of the state coordinate, and

displays a representative image, which represents the frame corresponding to the state, and the annotation to be added to the frame corresponding to the state in the state on the model map.

11. The information processing device according to claim 2, wherein

the feature amount extracting means performs dimension reduction to reduce a dimension of the image feature amount and the text feature amount, and

the model learning means learns the annotation model by using the multi-stream including the image feature amount and the text feature amount after the dimension reduction as the annotation sequence.

12. The information processing device according to claim 11, wherein

the feature amount extracting means

obtains basis space data of a basis space for image of which dimension is lower than the dimension of the image feature amount for mapping the image feature amount by using the image feature amount,

performs the dimension reduction of the image feature amount based on the basis space data of the basis space for image,

obtains basis space data of a basis space for text of which dimension is lower than the dimension of the text feature amount for mapping the text feature amount by using the text feature amount, and

performs the dimension reduction of the text feature amount based on the basis space data of the basis space for text.

13. The information processing device according to claim 12, wherein

the feature amount extracting means

obtains a code book used for vector quantization as the basis space data of the basis space for image by using the image feature amount, and

obtains a code representing a centroid vector as the image feature amount after the dimension reduction by performing the vector quantization of the image feature amount by using the code book.

14. The information processing device according to claim 12, wherein

the feature amount extracting means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals,

extracts a frequency of appearance of each word in the document as the text feature amount,

obtains a parameter of LDA (latent Dirichlet allocation) as the basis space data of the basis space for text by learning the LDA by using the document obtained from the learning content, and

converts the text feature amount obtained from the document to topic likelihood, which is likelihood of each latent topic of the LDA for the document, by using the parameter of the LDA, to obtain a topic label representing the latent topic of which topic likelihood is the maximum as the text feature amount after the dimension reduction.

15. The information processing device according to claim 14, further comprising:

annotation adding means for adding an annotation to target content to which the annotation is to be added by using the annotation model, wherein

the feature amount extracting means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the annotation adding means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest, and

selects a word with high appearance frequency in the frame topic as the annotation to be added to the frame of interest based on the topic-to-frequently appearing word table.

16. The information processing device according to claim 14, further comprising

frame searching means for searching a keyword frame from target content from which the keyword frame, which is a frame of which content coincides with a predetermined keyword, is to be searched by using the annotation model, wherein

the feature amount extracting means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the frame searching means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest, and

obtains appearance frequency of the predetermined keyword in the frame topic based on the topic-to-frequently appearing word table and selects, when the appearance frequency of the predetermined keyword is high, the frame of interest as the keyword frame.

17. The information processing device according to claim 14, further comprising:

display controlling means for displaying an annotation to be added to a frame of target content to which the annotation is to be added by using the annotation model, wherein

the feature amount extracting means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the display controlling means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a state corresponding to each frame of the target content by obtaining a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in the state as a frame topic representing a content of the frame corresponding to the state,

obtains a word with high appearance frequency in the frame topic as the annotation to be added to the frame of which content is represented by the frame topic based on the topic-to-frequently appearing word table, and

displays the annotation to be added to the frame corresponding to the state for each state of the annotation model.

18. The information processing device according to claim 14, further comprising:

display controlling means for displaying an annotation to be added to a frame of target content to which the annotation is to be added by using the annotation model, wherein

the feature amount extracting means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the display controlling means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a state corresponding to each frame of the target content by obtaining a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in the state as a frame topic representing a content of the frame corresponding to the state,

obtains a word with high appearance frequency in the latent topic as the annotation to be added to the frame of which frame topic is the latent topic based on the topic-to-frequently appearing word table, and

displays the annotation to be added to the frame of which frame topic is the latent topic for each latent topic.

19. An information processing method to be performed by an information processing device, comprising the steps of:

extracting an image feature amount of each frame of an image of learning content and extracting word frequency information regarding frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text; and

learning an annotation model, which is a multi-stream HMM (hidden Markov model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount.

20. A program for allowing a computer to function as

feature amount extracting means for extracting an image feature amount of each frame of an image of learning content and extracting word frequency information regarding frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text;

and

model learning means for learning an annotation model, which is a multi-stream HMM (hidden Markov model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount.