Information processing device, information processing method and program

• US 9,232,205 B2
• Filed: 08/02/2011
• Issued: 01/05/2016
• Est. Priority Date: 08/11/2010
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suzuki, Hirotaka, Ito, Masato
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Aghevli, Reza
• Application Number:
  US13/814,181
• Publication Number:
  US 20130183022A1
• Time in Patent Office: 1,617 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04N 9/87 Regeneration of colour television signals H04N9/80 takes precedence

Abstract

An information processing device, method and a program adds an annotation to content and provides an application which utilizes the annotation. A learning module extracts an image feature amount of each frame of an image of learning content and extracts word frequency information regarding a frequency of appearance of each word in a description text, and learns an annotation models which is a multi-stream Hidden Markov Model (HMM) by using a multi-stream including the image feature amount and the text feature amount. A browsing controller extracts a scene which is a group of one or more temporally continuous frames from target content by using the annotation model and displays representative images of the scenes so as to be arranged in chronological order.

Claims

1. An information processing device, comprising:

a learning module configured to

extract an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,

wherein the learning content includes a text of a caption, and

wherein the description text is the text of the caption included in the learning content, and

learn an annotation model, which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount; and

a browsing controller configured to extract a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model, and

display representative images of scenes so as to be arranged in chronological order.

2. The information processing device according to claim 1, wherein

the learning means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount.

3. The information processing device according to claim 1, wherein

the learning means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and

the browsing controlling means

extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects a word with high frequency in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as an annotation to be added to the frame of interest,

extracts a group of one or more temporally continuous frames to which the same annotation is added as the scene from the target content, and

displays the representative images of the scenes so as to be arranged in chronological order.

4. The information processing device according to claim 3, wherein

the target content is content of a broadcast program, and

the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide).

5. The information processing device according to claim 3, wherein

the browsing controlling means also displays the annotation added to the frame, which composes the scene, together with a representative image of the scene.

6. The information processing device according to claim 1, wherein

the learning means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals, and

extracts multinomial distribution, which represents a frequency of appearance of each word in the document, as the text feature amount, and

the browsing controlling means

extracts the image feature amount of each frame of the image of the target content and composes the annotation sequence by using the image feature amount,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects, when a frequency of a predetermined keyword is high in the multinomial distribution observed in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence, the frame of interest as a keyword frame, which is the frame of which content coincides with a predetermined keyword,

extracts a group of one or more temporally continuous frames as the scene from the keyword frame, and

displays the representative images of the scenes so as to be arranged in chronological order.

7. The information processing device according to claim 6, wherein

the target content is content of a broadcast program, and

the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide).

8. The information processing device according to claim 1, wherein

the learning means

performs dimension reduction to reduce a dimension of the image feature amount and the text feature amount, and

learns the annotation model by using the multi-stream including the image feature amount and the text feature amount after the dimension reduction as the annotation sequence.

9. The information processing device according to claim 8, wherein the learning means

obtains basis space data of a basis space for image of which dimension is lower than the dimension of the image feature amount for mapping the image feature amount by using the image feature amount,

performs the dimension reduction of the image feature amount based on the basis space data of the basis space for image,

obtains basis space data of a basis space for text of which dimension is lower than the dimension of the text feature amount for mapping the text feature amount by using the text feature amount, and

performs the dimension reduction of the text feature amount based on the basis space data of the basis space for text.

10. The information processing device according to claim 9, wherein

the learning means

obtains a code book used for vector quantization as the basis space data of the basis space for image by using the image feature amount, and

obtains a code representing a centroid vector as the image feature amount after the dimension reduction by performing the vector quantization of the image feature amount by using the code book.

11. The information processing device according to claim 9, wherein

the learning means

extracts words included in the text of the caption displayed in a window as one document while shifting the window of a predetermined time length at regular intervals,

extracts a frequency of appearance of each word in the document as the text feature amount,

obtains a parameter of LDA (latent Dirichlet allocation) as the basis space data of the basis space for text by learning the LDA by using the document obtained from the learning content, and

converts the text feature amount obtained from the document to topic likelihood, which is likelihood of each latent topic of the LDA for the document, by using the parameter of the LDA, to obtain a topic label representing the latent topic of which topic likelihood is the maximum as the text feature amount after the dimension reduction.

12. The information processing device according to claim 11, wherein

the learning means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the browsing controlling means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest,

selects a word with high appearance frequency in the frame topic as the annotation to be added to the frame of interest based on the topic-to-frequently appearing word table,

extracts a group of one or more temporally continuous frames to which the same annotation is added as the scene from the target content, and

displays the representative images of the scenes so as to be arranged in chronological order.

13. The information processing device according to claim 12, wherein

the target content is content of a broadcast program, and

the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in a program listing of the broadcast program on an EPG (electronic program guide) in chronological order.

14. The information processing device according to claim 12, wherein

the browsing controlling means also displays the annotation added to the frame, which composes the scene, together with a representative image of the scene.

15. The information processing device according to claim 12, wherein

the learning means

generates a word dictionary of the words appearing in the document by using the document obtained from the learning content and creates a topic-to-frequently appearing word table of a word with high appearance frequency in the latent topic of the LDA and the appearance frequency of the word by using occurrence probability of occurrence of each word in the word dictionary in each latent topic of the LDA obtained by learning the LDA, and

the browsing controlling means

extracts the image feature amount of each frame of the image of the target content, performs the dimension reduction, and composes the annotation sequence by using the image feature amount after the dimension reduction,

obtains a maximum likelihood state sequence in which the annotation sequence is observed in the annotation model,

selects the latent topic represented by the topic label with high output probability in a state corresponding to a noted frame of interest out of states of the maximum likelihood state sequence as a frame topic representing a content of the frame of interest,

obtains an appearance frequency of a predetermined keyword in the frame topic based on the topic-to-frequently appearing word table and selects, when the appearance frequency of the predetermined keyword is high, the frame of interest as a keyword frame, which is a frame of which content coincides with the predetermined keyword,

extracts a group of one or more temporally continuous frames from the keyword frame as the scene, and

displays the representative images of the scenes so as to be arranged in chronological order.

16. The information processing device according to claim 15, wherein

the target content is content of a broadcast program, and

the browsing controlling means displays the representative images of the scenes of the broadcast program so as to be arranged in chronological order in a program listing of the broadcast program on an EPG (electronic program guide).

17. An information processing method to be performed by an information processing device, comprising the steps of:

extracting an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,

wherein the learning content includes a text of a caption, and

wherein the description text is the text of the caption included in the learning content;

learning an annotation models which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount;

extracting a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model; and

displaying representative images of scenes so as to be arranged in chronological order.

18. A non-transitory computer-readable medium having a set of computer-executable instructions embodied thereon to perform a method in a computing device comprising:

extract an image feature amount of each frame of an image of learning content and extracting word frequency information regarding a frequency of appearance of each word in a description text describing a content of the image of the learning content as a text feature amount of the description text,

wherein the learning content includes a text of a caption, and wherein the description text is the text of the caption included in the learning content;

learn an annotation model, which is a multi-stream HMM (Hidden Markov Model), by using an annotation sequence for annotation, which is a multi-stream including the image feature amount and the text feature amount;

extract a scene, which is a group of one or more temporally continuous frames, from target content from which the scene is to be extracted by using the annotation model; and

display representative images of scenes so as to be arranged in chronological order.