Image processing device, animation display method and computer readable medium

• US 10,304,439 B2
• Filed: 12/22/2016
• Issued: 05/28/2019
• Est. Priority Date: 03/16/2016
• Status: Active Grant

• Current Assignee: Casio Computer Company Limited
• Original Assignee: Casio Computer Company Limited
• Inventors: Okaniwa, Shoichi, Negishi, Hiroaki, Moriya, Shigekatsu, Kanda, Hirokazu
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US15/388,053
• Publication Number:
  US 20170270701A1
• Time in Patent Office: 887 Days
• Field of Search: None
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An image processing device includes a controller and a display. The controller adds an expression to a displayed face image in accordance with an audio when the audio is output. Further, the controller generates an animation in which a mouth contained in the face image with the expression moves in sync with the audio. The display displays the generated animation.

Claims

1. An image processing device comprising:

a processor configured to;

detect words or phrases within text of a sentence or a clause, wherein the text corresponds to audio to be reproduced;

determine, for at least one of the words or phrases detected within the text of the sentence or the clause, a corresponding one of a plurality of word/phrase-expressions;

determine that at least one of the words or phrases within the text of the sentence or the clause is a context-dependent word/phrase-expression;

assign a most frequent one of the word/phrase-expression determined for the at least one of the words or phrases detected, while ignoring the context-dependent word/phrase-expression determined, as one of a plurality of sentence/clause-expressions to the text of the sentence or the clause; and

generate frames of animation of a face of increased expressiveness to be displayed in sync with a reproduction of the audio, by at least performing;

generate a mouth shape of a mouth of the face for each of the frames based on the words or phrases detected within the text; and

generate an emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned to the text of the sentence or the clause.

2. The image processing device according to claim 1,

wherein the processor is configured to:

determine whether to generate the animation in a different language from that of the audio; and

in response to determining to generate the animation in the different language from that of the audio;

generate a mouth shape of the mouth of the face for each of the frames based on words or phrases detected within a text of audio in the different language; and

generate the emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned.

3. A method comprising:

detecting words or phrases within text of a sentence or a clause, wherein the text corresponds to audio to be reproduced;

determining, for at least one of the words or phrases detected within the text of the sentence or the clause, a corresponding one of a plurality of word/phrase expressions;

determining that at least one of the words or phrases within the text of the sentence or the clause is a context-dependent word/phrase-expression;

assigning a most frequent one of the word/phrase-expression determined for the at least one of the words or phrases detected, while ignoring the context-dependent word/phrase-expression determined, as one of a plurality of sentence/clause-expressions to the text of the sentence or the clause; and

generating frames of animation of a face of increased expressiveness to be displayed in sync with a reproduction of the audio, by at least;

generating a mouth shape of a mouth of the face for each of the frames based on the words or phrases detected within the text; and

generating an emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned to the text of the sentence or the clause.

4. The method according to claim 3, comprising:

determining whether to generate the animation in a different language from that of the audio; and

in response to determining to generate the animation in the different language from that of the audio;

generating a mouth shape of the mouth of the face for each of the frames based on words or phrases detected within a text of audio in the different language; and

generating the emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned.

5. A non-transitory computer readable storage medium storing a program to cause a computer to at least perform:

detecting words or phrases within text of a sentence or a clause, wherein the text corresponds to audio to be reproduced;

determining, for at least one of the words or phrases detected within the text of the sentence or the clause, a corresponding one of a plurality of word/phrase-expressions;

determining that at least one of the words or phrases within the text of the sentence or the clause is a context-dependent word/phrase-expression;

assigning a most frequent one of the word/phrase-expression determined for the at least one of the words or phrases detected, while ignoring the context-dependent word/phrase-expression determined, as one of a plurality of sentence/clause-expressions to the text of the sentence or the clause; and

generating frames of animation of a face of increased expressiveness to be displayed in sync with a reproduction of the audio, by at least;

generating a mouth shape of a mouth of the face for each of the frames based on the words or phrases detected within the text; and

generating an emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned to the text of the sentence or the clause.

6. The non-transitory computer readable storage medium according to claim 5, wherein the program causes the computer to perform:

determining whether to generate the animation in a different language from that of the audio; and

in response to determining to generate the animation in the different language from that of the audio;

generating a mouth shape of the mouth of the face for each of the frames based on words or phrases detected within a text of audio in the different language; and

generating the emotional expression of the face for each of the frames based on the one of the plurality of sentence/clause-expressions assigned.