Image displaying apparatus

• US 7,015,934 B2
• Filed: 11/05/2001
• Issued: 03/21/2006
• Est. Priority Date: 11/08/2000
• Status: Active Grant

• Current Assignee: Minolta Corporation Limited (Konica Minolta Inc.)
• Original Assignee: Minolta Corporation Limited (Konica Minolta Inc.)
• Inventors: Toyama, Osamu, Fujii, Eiro, Tanaka, Yuzuru
• Primary Examiner(s): YANG, RYAN R
• Application Number:
  US09/985,546
• Publication Number:
  US 20020054047A1
• Time in Patent Office: 1,597 Days
• Field of Search: 345/629, 345/619, 345/733, 345/751, 345/716, 345/753, 345/759, 345/756, 345/757, 345/798, 725/135, 370/260, 348/14.09
• US Class Current: 345/629
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

The present invention aims at realizing displaying of a realistic image of a conversation scene in which a speaker can be visually recognized by people watching this image regardless of the content of the image. To this end, according to the present invention, a two-dimensional or three-dimensional face model is deformed, and animations A1_i through A3_i which express a state in which a person is speaking are consequently created and displayed as auxiliary images.

Claims

1. An actually shot image displaying apparatus comprising:

a first displaying portion at which an actually shot image is displayed;

a voice information obtainer for obtaining voice information;

an image creator for creating an animated face image which expresses appearances of speech by a person by means of deforming a two-dimensional face model or a three-dimensional face model in accordance with the obtained voice information; and

a second displaying portion at which a plurality of animated face images are displayed and laid out as an auxiliary image including the animated face image which expresses appearances of speech by a person, the plurality of animated face images correspond to persons in the actually shot image, wherein

the first displaying portion and second displaying portion are arranged on a same display screen, and the animated face image, which expresses appearances of speech by a person and corresponds to a person who is speaking, is displayed distinctively from the other animated face images; and

when a person without an associated face model Mp registered is included in those who are to be displayed, a texture extracted from a main image may be pasted to a standard model to thereby create a face model of this person.

2. The image displaying apparatus according to claim 1, wherein a unit including the first displaying portion and a unit including the second displaying portion are different units.

3. The image displaying apparatus according to claim 1, wherein the first displaying portion and the second displaying portion are included in a unit.

4. The image displaying apparatus according to claim 1, wherein the animated face image expresses a full-face pose.

5. The image displaying apparatus according to claim 1, wherein there is a two-dimensional face model or the three-dimensional face model corresponds to each person represented in the actually shot image.

6. An actually shot image displaying apparatus comprising;

a first displaying portion at which an actually shot image is displayed;

a voice information obtainer for obtaining voice information;

an image creator for creating an animated face image which expresses appearances of speech by a person by means of deforming a two-dimensional face model or a three-dimensional face model in accordance with the obtained voice information;

a second displaying portion at which a plurality of animated face images are displayed and laid out as an auxiliary image including the animated face image which expresses appearances of speed by a person, the plurality of animated face images correspond to persons in the actually shot image; and

a determiner for determining whether a direction of a person represeted in the actually shot image is full-face pose or not, wherein

a first displaying portion and second displaying portion are arranged on a same display screen, and the animated face image, which expresses appearances of speech by a person and corresponds to a person who is speaking, is displayed distinctively from the other animated face images, and

the second displaying portion displays an animated face image of a person which direction is determined as a not full-face pose by the determiner and does not display an animated face of a person which direction is determined as a full-face pose as the actually shot image will sufficiently show facial expressions when the person is speaking.

7. An image creating system comprising:

an image acquiring portion for acquiring image signals representing an actually shot image;

a voice acquiring portion for acquiring voice signals for representing voice accompanying the actually shot image;

a model storing memory for storing a plurality of models;

a specifying processor for specifying a model from the plurality of models stored in the model storing memory on the basis of the acquired voice signals;

a memory controller for read out of the specified model from the model storing memory;

an image creating processor for creating an animated face image by deforming the read out model; and

a display including

a first displaying portion for displaying the image signals representing the actually shot image, and

a second displaying portion for displaying, as an auxiliary image, a plurality of animated face images including the animated face image created by deforming the read out model, the plurality of animated face images correspond to persons in the actually shot image, wherein

the first displaying portion and second displaying portion are arranged together on the display, and the animated face image created by deforming the read out model, which corresponds to a person who is speaking, is displayed distinctively from the other animated face images; and

when a person without an associated face model Mp registered is included in those who are to be displayed, a texture extracted from a main image may be pasted to a standard model to thereby create a face model of this person.

8. The image creating system according to claim 7, further comprising a voice feature storing memory for storing a plurality of voice features respectively corresponding to the plurality of models, wherein

the specifying processor refers the voice features to specify a model.

9. The image creating system according to claim 7, wherein

the voice signals include voice data and data for specifying a speaker, and

the specifying processor refers the data for specifying a speaker to specify a model corresponding to the voice data.

10. The image creating system according to claim 9, wherein the data for specifying a speaker is obtained from an apparatus for creating the actually shot image.

11. The image creating system according to claim 9, wherein

the voice signals comprise multiple channels, and

the data for specifying a speaker is one channel of the multiple channels.

12. The image creating system according to claim 9, wherein the display displays the plurality of animated face images over the actually shot image, and a location of each displayed animated face image is unrelated to the location of the person in the actually shot image that said each displayed animated face image corresponds to.

16. The image creating system according to claim 7, wherein the specifying processor specifies a plurality of models, and the display displays a plurality of animated face images corresponding to the specified plurality of models.

17. The image creating system according to claim 7, wherein the auxiliary image creating processor creates the animated face image by modifying the model based on voice information.

13. An actually shot image displaying system for displaying a plurality of persons, the system comprising:

a main image displaying portion for displaying an actually shot image which represents positional relationship of a plurality of persons;

a speaker for producing sounds which represents voice of the plurality of persons;

an auxiliary image displaying portion which has a plurality of frames each of which displays an animated full-face pose image of a person, the animated full-face pose image in each of the plurality of frames corresponds to a person in the actually shot image;

a determining processor for determining from which person of the plurality of the persons a voice is produced, on the basis of the voice to be represented by the speaker;

a creating processor for creating an animated full-face pose image of a person determined by the determining processor as the person from which the voice is produced; and

a display controller for controlling the auxiliary image displaying portion to display the created animated full-face pose image within one frame of the plurality of frames, wherein

the main image displaying portion and the auxiliary image displaying are arranged on a same display screen and the created animated full-face pose image of the person determined as the person from which the voice is produced is displayed distinctively from the other animated full-face pose images; and

when a person without an associated face model Mp registered is included in those who are to be displayed, a texture extracted from a main image may be pasted to a standard model to thereby create a face model of this person.

14. The image displaying system according to claim 13, wherein the one frame with the animated full-face pose image is predetermined for every person.

15. The image displaying system according to claim 13, wherein the display controller controls the auxiliary image displaying portion to display only the animated full-face pose image that is determined by the determining processor in a predetermined frame among the plurality of frames, and the actually shot image and/or other images are displayed where the plurality of frames are not displayed.

18. An actually shot image displaying apparatus, comprising:

a visual information input section which obtains visual information by actually shooting an image with a camera;

an audio information input section which obtain audio information;

an image generator which selects a face model stored in a memory, and deforms the face model selected based on the audio information to create an animated face image; and

a controller which controls to display on a display device an image of the actually shot image together with an auxiliary image having a plurality of animated face images including the animated face image created by the image generator, the plurality of animated face images correspond to persons in the actually shot image, wherein

the animated face image created by the image generator, which corresponds to a person who is speaking, is displayed distinctively from the other animated face images; and

when a person without an associated face model Mp registered is included in those who are to be displayed, a texture extracted from a main image may be pasted to a standard model to thereby create a face model of this person.

19. The image displaying apparatus according to claim 18, wherein the controller controls to display a plurality of deformed face model in a same screen of the display device.

20. The image displaying apparatus according to claim 18, wherein the controller controls to display a plurality of deformed face model adjacent mutually in a same screen of the display device.