Method and system for 3-D shared virtual environment display communication virtual conference and programs therefor

• US 6,753,857 B1
• Filed: 04/13/2000
• Issued: 06/22/2004
• Est. Priority Date: 04/16/1999
• Status: Expired due to Fees

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Matsuura, Norihiko, Sugawara, Shohei
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/549,063
• Time in Patent Office: 1,531 Days
• Field of Search: 345/418, 345/419, 345/427, 345/420, 345/424, 345/426
• US Class Current: 345/419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

The purpose is to provide a system and a method for 3-D shared virtual environment communication system to enable a plurality of users to share a common virtual environment so that real-time facial expressions of other users can be displayed on user'"'"'s own terminal and voices heard. The system thus facilitate communications among the users in such cases as virtual conferencing environment. The method of operating the communication system is executed by computer application programs that contains a user information control section 102 broadcasts information input from the facial expression input section 103, voice input section 104 and motion information input section 105, to other user terminals through the network 30 under the control of the network control section 101. The avatar information managing section 107, script control section 108, virtual environment control section 109 and virtual environment generation section 110 displays other users'"'"' avatar on user'"'"'s own terminal.

Claims

1. A method for displaying a three-dimensional shared virtual environment comprising the steps of:

displaying a first display state of a virtual environment, generated by using a three-dimensional computer graphics technique, to be displayed in individual user terminals of a plurality of users so that a computer generated graphic object representing a subject user may be moved in accordance with instructions give by said subject user; and

displaying a second display state of said virtual environment so that computer generated graphic objects representing said plurality of users are always located in constant positions within said virtual environment.

2. A method according to claim 1, wherein said method further includes a step of selecting either said first display state or said second display state according to instructions given by a user so as to display a selected virtual environment on said user terminal.

3. A method according to claim 1, wherein said method further includes the steps of:

displaying pasted images of individual users by pasting corresponding real-time facial images on respective computer generated graphic objects representing said individual users so that, when displaying said second display state on a user terminal, front views of said computer generated graphic objects and corresponding pasted images are displayed.

4. A method according to claim 1, wherein a plurality of motion states for expressing intentions of a user represented by said computer generated graphic object are stored, and a motion state selected uniquely according to a user instruction is pasted onto said computer generated graphic object and a pasted image is displayed.

14. A method according to claim 2, wherein said method further includes the steps of:

displaying pasted images of individual users by pasting corresponding real-time facial images on respective computer generated graphic objects representing said individual users so that, when displaying said second display state on a user terminal, front views of said computer generated graphic objects and corresponding pasted images are displayed.

15. A method according to claim 2, wherein a plurality of motion states for expressing intentions of a user represented by said computer generated graphic object are stored, and a motion state selected uniquely according to a user instruction is pasted onto said computer generated graphic object and a pasted image is displayed.

16. A method according to claim 3, wherein a plurality of motion states for expressing intentions of a user represented by said computer generated graphic object are stored, and a motion state selected uniquely according to a user instruction is pasted onto said computer generated graphic object and a pasted image is displayed.

5. A three-dimensional shared communication system connecting a plurality of user terminals to central control means through communication lines, wherein each of said user terminals is comprised by:

image inputting means for inputting real-time images of a user;

voice audio inputting means for inputting voice data of said user;

motion information inputting means for inputting motion information of said user within a virtual environment described by three-dimensional computer graphics;

action indicating means for indicating actions of a computer generated graphic object representing said user to be displayed on terminal display means of other users;

simple program outputting means for outputting a pre-determined simple program according to an instruction given by said action indicating means;

user information input/output means for broadcasting received real-time image data, voice data, motion data and simple program data to other user terminals through said central control means, and receiving real-time image data, voice data, motion data and simple program data of said other users through said central control means;

voice audio reproduction means for reproducing voices of said other users received by said user information inputting/outputting means;

virtual environment control means for generating information required for displaying computer generated graphic objects representing said other users according to motion information and said simple program received by said user information input/output means from other user terminals, and controlling images displayed in such a way that said computer generated graphic objects are always displayed in assigned positions and facing frontward; and

virtual environment display means for pasting real-time facial images of said other users in said computer generated graphic objects according to information generated by said virtual environment control means and displaying a combined image of said virtual environment and said pasted images.

7. A virtual conferencing system for conducting a conference among users of said user terminals using one communication system chosen from said three-dimensional shared virtual environment communication systems according to claim 5.

6. A three-dimensional shared communication system connecting a plurality of user terminals to central control means through communication lines, wherein said central control means stores information defining a plurality of types of virtual environments with individual addresses, and having at least one information defining a virtual conferencing environment;

and

each of said plurality of user terminals is comprised by;

address specifying means for specifying addresses of information defining said virtual environments;

image inputting means for inputting real-time images of a user;

voice audio inputting means for inputting voice data of said user;

motion information inputting means for inputting motion information of said user within a virtual environment described by three-dimensional computer graphics;

action indicating means for indicating actions of a computer generated graphic object representing said user to be displayed on terminal display means of other users;

simple program outputting means for outputting a pre-determined simple program according to an instruction given by said action indicating means;

user information input/output means for broadcasting received real-time image data, voice data, motion data and simple program data to other user terminals through said central control means, and receiving real-time image data, voice data, motion data and simple program data of said other users through said central control means;

voice audio reproduction means for reproducing voices of said other users received by said user information inputting/outputting means;

virtual environment control means for generating information required for displaying said virtual environment according to information defining said virtual environment and motion information received by said user information input/output means from other user terminals logging in to said virtual environment, and generating information required to display computer generated graphic objects representing said other users according to motion information and said simple program received;

virtual environment display means for pasting real-time facial images of said other users in said computer generated graphic object according to information generated by said virtual environment control means, and displaying a combined image of said virtual environment and said pasted images, and wherein,

when an address of information defining said virtual conferencing environment is specified by said address specifying means,

said virtual environment control means control displays so that said computer generated graphic objects always remain in assigned locations and facing frontward.

17. A virtual conferencing system for conducting a conference among users of said user terminals using one communication system chosen from said three-dimensional shared virtual environment communication systems according to claim 6.

8. A method for communicating among a plurality of user terminals in a three-dimensional virtual environment shared through communication lines connected to each user terminal so as to exchange data between any one user terminal and other user terminals through central control means, comprising the steps of:

inputting real-time images of a user of one user terminal produced by an imaging device provided in each user terminal;

inputting voice of said user obtained by a sound recording device provided in each user terminal;

inputting motion information of said user within said virtual environment described by three-dimensional computer graphics produced by a motion information input device provided in each user terminal;

inputting action instructions of a computer generated graphic object, representing said user to be displayed in other user terminals;

outputting a pre-determined simple program from said central control means in response to motion instructions input from said one user terminal;

broadcasting real-time facial image data, voice audio data, motion information data, and said simple program, output from said one user terminal, to other user terminals through said central control means, and receiving real-time facial image data, voice audio data, motion information data of other users and said simple program, output from said other user terminals, in said one user terminal through said central control means;

reproducing voices of other users received from said other user terminals at said one user terminal;

generating information required for displaying computer generated graphic objects representing said other users using said user information input/output means according to motion information and said simple program, and controlling image orientations in such a way that said computer generated graphic objects are always displayed in assigned locations and facing frontward in said virtual conferencing environment; and

pasting real-time facial images of said other users in said computer generated graphic objects according to generated information and displaying a combined image of said virtual environment and said pasted images.

9. A method according to claim 8, wherein said method further comprises the steps of:

obtaining information related to users participating in said virtual environment being managed by said central control means;

judging whether there has been a change in a number of users participating, based on information obtained; and

altering computer generated graphic objects to be displayed in said virtual environment according to judged results.

10. A method for communicating among a plurality of user terminals in a three-dimensional virtual environment shared through communication lines connected to each user terminal so as to exchange data between any one user terminal and other user terminals through central control means, comprising the steps of:

inputting real-time images of a user of one user terminal produced by an imaging device provided in each user terminal;

inputting voice of said user obtained by a sound recording device provided in each user terminal;

inputting motion information of said user within said virtual environment described by three-dimensional computer graphics produced by a motion information input device provided in each user terminal;

inputting action instructions of a computer generated graphic object, representing said user to be displayed in other user terminals;

outputting a pre-determined simple program from said central control means in response to motion instructions input from said one user terminal;

broadcasting real-time facial image data, voice audio data, motion information data, and said simple program, output from said one user terminal, to other user terminals through said central control means, and receiving real-time facial image data, voice audio data, motion information data of other users and said simple program, output from said other user terminals, in said one user terminal through said central control means;

reproducing voices of other users received from said other user terminals at said one user terminal;

specifying addresses to correspond to at least one of a plurality of types of virtual environments, including a virtual conferencing environment, identified by own addresses stored in said central control means;

generating information necessary for displaying a relevant virtual environment according to information defining said relevant virtual environment identified by a specified address and motion information received from other user terminals logging in to said relevant virtual environment, and generating information necessary for displaying computer generated graphic objects representing said other users according to received information and said simple program;

pasting real-time facial images of said other users in said computer generated graphic objects according to generated information and displaying a combined image of said virtual environment and said pasted images; and

controlling orientation of images in such a way that said computer generated graphic objects are always displayed in assigned positions and facing frontward in said virtual conferencing environment.

11. A computer-readable recorded medium containing an application program for executing a method of communicating in a three-dimensional shared virtual environment using each of a plurality of user terminals, connected to central control means through communication lines, by means of a common virtual environment shared by said plurality of user terminals, comprising the steps of:

inputting real-time images of a user of one user terminal produced by an imaging device provided in each user terminal;

inputting voice of said user obtained by a sound recording device provided in each user terminal;

inputting motion information of said user within said virtual environment described by three-dimensional computer graphics produced by a motion information input device provided in each user terminal;

inputting action instructions of a computer generated graphic object, representing said user to be displayed in other user terminals;

outputting a pre-determined simple program from said central control means in response to motion instructions input from said one user terminal;

broadcasting real-time facial image data, voice audio data, motion information data, and said simple program, output from said one user terminal, to other user terminals through said central control means, and receiving real-time facial image data, voice audio data, motion information data of other users and said simple program, output from said other user terminals, in said one user terminal through said central control means;

reproducing voices of other users received from said other user terminals at said one user terminal;

generating information required for displaying computer generated graphic objects representing said other users using said user information input/output means according to motion information and said simple program, and controlling image orientations in such a way that said computer generated graphic objects are always displayed in assigned locations and facing frontward in said virtual conferencing environment; and

pasting real-time facial images of said other users in said computer generated graphic objects according to generated information and displaying a combined image of said virtual environment and said pasted images.

12. A computer-readable recorded medium according to claim 11, wherein said application program further contains the steps of:

obtaining information related to users participating in said virtual environment being managed by said central control means;

judging whether there has been a change in a number of users participating, based on information obtained; and

altering computer generated graphic objects to be displayed in said virtual environment according to judged results.

13. A computer-readable recorded medium containing an application program for executing a method of communicating in a three-dimensional shared virtual environment using each of a plurality of user terminals, connected to central control means through communication lines, by means of a common virtual environment shared by said plurality of user terminals, comprising the steps of:

inputting real-time images of a user of one user terminal produced by an imaging device provided in each user terminal;

inputting voice of said user obtained by a sound recording device provided in each user terminal;

inputting motion information of said user within said virtual environment described by three-dimensional computer graphics produced by a motion information input device provided in each user terminal;

inputting action instructions of a computer generated graphic object, representing said user to be displayed in other user terminals;

outputting a pre-determined simple program from said central control means in response to motion instructions input from said one user terminal;

broadcasting real-time facial image data, voice audio data, motion information data, and said simple program, output from said one user terminal, to other user terminals through said central control means, and receiving real-time facial image data, voice audio data, motion information data of other users and said simple program, output from said other user terminals, in said one user terminal through said central control means;

reproducing voices of other users received from said other user terminals at said one user terminal;

specifying addresses to correspond to at least one of a plurality of types of virtual environments, including a virtual conferencing environment, identified by own addresses stored in said central control means;

generating information necessary for displaying a relevant virtual environment according to information defining said relevant virtual environment identified by a specified address and motion information received from other user terminals logging in to said relevant virtual environment, and generating information necessary for displaying computer generated graphic objects representing said other users according to received information and said simple program;

pasting real-time facial images of said other users in said computer generated graphic objects according to generated information and displaying a combined image of said virtual environment and said pasted images; and

controlling orientation of images in such a way that said computer generated graphic objects are always displayed in assigned positions and facing frontward in said virtual conferencing environment.