Method and system for generating objects for a multi-person virtual world using data flow networks

• US 5,588,139 A
• Filed: 10/08/1993
• Issued: 12/24/1996
• Est. Priority Date: 06/07/1990
• Status: Expired due to Term

• Current Assignee: VPL Newco, Inc.
• Original Assignee: VPL Research, Inc.
• Inventors: Teitel, Michael A., Oberman, Mark L., Blanchard, Chuck L., Lanier, Jaron Z., Harvill, Young L., Grimaud, Jean-Jacques G., Lasko-Harvill, Ann
• Primary Examiner(s): Treat, William M.
• Application Number:
  US08/133,802
• Time in Patent Office: 1,173 Days
• Field of Search: 395/119, 395/159, 395/500, 364/709.01, 345/7, 345/8, 345/9
• US Class Current: 703/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A computer model of a virtual environment is continuously modified by input from various participants. The virtual environment is displayed to the participants using sensory displays such as head-mounted visual and auditory displays which travel with the wearer and track the position and orientation of the wearer'"'"'s head in space. Participants can look at each other within the virtual environment and see virtual body images of the other participants in a manner similar to the way that people in a physical environment see each other. Each participant can also look at his or her own virtual body in exactly the same manner that a person in a physical environment can look at his or her own real body. The participants may work on a common task together and view the results of each other'"'"'s actions.

Claims

1. A simulating apparatus comprising:

modeling means for creating a model of a physical environment in a computer database;

first body sensing means, disposed in close proximity to a part of a first body, for sensing a physical status of the first body part relative to a first reference position;

second body sensing means, disposed in close proximity to a part of a second body, for sensing a physical status of the second body part relative to a second reference position;

first body emulating means, coupled to the first body sensing means, for creating a first cursor in the computer database, the first cursor including plural first cursor nodes and emulating the physical status of the first body part, the first body emulating means including a first point hierarchy and a first data flow network, the first point hierarchy for controlling a shape and an orientation of the first cursor and for attaching each of the plural first cursor nodes hierarchically with at least one other of the plural first cursor nodes, the first data flow network for controlling motion of the first cursor and the first data flow network including a first interconnection of first input units, first function units and first output units, the first input unity receiving the physical status of the first body part, each first function unit including at least one input and at least one output and calculating, based on the at least one input, a value for each of the at least one output, and the first output units for producing position and orientation values for a portion of the plural first cursor nodes;

first integrating means, coupled to the modeling means and to the first emulating means, for integrating the first cursor with the model;

second body emulating means, coupled to the second body sensing means, for creating a second cursor in the computer database, the second cursor including plural second cursor nodes and emulating the physical status of the second body part, the second body emulating means including a second point hierarchy and a second data flow network, the second point hierarchy for controlling a shape and an orientation of the second cursor and for attaching each of the plural second cursor nodes hierarchically with at least one other of the plural second cursor nodes, the second data flow network for controlling motion of the second cursor and the second data flow network including a second interconnection of second input units, second function units and second output units, the second input units receiving the physical status of the second body part, each second function unit including at least one input and at least one output and calculating, based on the at least one input, a value for each of the at least one output, and the second output units for producing position and orientation values for a portion of the plural second cursor nodes; and

second integration means, coupled to the modeling means and to the second body emulating means, for integrating the second cursor with the model.

2. The apparatus according to claim 1 further comprising first model display means for displaying a view of the model.

3. The apparatus according to claim 2 wherein the first model display means includes view changing means for changing the view of the model in response to a change in the physical status of the second cursor in the model.

4. The apparatus according to claim 3 wherein the second cursor includes a first optical axis which moves together therewith, and wherein the view of the model produced by the first model display means corresponds to the view taken along the first optical axis.

5. The apparatus according to claim 4 wherein the first model display means displays the first cursor together with the model when the first optical axis faces the location of the first cursor.

6. The apparatus according to claim 5 wherein the first cursor depicts the first body part being emulated.

7. The apparatus according to claim 1 wherein the model includes a virtual object, and further comprising first object manipulating means, coupled to the first body emulating means, for manipulating the virtual object with the first cursor in accordance with corresponding gestures of the first body part.

8. The apparatus according to claim 7 further comprising second object manipulating means, coupled to the second body emulating means, for manipulating the virtual object with the second cursor in accordance with corresponding gestures of the second body part.

9. The apparatus according to claim 8 further comprising first model display means for displaying a view of the model.

10. The apparatus according to claim 9 wherein the first model display means includes view changing means for changing the view of the model in response to a change in the physical status of the second cursor in the model.

11. The apparatus according to claim 10 wherein the second cursor includes an optical axis which moves together therewith, and wherein the view of the model corresponds to the view taken along the optical axis.

12. The apparatus according to claim 11 wherein the first model display means displays the first cursor together with the model when the optical axis faces the location of the first cursor.

13. The apparatus according to claim 12 wherein the first cursor depicts the first body part being emulated.

14. The apparatus according to claim 13 wherein the first model display means displays the second cursor together with the model when the optical axis faces the location of the second cursor.

15. The apparatus according to claim 14 wherein the second cursor depicts the second body part being emulated.

16. The apparatus according to claim 15 further comprising second model display means for displaying a view of the model, the view of the model changing in response to the physical status of the first cursor in the model.

17. The apparatus according to claim 16 wherein the first cursor includes a second optical axis which moves together therewith, and wherein the view of the model produced by the second model display means corresponds to the view taken along the second optical axis.

18. The apparatus according to claim 17 wherein the second model display means displays the second cursor together with the model when the second optical axis faces the location of the second cursor.

19. The apparatus according to claim 18 wherein the first body part is a part of a body of a first human being.

20. The apparatus according to claim 19 wherein the first model display means comprises a first head-mounted display.

21. The apparatus according to claim 20 wherein the first head-mounted display comprises:

a first display for displaying the model to a first eye; and

a second display for displaying the model to a second eye.

22. The apparatus according to claim 1 wherein the first and second displays together produce a stereophonic image.

23. The apparatus according to claim 21 wherein the first head-mounted display further comprises:

a first audio display for displaying a sound model to a first ear; and

a second audio display for displaying the sound model to a second ear.

24. The apparatus according to claim 21 wherein the first and second displays display the model as a series of image frames, and wherein the model display means further comprises frame synchronization means, coupled to the first and second displays, for synchronizing the display of the series of frames to the first and second displays.

25. The apparatus according to claim 19 wherein the second body part is a part of a body of a second human being.

27. An apparatus according to claim 21, wherein the first body sensing means includes a facial expression sensor using conductive ink.

28. An apparatus according to claim 1, wherein the first body sensing means includes a facial expression sensor including a strain gauge.

29. An apparatus according to claim 1, wherein the first body sensing means includes a pneumatic input device.

26. A simulating apparatus comprising:

a modeling means for creating a virtual world model of a physical environment in a computer database;

a first sensor for sensing a first real world parameter;

first emulating means, coupled to the first sensor for emulating a first virtual world phenomenon in the virtual world model, the first emulating means including a first point hierarchy and a first data flow network, the first point hierarchy for controlling a shape and an orientation of a first cursor, including plural first cursor nodes, and for attaching each of the plural first cursor nodes hierarchically with at least one other of the plural first cursor nodes, the first data flow network for controlling motion of the first cursor and the first data flow network including a first interconnection of first input units, first function units and first output units, the first input units receiving the physical status of the first body part, each first function unit including at least one input and at least one output and calculating, based on the at least one input, a value for each of the at least one output, and the first output units for producing position and orientation values for a portion of the plural first cursor nodes;

a second sensor for sensing a second real world parameter; and

second emulating means, coupled to the second sensor, for emulating a second virtual world phenomenon in the virtual world model, the second emulating means including a second point hierarchy and a second data flow network, the second point hierarchy for controlling a shape and an orientation of a second cursor, including plural second cursor nodes, and for attaching each of the plural second cursor nodes hierarchically with at least one other of the plural second cursor nodes, the second data flow network for controlling motion of the second cursor and the second data flow network including a second interconnection of second input units, second function units and second output units, the second input units receiving the physical status of the second body part, each second function unit including at least one input and at least one output and calculating, based on the at least one input, a value for each of the at least one output, and the second output units for producing position and orientation values for a portion of the plural second cursor nodes.

30. A simulating method, comprising the steps of:

creating a virtual environment;

constructing virtual objects within the virtual environment using a point hierarchy and a data flow network for controlling motion of nodes of the virtual objects wherein the step of constructing includes

attaching each node of the virtual objects hierarchically with at least one other of the nodes to form the point hierarchy, each of the nodes of the virtual objects having a position and an orientation, and

building the data flow network as an interconnection of input units, function units and output units, wherein said input units receive data from sensors and output the received data to at least one of said function units, wherein each of said function units includes at least one input and at least one output, each function unit generating a value for the at least one output based on at least one of data received from at least one of the input units and data received from an output of at least one other of said function units, and wherein the output units generate the position and the orientation of a portion of the nodes of the virtual objects;

inputting data from sensors worn on bodies of at least two users;

converting the inputted data to position and orientation data;

modifying by using the data flow network, the position and the orientation of the nodes of the virtual objects based on the position and orientation data;

determining view points of said at least two users;

receiving a synchronization signal;

calculating image frames for each eye of each of said at least two users;

displaying the image frames to each of said eyes of said at least two users;

obtaining updated position and orientation values of said at least two users;

determining if the virtual environment has been modified;

redefining positions and orientations of the nodes of the virtual object if the virtual environment has been modified;

recalculating the image frames for each of said eyes of said at least two users; and

displaying the recalculated image frames to each of said eyes of said at least two users.