System and method for operating in a virtual environment

• US 7,646,394 B1
• Filed: 03/07/2005
• Issued: 01/12/2010
• Est. Priority Date: 03/05/2004
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: Raytheon Systems Limited (Rtx Corporation), HRL Laboratories LLC (The Boeing Co.)
• Inventors: Jarald, Jason, Shomphe, Matt, Neely, Howard III, Kolsch, Mathias, Fox, Jason, Daily, Mike
• Primary Examiner(s): Yang; Ryan R
• Application Number:
  US11/075,354
• Time in Patent Office: 1,772 Days
• Field of Search: 345/419, 345/633, 715/863
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

The present invention relates to a system, method and computer program product for enabling user interaction with objects in a virtual environment independently of apparent virtual viewpoint altitude, by non-linearly scaling the virtual actuator. In doing so, the system receives a virtual-viewpoint position and a virtual actuator position from a virtual environment processing subsystem, and a real-viewpoint position and a real-actuator position from a real-world environment tracking subsystem. An xy-scale factor is then calculated based on the virtual-viewpoint position. A non-linear mapping is thereafter calculated between a real dataset and a virtual dataset based on the xy-scale-factor. The real dataset comprises the real-actuator position and the real-viewpoint position in the real-world environment, and the virtual dataset comprises the virtual-actuator position and the virtual-viewpoint position in the virtual environment.

Claims

1. A system for enabling user interaction with objects in a virtual environment independently of apparent virtual viewpoint altitude, by non-linearly scaling the virtual actuator, the system comprising a processor and a memory coupled with the processor, wherein the memory includes means that are executable by the processor for causing the processor to perform operations of:

receiving input, at a graphics subsystem, from a virtual environment processing subsystem and a real-world environment tracking subsystem, where the virtual environment processing subsystem provides a virtual-viewpoint position and a virtual actuator position, and the real-world environment tracking subsystem provides a real-viewpoint position and a real-actuator position, the graphics subsystem performing operations of;

receiving, at a xy-scale factor module, the virtual-viewpoint position and outputting, from the xy-scale factor module, a xy-scale-factor based on the virtual-viewpoint position; and

receiving, at a non-linear mapping module, the virtual-viewpoint position, the virtual-actuator position, the real-viewpoint position, the real-actuator position, and the xy-scale-factor and generating, at the non-linear mapping module, a non-linear mapping between a real dataset and a virtual dataset, wherein the non-linear mapping is based on the xy-scale-factor, and wherein the real dataset comprises the real-actuator position and the real-viewpoint position in the real-world environment, and the virtual dataset comprises the virtual-actuator position and the virtual-viewpoint position in the virtual environment.

2. A system for enabling user interaction with objects in a virtual environment as set forth in claim 1, wherein the non-linear mapping comprises coordinates for a new virtual actuator position, and the coordinates are sent to the virtual environment processing subsystem, where the virtual environment processing subsystem calculates a new virtual actuator position based on the coordinates.

3. A system for enabling user interaction with objects in a virtual environment as set forth in claim 2, wherein the graphics subsystem further performs operations of receiving, at a z-scale factor module, the new virtual-actuator position and outputting, from the z-scale factor module, a z-scale-factor based on the new virtual-actuator position, and wherein the non-linear mapping module performs an operation of generating a non-linear mapping between the real dataset and the virtual dataset based on the xy-scale-factor and the z-scale factor.

4. A system for enabling user interaction with objects in a virtual environment as set forth in claim 3, wherein the xy-scale factor module performs an operation of calculating a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator.

5. A system for enabling user interaction with objects in a virtual environment as set forth in claim 4, wherein the non-linear mapping module performs an operation of setting a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.

6. A system for enabling user interaction with objects in a virtual environment as set forth in claim 5, wherein the actuator is selected from a group consisting of a user'"'"'s hand, a virtual craft, a cursor, a joystick, a computer mouse, and a pointer.

7. A system for enabling user interaction with objects in a virtual environment as set forth in claim 1, wherein the xy-scale factor module performs an operation of calculating a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator,

and wherein the non-linear mapping module performs an operation of setting a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.

8. A computer implemented method for enabling user interaction with objects in a virtual environment independently of apparent virtual viewpoint altitude, by non-linearly scaling the virtual actuator, the method comprising an act of causing a processor to perform operations of:

receiving a virtual-viewpoint position and a virtual actuator position from a virtual environment processing subsystem, and receiving a real-viewpoint position and a real-actuator position from a real-world environment tracking subsystem;

calculating a xy-scale factor based on the virtual-viewpoint position; and

calculating a non-linear mapping between a real dataset and a virtual dataset based on the xy-scale-factor, and wherein the real dataset comprises the real-actuator position and the real-viewpoint position in the real-world environment, and the virtual dataset comprises the virtual-actuator position and the virtual-viewpoint position in the virtual environment.

9. A method for enabling user interaction with objects in a virtual environment as set forth in claim 8, wherein in the operation of calculating a non-linear mapping, the non-linear mapping comprises coordinates for a new virtual actuator position, and the coordinates are sent to the virtual environment processing subsystem, where the virtual environment processing subsystem calculates a new virtual actuator position based on the coordinates.

10. A method for enabling user interaction with objects in a virtual environment as set forth in claim 9, further comprising an operation of calculating a z-scale factor based on the new virtual-actuator position, and wherein the operation of calculating a non-linear mapping calculates a non-linear mapping between the real dataset and the virtual dataset based on the xy-scale-factor and the z-scale factor.

11. A method for enabling user interaction with objects in a virtual environment as set forth in claim 10, wherein the operation of calculating a xy-scale factor calculates a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator.

12. A method for enabling user interaction with objects in a virtual environment as set forth in claim 11, wherein the operation of calculating a non-linear mapping sets a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.

13. A method for enabling user interaction with objects in a virtual environment as set forth in claim 12, wherein the actuator is selected from a group consisting of a user'"'"'s hand, a virtual craft, a cursor, a joystick, a computer mouse, and a pointer.

14. A method for enabling user interaction with objects in a virtual environment as set forth in claim 8, wherein the operation of calculating a xy-scale factor calculates a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator,

and wherein the operation of calculating a non-linear mapping sets a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.

15. A computer program product for enabling user interaction with objects in a virtual environment independently of apparent virtual viewpoint altitude, by non-linearly scaling the virtual actuator, the computer program product comprising computer-readable means stored on a computer readable medium that are executable by a computer having a processor for causing the processor to perform the operations of:

receiving a virtual-viewpoint position and a virtual actuator position from a virtual environment processing subsystem, and receiving a real-viewpoint position and a real-actuator position from a real-world environment tracking subsystem;

calculating a xy-scale factor based on the virtual-viewpoint position; and

calculating a non-linear mapping between a real dataset and a virtual dataset based on the xy-scale-factor, and wherein the real dataset comprises the real-actuator position and the real-viewpoint position in the real-world environment, and the virtual dataset comprises the virtual-actuator position and the virtual-viewpoint position in the virtual environment.

16. A computer program product for enabling user interaction with objects in a virtual environment as set forth in claim 15, wherein in the operation of calculating a non-linear mapping, the non-linear mapping comprises coordinates for a new virtual actuator position, and the coordinates are sent to the virtual environment processing subsystem, where the virtual environment processing subsystem calculates a new virtual actuator position based on the coordinates.

17. A computer program product for enabling user interaction with objects in a virtual environment as set forth in claim 16, further comprising an operation of calculating a z-scale factor based on the new virtual-actuator position, and wherein the operation of calculating a non-linear mapping calculates a non-linear mapping between the real dataset and the virtual dataset based on the xy-scale-factor and the z-scale factor.

18. A computer program product for enabling user interaction with objects in a virtual environment as set forth in claim 17, wherein the operation of calculating a xy-scale factor calculates a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator.

19. A computer program product for enabling user interaction with objects in a virtual environment as set forth in claim 18, wherein the operation of calculating a non-linear mapping sets a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.

20. A computer program product for enabling user interaction with objects in a virtual environment as set forth in claim 15, wherein the operation of calculating a xy-scale factor calculates a vertical distance of the virtual-viewpoint position to a virtual ground plane and calculates the xy-scale factor by

$\mathrm{xy}$

- ⁢

scale ⁢

- ⁢

factor = maxReachDistance - armLength armLength 2

where

maxReachDistance denotes twice the vertical distance of the virtual-viewpoint position to a virtual ground plane; and

armLength denotes maximum distance between the real-viewpoint position and the real-actuator position, when a user fully extends the real-actuator,

and wherein the operation of calculating a non-linear mapping sets a threshold such that the non-linear mapping is generated when the vertical distance of the virtual-viewpoint position to a virtual ground plane is greater than the distance between the real-viewpoint position and the real-actuator position.