Method and system for hybrid trackball and immersive navigation in a virtual environment

• US 7,893,935 B1
• Filed: 11/30/2009
• Issued: 02/22/2011
• Est. Priority Date: 03/05/2004
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Neely, Howard III, Shomphe, Matthew, Kolsch, Mathias, Jerald, Jason, Fox, Jason
• Primary Examiner(s): Yang; Ryan R
• Application Number:
  US12/592,686
• Time in Patent Office: 449 Days
• Field of Search: 345/419, 345/420, 345/633
• US Class Current: 345/419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

The present invention relates to a system, method, and computer program product for immersive navigation in a virtual environment (VE) suitable for allowing a user to change a view orientation in the VE independently of physical orientation of a user input, such as orientation of the user'"'"'s head. Specifically, the present invention combines three distinct virtual reality navigation metaphors (trackball navigation, grab navigation, and immersive navigation) into a hybrid navigation approach generating a final virtual-viewpoint that correlates to the characteristics of an intermediate virtual-viewpoint defined by radius, pitch, and heading of a virtual reference sphere (as in trackball navigation), while the final virtual-viewpoint also adjusts the location and gaze direction of the intermediate virtual-viewpoint based on pitch, heading, and roll of the user'"'"'s head (as in immersive navigation), thus allowing the user to view a point of interest in the VE while simultaneously facing comfortably forward in the physical world.

Claims

1. A system for navigating in a virtual environment and suitable for allowing a user to change a view orientation in the virtual environment independently of physical orientation of a user input, the system comprising a processor and a memory coupled with the processor, wherein the memory includes means that are executable by the processor for causing the processor to perform operations of:

receiving data from a tracking subsystem having a first sensor set, a second sensor set, and a third sensor set, wherein the first sensor set and the second sensor set are different sensor sets, the first sensor set and the third sensor set are different sensor sets, and the second sensor set and the third sensor set are different sensor sets;

locating a point of interest at a center of a reference sphere based on a first set of data inputs from the first sensor set;

selecting a radius of the reference sphere;

computing a position of an intermediate virtual-viewpoint on the reference sphere, wherein the radius of the reference sphere and the location of the intermediate virtual-viewpoint are based on a second set of data inputs from the second sensor set;

determining a gaze direction from the intermediate virtual-viewpoint to the point of interest;

defining a final virtual-viewpoint by adjusting the location and gaze direction of the intermediate virtual-viewpoint based on a third set of data inputs from the third sensor set, wherein the final virtual viewpoint allows a user to view the point of interest in the virtual environment from an orientation view correlated to the intermediate virtual viewpoint, while simultaneously allowing a head of the user to face comfortably forward in the physical world; and

providing the final virtual viewpoint to a rendering engine to generate a virtual display to be displayed on a display device, whereby a user views, through the display device, the point of interest in a virtual environment from the final virtual viewpoint independently of physical orientation of a user input in the physical world.

2. A system for navigating in a virtual environment as set forth in claim 1, further comprising the operations of:

computing a reference vector between the final virtual viewpoint and the point of interest;

receiving a fourth set of data inputs from the third sensor set, wherein the fourth set of data inputs corresponds to new position of the head of the user and further correlates to a second point of interest; and

performing immersive navigation by utilizing the fourth set of data inputs and the reference vector, wherein immersive navigation allows the user to view the second point of interest in the virtual environment relative to the reference vector from the final virtual viewpoint to the point of interest.

3. A system for navigating in a virtual environment as set forth in claim 2,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, headY, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

4. A system for navigating in a virtual environment as set forth in claim 3, wherein a sensor set is selected from a group consisting of a tracker, a keyboard, a computer mouse, a video camera, a gesture recognizer, and speech;

and wherein the display is selected from a group consisting of a head mounted display, a plurality of video monitors, and a CAVE with supporting three-dimensional rendering software and hardware.

5. A system for navigating in a virtual environment as set forth in claim 4, wherein the sphere parameters correlate with wrist movements of the user, wherein the heading of the sphere correlates with the heading of the wrist, the pitch of the sphere correlates with the pitch of the wrist, and the radius of the sphere correlates with roll of the wrist and thumb of the user.

6. A system for navigating in a virtual environment as set forth in claim 5, wherein the initial viewpoint translation matrix is computed by

7. A system for navigating in a virtual environment as set forth in claim 1, wherein reference sphere parameters correlate with wrist movements of a user, wherein heading of the sphere correlates with heading of the wrist, pitch of the sphere correlates with pitch of the wrist, and radius of the sphere correlates with roll of the wrist and thumb of the user.

8. A system for navigating in a virtual environment as set forth in claim 1,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, headY, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

9. A system for navigating in a virtual environment as set forth in claim 8, wherein the initial viewpoint translation matrix is computed by

10. A computer implemented method for navigating in a virtual environment and suitable for allowing a user to change a view orientation in the virtual environment independently of physical orientation of a user input, the method comprising an act of causing a processor to perform operations of:

receiving data from a tracking subsystem having a first sensor set, a second sensor set, and a third sensor set, wherein the first sensor set and the second sensor set are different sensor sets, the first sensor set and the third sensor set are different sensor sets, and the second sensor set and the third sensor set are different sensor sets;

locating a point of interest at a center of a reference sphere based on a first set of data inputs from the first sensor set;

selecting a radius of the reference sphere;

computing a position of an intermediate virtual-viewpoint on the reference sphere, wherein the radius of the reference sphere and the location of the intermediate virtual-viewpoint are based on a second set of data inputs from the second sensor set;

determining a gaze direction from the intermediate virtual-viewpoint to the point of interest;

defining a final virtual-viewpoint by adjusting the location and gaze direction of the intermediate virtual-viewpoint based on a third set of data inputs from the third sensor set, wherein the final virtual viewpoint allows a user to view the point of interest in the virtual environment from an orientation view correlated to the intermediate virtual viewpoint, while simultaneously allowing a head of the user to face comfortably forward in the physical world; and

providing the final virtual viewpoint to a rendering engine to generate a virtual display to be displayed on a display device, whereby a user views, through the display device, the point of interest in a virtual environment from the final virtual viewpoint independently of physical orientation of a user input in the physical world.

11. A computer implemented method for navigating in a virtual environment as set forth in claim 10, further comprising the operations of:

computing a reference vector between the final virtual viewpoint and the point of interest;

receiving a fourth set of data inputs from the third sensor set, wherein the fourth set of data inputs corresponds to new position of the head of the user and further correlates to a second point of interest; and

performing immersive navigation by utilizing the fourth set of data inputs and the reference vector, wherein immersive navigation allows the user to view the second point of interest in the virtual environment relative to the reference vector from the final virtual viewpoint to the point of interest.

12. A computer implemented method for navigating in a virtual environment as set forth in claim 11,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, headY, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

13. A computer implemented method for navigating in a virtual environment as set forth in claim 12, wherein a sensor set is selected from a group consisting of a tracker, a keyboard, a computer mouse, a video camera, a gesture recognizer, and speech;

and wherein the display is selected from a group consisting of a head mounted display, a plurality of video monitors, and a CAVE with supporting three-dimensional rendering software and hardware.

14. A computer implemented method for navigating in a virtual environment as set forth in claim 13, wherein the sphere parameters correlate with wrist movements of the user, wherein the heading of the sphere correlates with the heading of the wrist, the pitch of the sphere correlates with the pitch of the wrist, and the radius of the sphere correlates with roll of the wrist and thumb of the user.

15. A computer implemented method for navigating in a virtual environment as set forth in claim 14, wherein the initial viewpoint translation matrix is computed by

16. A computer implemented method for navigating in a virtual environment as set forth in claim 10, wherein reference sphere parameters correlate with wrist movements of a user, wherein heading of the sphere correlates with heading of the wrist, pitch of the sphere correlates with pitch of the wrist, and radius of the sphere correlates with roll of the wrist and thumb of the user.

17. A computer implemented method for navigating in a virtual environment as set forth in claim 10,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, heady, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

18. A computer implemented method for navigating in a virtual environment as set forth in claim 17, wherein the initial viewpoint translation matrix is computed by

19. A computer program product for navigating in a virtual environment and suitable for allowing a user to change a view orientation in the virtual environment independently of physical orientation of a user input, the computer program product comprising computer-readable means stored on a computer readable medium that are executable by a computer having a processor for causing the processor to perform the operations of:

receiving data from a tracking subsystem having a first sensor set, a second sensor set, and a third sensor set, wherein the first sensor set and the second sensor set are different sensor sets, the first sensor set and the third sensor set are different sensor sets, and the second sensor set and the third sensor set are different sensor sets;

locating a point of interest at a center of a reference sphere based on a first set of data inputs from the first sensor set;

selecting a radius of the reference sphere;

computing a position of an intermediate virtual-viewpoint on the reference sphere, wherein the radius of the reference sphere and the location of the intermediate virtual-viewpoint are based on a second set of data inputs from the second sensor set;

determining a gaze direction from the intermediate virtual-viewpoint to the point of interest;

defining a final virtual-viewpoint by adjusting the location and gaze direction of the intermediate virtual-viewpoint based on a third set of data inputs from the third sensor set, wherein the final virtual viewpoint allows a user to view the point of interest in the virtual environment from an orientation view correlated to the intermediate virtual viewpoint, while simultaneously allowing a head of the user to face comfortably forward in the physical world; and

providing the final virtual viewpoint to a rendering engine to generate a virtual display to be displayed on a display device, whereby a user views, through the display device, the point of interest in a virtual environment from the final virtual viewpoint independently of physical orientation of a user input in the physical world.

20. A computer program product for navigating in a virtual environment as set forth in claim 19, further comprising the operations of:

computing a reference vector between the final virtual viewpoint and the point of interest;

receiving a fourth set of data inputs from the third sensor set, wherein the fourth set of data inputs corresponds to new position of the head of the user and further correlates to a second point of interest; and

performing immersive navigation by utilizing the fourth set of data inputs and the reference vector, wherein immersive navigation allows the user to view the second point of interest in the virtual environment relative to the reference vector from the final virtual viewpoint to the point of interest.

21. A computer program product for navigating in a virtual environment as set forth in claim 20,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, headY, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

22. A computer program product for navigating in a virtual environment as set forth in claim 21, wherein a sensor set is selected from a group consisting of a tracker, a keyboard, a computer mouse, a video camera, a gesture recognizer, and speech;

and wherein the display is selected from a group consisting of a head mounted display, a plurality of video monitors, and a CAVE with supporting three-dimensional rendering software and hardware.

23. A computer program product for navigating in a virtual environment as set forth in claim 22, wherein the sphere parameters correlate with wrist movements of the user, wherein the heading of the sphere correlates with the heading of the wrist, the pitch of the sphere correlates with the pitch of the wrist, and the radius of the sphere correlates with roll of the wrist and thumb of the user.

24. A computer program product for navigating in a virtual environment as set forth in claim 23, wherein the initial viewpoint translation matrix is computed by

25. A computer program product for navigating in a virtual environment as set forth in claim 19, wherein reference sphere parameters correlate with wrist movements of a user, wherein heading of the sphere correlates with heading of the wrist, pitch of the sphere correlates with pitch of the wrist, and radius of the sphere correlates with roll of the wrist and thumb of the user.

26. A computer program product for navigating in a virtual environment as set forth in claim 19,

wherein the first set of data inputs comprises an initial position of the user with respect to a three-dimensional coordinate system (headX, headY, and headZ), and wherein the operation of locating a point of interest comprises computing an initial viewpoint translation matrix;

wherein the second set of data inputs comprises radius of reference sphere, pitch of reference sphere, and heading of reference sphere, and wherein the operation of computing a position of an intermediate virtual-viewpoint comprises computing a trackball matrix to rotate and position intermediate virtual-viewpoint based on sphere parameters; and

wherein the third set of data inputs comprises pitch of the head of the user, heading of the head of the user, roll of the head of the user, and three-dimensional position of the head of the user within a three-dimensional coordinate system (headX, headY, headZ), and wherein the operation of defining a final virtual viewpoint comprises computing a final hybrid viewpoint matrix.

27. A computer program product for navigating in a virtual environment as set forth in claim 26, wherein the initial viewpoint translation matrix is computed by