Indirect 3D scene positioning control

• US 9,864,495 B2
• Filed: 03/21/2016
• Issued: 01/09/2018
• Est. Priority Date: 11/18/2011
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Hosenpud, Jonathan J., Vesely, Michael A.
• Primary Examiner(s): KHAN, IBRAHIM A
• Application Number:
  US15/075,725
• Publication Number:
  US 20160202876A1
• Time in Patent Office: 659 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Embodiments of the present invention generally relate to interacting with a virtual scene at a perspective which is independent from the perspective of the user. Methods and systems can include either tracking and defining a perspective of the user based on the position and orientation of the user in the physical space, projecting a virtual scene for the user perspective to a virtual plane, tracking and defining a perspective of the a freehand user input device based on the position and orientation of the a freehand user input device, identifying a mark in the virtual scene which corresponds to the position and orientation of the device in the physical space, creating a virtual segment from the mark and interacting with virtual objects in the virtual scene at the end point of the virtual segment, as controlled using the device.

Claims

1. A method for interacting with a virtual object from within a virtual scene projected to a virtual render plane, comprising:

determining, via an optical tracking device, a first position and orientation of a physical endpoint of a user interface device in a physical space, wherein the position and orientation is with respect to a display device rendering the virtual scene;

determining a perspective of the user interface device based on the determined position and orientation, wherein the perspective is with respect to the display device, and wherein the perspective is not normal to the virtual render plane;

identifying a virtual position and orientation of a virtual endpoint in a virtual scene, wherein the virtual position and orientation of the virtual endpoint in the virtual scene correspond to the position and orientation of the physical endpoint in the physical space;

identifying a segment in the virtual scene along a path from the virtual endpoint based on the perspective of the user interface device, wherein the segment indicates a path that is not normal to the virtual render plane; and

correlating an intersection between the virtual object and the segment, including receiving, via the user interface device, user input to select a virtual object via a selection shape, comprising;

receiving user input to select a first point in the virtual scene from the perspective in accordance with a first intersection;

receiving user input to select a second point in the virtual scene from a different perspective in accordance with a second intersection, wherein the second perspective is based on a second position and orientation of the user interface device in the physical space, and wherein the second position and orientation is determined via the optical tracking device; and

creating a rectangular volume between the first point and the second point, wherein planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

2. The method of claim 1,

wherein the user interface device is a stylus.

3. The method of claim 1, further comprising:

tracking changes in position and orientation of the physical endpoint in the physical space; and

moving the virtual endpoint within the virtual scene based upon the tracking.

4. The method of claim 1, further comprising:

receiving telemetry data from the user interface device, wherein the telemetry data is used in conjunction with information collected by the optical tracking device to determine the position and orientation of the user interface device.

5. The method of claim 1, further comprising:

altering the virtual object within the virtual scene, wherein the altering comprises correlating changes in position and orientation of the physical endpoint in physical space to changes in position and orientation of the virtual endpoint in the virtual space.

6. The method of claim 5,

wherein altering the virtual object within the virtual scene comprises changing the virtual object such that the virtual object, when viewed from a perspective of a user, is projected in the virtual scene to be substantially identical to the virtual object projected in the virtual scene as the virtual object would appear when viewed from the perspective prior to moving.

7. The method of claim 1,

wherein the perspective correlates to the position and orientation of the physical endpoint and wherein the perspective changes with changes in the position and/or orientation of the physical endpoint.

8. The method of claim 1,

wherein the second point is selected at the third perspective which is changed based on a new position and orientation of the physical endpoint.

9. A non-transitory computer readable memory medium storing program instructions executable by a processor to:

determine, via an optical tracking system, a first position and orientation of a physical endpoint of a user interface device in a physical space, wherein the position and orientation is with respect to a display device rendering a virtual scene projected to a virtual render plane;

determine a perspective of the user interface device based on the determined position and orientation, wherein the perspective is with respect to the display device, and wherein the perspective is not normal to the virtual render plane;

identify a virtual position and orientation of a virtual endpoint in a virtual scene, wherein the virtual position and orientation of the virtual endpoint in the virtual scene correspond to the position and orientation of the physical endpoint in the physical space;

identify a segment in the virtual scene along a path from the virtual endpoint based on the perspective of the user interface device, wherein the segment indicates a path that is not normal to the virtual render plane; and

correlate an intersection between a virtual object and the segment, via a selection shape, wherein to correlate the intersection, the program instructions are further executable to;

receive user input to select a first point in the virtual scene from the perspective in accordance with a first intersection;

receive user input to select a second point in the virtual scene from a different perspective in accordance with a second intersection, wherein the second perspective is based on a second position and orientation of the user interface device in the physical space, and wherein the second position and orientation is determined via the optical tracking device; and

create a rectangular volume between the first point and the second point, wherein planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

10. The non-transitory computer readable memory medium of claim 9,

wherein the program instructions are further executable to:

track changes in position and orientation of the physical endpoint in the physical space; and

move the virtual endpoint within the virtual scene based upon the tracking.

11. The non-transitory computer readable memory medium of claim 9,

wherein the program instructions are further executable to:

receive telemetry data from the user interface device, wherein the telemetry data is used in conjunction with information collected by the optical tracking device to determine the position and orientation of the user interface device.

12. The non-transitory computer readable memory medium of claim 9,

wherein the program instructions are further executable to:

alter the virtual object within the virtual scene, wherein to alter the virtual object the program instructions are further executable to correlate changes in position and orientation of the physical endpoint in physical space to changes in position and orientation of the virtual endpoint in the virtual space.

13. The non-transitory computer readable memory medium of claim 9,

wherein the perspective correlates to the position and orientation of the physical endpoint and wherein the perspective changes with changes in the position and/or orientation of the physical endpoint.

14. The non-transitory computer readable memory medium of claim 9,

wherein the second point is selected at the third perspective which is changed based on a new position and orientation of the physical endpoint.

15. A system comprising:

a display device;

an optical tracking system;

a memory; and

a processor in communication with the memory, wherein the processor is configured to;

determine, via the optical tracking system, a first position and orientation of a physical endpoint of a user interface device in a physical space, wherein the position and orientation is with respect to the display device rendering a virtual scene projected to a virtual render plane;

determine a perspective of the user interface device based on the determined position and orientation, wherein the perspective is with respect to the display device, and wherein the perspective is not normal to the virtual render plane;

identify a virtual position and orientation of a virtual endpoint in a virtual scene, wherein the virtual position and orientation of the virtual endpoint in the virtual scene correspond to the position and orientation of the physical endpoint in the physical space;

identify a segment in the virtual scene along a path from the virtual endpoint based on the perspective of the user interface device, wherein the segment indicates a path that is not normal to the virtual render plane; and

correlate an intersection between a virtual object and the segment via a selection shape, wherein to correlate the intersection, the processor is further configured to;

receive user input to select a first point in the virtual scene from the perspective in accordance with a first intersection;

receive user input to select a second point in the virtual scene from a different perspective in accordance with a second intersection, wherein the different perspective is based on a second position and orientation of the user interface device in the physical space, and wherein the second position and orientation is determined via the optical tracking device; and

create a rectangular volume between the first point and the second point, wherein planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

16. The system of claim 15,

wherein the processor is further configured to:

track changes in position and orientation of the physical endpoint in the physical space; and

move the virtual endpoint within the virtual scene based upon the tracking.

17. The system of claim 15,

wherein the processor is further configured to:

receive telemetry data from the user interface device, wherein the telemetry data is used in conjunction with information collected by the optical tracking device to determine the position and orientation of the user interface device.

18. The system of claim 15,

wherein the processor is further configured to:

alter the virtual object within the virtual scene, wherein to alter the virtual object the program is configured to correlate changes in position and orientation of the physical endpoint in physical space to changes in position and orientation of the virtual endpoint in the virtual space.

19. The system of claim 15,

wherein the perspective correlates to the position and orientation of the physical endpoint and wherein the perspective changes with changes in the position and/or orientation of the physical endpoint.

20. The system of claim 15,

wherein the second point is selected at the third perspective which is changed based on a new position and orientation of the physical endpoint.