Indirect 3D scene positioning control

• US 9,292,184 B2
• Filed: 11/16/2012
• Issued: 03/22/2016
• Est. Priority Date: 11/18/2011
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Hosenpud, Jonathan J., Vesely, Michael A.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): KHAN, IBRAHIM A
• Application Number:
  US14/358,762
• Publication Number:
  US 20140300547A1
• Time in Patent Office: 1,222 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Embodiments of the present invention generally relate to interacting with a virtual scene at a perspective which is independent from the perspective of the user. Methods and systems can include either tracking and defining a perspective of the user based on the position and orientation of the user in the physical space, projecting a virtual scene for the user perspective to a virtual plane, tracking and defining a perspective of the a freehand user input device based on the position and orientation of the a freehand user input device, identifying a mark in the virtual scene which corresponds to the position and orientation of the device in the physical space, creating a virtual segment from the mark and interacting with virtual objects in the virtual scene at the end point of the virtual segment, as controlled using the device.

Claims

1. A method, comprising:

rendering a first image of a virtual scene projected to a virtual render plane from a first projection to be displayed on a screen;

displaying the first image;

viewing the image of the projection from a first eyepoint that is a normal perspective to the screen such that the projected image corresponds to the first perspective;

positioning and orienting a freehand user interface device that contains a physical endpoint, wherein the physical endpoint defines a direction in a physical space relative to the screen, wherein the physical endpoint defines a second perspective that is other than normal to the screen such that the position and orientation of the freehand user interface device defines a second perspective that is distinct from the orientation of the first perspective;

identifying a virtual position/orientation mark in the virtual scene corresponding to the physical end point in the physical space;

identifying a segment in the virtual scene along a path from the mark as suggested by the second perspective direction in the physical space, where the segment suggests a path that is not normal to the render plane; and

interacting with a virtual object from within the virtual scene that correlates to an intersection between the virtual object and the segment, including selecting the virtual object via a selection shape, comprising;

selecting a first point in the virtual scene from the second perspective in accordance with a first intersection;

selecting a second point in the virtual scene from a third perspective in accordance with a second intersection; and

creating a rectangular volume between the first point and the second point, wherein the planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

2. The method of claim 1, wherein the freehand user interface device is a stylus.

3. The method of claim 1, further comprising:

moving and/or orienting the freehand user interface device in the physical space;

tracking the moving and/or orienting of the freehand user interface device in the physical space; and

moving the mark within the virtual scene based upon the tracking of moving the freehand user interface device.

4. The method of claim 1, further comprising receiving telemetry data from the freehand user interface device.

5. The method of claim 1, further comprising altering the virtual object within the virtual scene, wherein the altering comprises moving and orienting the freehand user interface device in the physical space and correlating the moving of the freehand user interface device with movement of the mark in the virtual scene.

6. The method of claim 5, wherein altering the virtual object within the virtual scene comprises changing the virtual object such that the virtual object, when viewed from the first perspective, is projected in the virtual scene to be substantially identical to the virtual object projected in the virtual scene as the virtual object would appear when viewed from the second perspective prior to moving.

7. The method of claim 1, wherein the second perspective correlates to the position and orientation of the physical endpoint and wherein the second perspective changes with changes in the position and/or orientation of the physical endpoint.

8. The method of claim 1, wherein the second point is selected at the third perspective which is changed based on a new position and orientation of the physical endpoint.

9. A method, comprising:

tracking a position and orientation of a first eye-point in a physical space as referenced from a display screen;

determining a tracked perspective of the first eye-point from the position and orientation of the first eye-point as referenced from the position and orientation of the display screen;

rendering a first image to be displayed on a screen of a virtual scene projected to a virtual render plane from the tracked perspective;

viewing the image of the projection from the first eye-point in a physical space, such that the virtual scene corresponds to the tracked perspective;

positioning and orienting a freehand user interface device comprising a physical endpoint in a physical space, such that the physical endpoint suggests a direction in the physical space relative to the screen;

defining a second perspective based on the position and orientation of the physical endpoint that is other than normal to the first eye-point, such that the freehand user interface device has a second perspective that is distinct from the first perspective;

identifying a virtual position and orientation mark in the virtual scene corresponding to the position and orientation of the physical endpoint in the physical space;

identifying a segment along a path from the virtual position and orientation mark in the virtual space in the suggested direction; and

interacting with a virtual object from within the virtual scene that correlates to an intersection between the object and the segment, including selecting the virtual object via a selection shape, comprising;

selecting a first point in the virtual scene from the second perspective in accordance with a first intersection;

selecting a second point in the virtual scene from a third perspective in accordance with a second intersection; and

creating a rectangular volume between the first point and the second point, wherein the planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

10. The method of claim 9, further comprising:

moving and/or orienting the freehand user interface device in the physical space;

tracking the moving and/or orienting of the freehand user interface device in the physical space; and

moving the mark within the virtual scene based upon the tracking of the moving of the freehand user interface device.

11. The method of claim 9, further comprising receiving telemetry data from the freehand user interface device.

12. The method of claim 9, further comprising altering the virtual object within the virtual scene, wherein the altering comprises moving and orienting the freehand user interface device in the physical space and correlating the moving of the freehand user interface device with movement of the mark within the virtual scene.

13. The method of claim 9, wherein altering the virtual object within the virtual scene comprises changing the virtual object such that the virtual object, when viewed from the first perspective, is projected in the virtual scene to be substantially identical to the virtual object projected in the virtual scene, as the virtual object would appear when viewed from the second perspective prior to moving.

14. The method of claim 9, wherein the second perspective correlates to the position and orientation of the physical endpoint and wherein the second perspective changes with changes in the position and/or orientation of the physical endpoint.

15. The method of claim 9, wherein the second point is selected at the third perspective which is changed based on a new position and orientation of the physical endpoint.

16. A system, comprising:

a display device configured to;

render a first image to be displayed on a screen of a virtual scene projected to a virtual render plane from a first perspective; and

display an image of the projection from a first eye-point in a physical space that is a normal perspective to the screen, such that the image corresponds to the first perspective;

a freehand user interface device comprising a physical endpoint in a physical space and configured to select a virtual object from within the virtual scene; and

a tracking device configured to;

track the position and orientation of a freehand user interface device to determine a suggested direction of the physical end-point in the physical space relative to the screen;

define a second perspective based on the position and orientation of the physical endpoint that is other than normal to the screen or the first eye-point, such that the freehand user interface device has a second perspective that is distinct from the first perspective;

identify a virtual position and orientation mark in the virtual space corresponding to the position and orientation of the physical endpoint in the physical space;

identify a segment along a path from the virtual position and orientation mark in the suggested direction of the physical endpoint; and

correlate an intersection between the virtual object and the segment for selection of the virtual object via a selection shape, comprising;

selecting a first point in the virtual scene from the second perspective in accordance with a first intersection;

selecting a second point in the virtual scene from a third perspective in accordance with a second intersection; and

creating a rectangular volume between the first point and the second point, wherein the planes of the rectangular volume are oblique in relation to the virtual render plane, wherein the rectangular volume forms the selection shape.

17. The system of claim 16, wherein the tracking device is configured to receive telemetry data from the freehand user interface device.

18. The system of claim 16, wherein the tracking device is configured to correlate movement of the selected virtual object within the virtual scene with movement of the freehand user interface device in the physical space.

19. The system of claim 16, wherein the second perspective is perpendicular to the first perspective.

20. The system of claim 16, wherein the display is a stereo display and wherein the first perspective comprises two projections.