Apparatus and methods for haptic rendering using a haptic camera view

• US 20060284834A1
• Filed: 06/28/2005
• Published: 12/21/2006
• Est. Priority Date: 06/29/2004
• Status: Abandoned Application

• Current Assignee: 3D Systems, Inc. (3D Systems Corporation)
• Original Assignee: SensAble Technologies, Inc. (3D Systems Corporation)
• Inventors: Midura, Marc Douglass, Handley, Joshua E., Goodwin, William Alexander, Shih, Loren C., Itkowitz, Brandon D.
• Application Number:
  US11/169,271
• Publication Number:
  US 20060284834A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/28 Force feedback
• CPC Class Codes: G06T 2215/06 Curved planar reformation of 3D line structures

Abstract

The invention provides systems and methods for using a “haptic camera” within a virtual environment and for using graphical data from the haptic camera to produce touch feedback. The haptic camera obtains graphical data pertaining to virtual objects within the vicinity and along the trajectory of a user-controlled haptic interface device. The graphical data from the camera is interpreted haptically, thereby allowing touch feedback corresponding to the virtual environment to be provided to the user.

Claims

1. A method for haptically rendering a virtual object in a virtual environment, the method comprising the steps of:

(a) determining a haptic interface location in a 3D virtual environment corresponding to a location of a haptic interface device in real space;

(b) positioning a first virtual camera substantially at the haptic interface location;

(c) accessing graphical data from the first virtual camera corresponding to the virtual environment;

(d) determining a position of the haptic interface location in relation to at least one geometric feature of a virtual object in the virtual environment using the graphical data from the first virtual camera; and

(e) determining an interaction force based at least in part on the position of the haptic interface location in relation to the at least one geometric feature of the virtual object.

2. The method of claim 1, wherein the at least one graphical feature of the virtual object comprises at least one of a surface, point, line, or plane.

3. The method of claim 1, further comprising the step of delivering the interaction force to a user through the haptic interface device.

4. The method of claim 1, wherein the position of the first camera is updated as the haptic interface location changes according to movement of the haptic interface device.

5. The method of claim 1, further comprising the steps of:

(f) positioning a second virtual camera at a location other than the haptic interface location; and

(g) accessing graphical data from the second virtual camera corresponding to the virtual environment.

6. The method of claim 5, wherein the second virtual camera is fixed and the first virtual camera moves.

7. The method of claim 1, wherein step (c) comprises accessing graphical data from a graphics pipeline of a 3D graphics application.

8. The method of claim 1, wherein step (d) comprises determining a world-view transform that maps world coordinates corresponding to the haptic virtual environment to view coordinates corresponding to the first virtual camera.

9. The method of claim 8, wherein the step of determining the world-view transform comprises determining an eye position and a look direction.

10. The method of claim 9, wherein determining the eye position comprises sampling the position of the haptic interface location and wherein determining the look direction comprises determining a vector representing a motion of the haptic interface location.

11. The method of claim 8, wherein step (d) further comprises at least one of the following:

determining a shape-world transformation;

determining a world-view transformation;

determining a view-clip transformation;

determining a clip-window transformation;

determining a view-touch transformation; and

determining a touch-workspace transformation.

12. The method of claim 1, wherein a view volume associated with the first virtual camera is sized to exclude geometric elements that lie beyond a desired distance from the haptic interface location.

13. The method of claim 1, further comprising the step of culling at least a portion of the graphical data from the first virtual camera to exclude data corresponding to geometric primitives that lie outside a view volume associated with the first camera.

14. The method of claim 13, wherein the culling step is performed using graphics hardware.

15. The method of claim 13, wherein the culling step is performed using a spatial partition.

16. The method of claim 15, wherein the spatial partition comprises a hierarchical data structure.

17. The method of claim 16, wherein the hierarchical data structure comprises at least one of an octree data structure and a BSP tree data structure.

18. The method of claim 13, wherein the culling step is performed using graphics hardware and a spatial partition.

19. The method of claim 1, wherein the graphical data from the first virtual camera comprises at least one of the following:

at least a portion of a depth buffer;

at least a portion of a feedback buffer;

at least a portion of a color buffer;

at least a portion of a selection buffer;

at least a portion of an accumulation buffer;

at least a portion of a texture map;

at least a portion of a fat framebuffer;

data from a pixel shading program;

data from a vertex shading program

rasterization primitives;

application programming interface input data; and

state data.

20. The method of claim 1, wherein the graphical data from the first virtual camera comprises at least a portion of a fat framebuffer.

21. The method of claim 20, wherein the fat framebuffer comprises at least one member of the group consisting of:

vertex positions;

normals;

color;

texture;

normal maps;

bump maps; and

depth data.

22. The method of claim 20, wherein step (c) comprises performing at least one of a pixel shading and a vertex shading.

23. The method of claim 22, wherein the shading is performed using graphics hardware.

24. The method of claim 1, wherein the graphical data from the first virtual camera comprises at least a portion of a depth buffer.

25. The method of claim 1, wherein the graphical data from the first virtual camera comprises rasterization primitives.

26. The method of claim 1, wherein step (d) comprises:

performing an intersection test to determine at least one intersection point and at least one intersection normal in screen space; and

transforming coordinates of the at least one intersection point and the at least one intersection normal from screen space to object space.

27. The method of claim 26, wherein step (d) comprises defining for each intersection point a plane tangent to the surface of the virtual object at the intersection point.

28. The method of claim 26, wherein step (d) comprises performing a projection test to determine a geometric feature nearest the haptic interface location.

29. A system for haptically rendering a virtual object in a virtual environment, the system comprising:

a graphics thread that generates a visual display of a virtual environment;

a collision thread that determines if a user-directed virtual proxy collides with at least one geometric feature within the virtual environment, wherein the collision thread uses input from the graphics thread; and

a servo thread that generates force to be applied to a user in real space through a haptic interface device according to input from the collision thread, wherein the servo thread is in communication with the haptic interface device.

30. The system of claim 29, wherein the graphics thread refreshes the visual display at a rate within a range from about 5 Hz to about 150 Hz.

31. The system of claim 29, wherein the graphics thread refreshes the visual display at a rate within a range from about 30 Hz to about 60 Hz.

32. The system of claim 29, wherein the collision thread performs a collision detection computation at a rate within a range from about 30 Hz to about 200 Hz.

33. The system of claim 29, wherein the collision thread performs a collision detection computation at a rate within a range from about 80 Hz to about 120 Hz.

34. The system of claim 29, wherein the servo thread refreshes the force to be applied through the haptic interface device at a rate within a range from about 1000 Hz to about 10,000 Hz.

35. The system of claim 29, wherein the servo thread comprises at least one of a force shader and a proxy shader.

36. An apparatus for providing haptic feedback to a user of a 3D graphics application, the apparatus comprising:

a user-controlled haptic interface device adapted to provide a user input to a computer and to transmit force to a user; and

computer software that, when operating with the computer and the user input, is adapted to determine force transmitted to the user by;

(a) determining a haptic interface location in a 3D virtual environment corresponding to a location of the haptic interface device in real space;

(b) positioning a first virtual camera substantially at the haptic interface location;

(c) accessing graphical data from the first virtual camera corresponding to the virtual environment;

(d) determining a position of the haptic interface location in relation to at least one geometric feature of a virtual object in the virtual environment using the graphical data from the first virtual camera; and

(e) determining an interaction force based at least in part on the position of the haptic interface location in relation to the at least one geometric feature of the virtual object.