Apparatus and methods for haptic rendering using data in a graphics pipeline

• US 7,990,374 B2
• Filed: 06/28/2005
• Issued: 08/02/2011
• Est. Priority Date: 06/29/2004
• Status: Active Grant

• Current Assignee: 3D Systems, Inc. (3D Systems Corporation)
• Original Assignee: SensAble Technologies, Inc. (3D Systems Corporation)
• Inventors: Midura, Marc Douglass, Handley, Joshua E., Goodwin, William Alexander, Shih, Loren C., Itkowitz, Brandon D.
• Primary Examiner(s): NGUYEN, PHU K
• Application Number:
  US11/169,175
• Publication Number:
  US 20060109266A1
• Time in Patent Office: 2,226 Days
• Field of Search: 345/419, 345/420, 345/422, 345/619, 345/620, 846/09, 703/4, 715/757
• US Class Current: 345/419
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/28 Force feedback
• CPC Class Codes: G06T 2215/06 Curved planar reformation of 3D line structures

Abstract

The invention provides methods for leveraging data in the graphics pipeline of a 3D graphics application for use in a haptic rendering of a virtual environment. The invention provides methods for repurposing graphical information for haptic rendering. Thus, at least part of the work that would have been performed by a haptic rendering process to provide touch feedback to a user is obviated by work performed by the graphical rendering process.

Claims

1. A method for interpreting data for use in a haptic rendering of a virtual environment, the method comprising the steps of:

accessing graphical rendering data in a graphics pipeline of a 3D graphics processor, wherein the graphical rendering data is collected from a virtual camera and comprises data generated to produce a 2D screen view of a 3D virtual object in the virtual environment; and

interpreting the graphical rendering data for use in a haptic rendering of the 3D virtual object in the virtual environment, wherein the interpreting step comprises defining a force to be transmitted to a user via a haptic interface device.

2. The method of claim 1, wherein the graphical rendering data comprises a representation of a geometric element of the virtual object in the virtual environment.

3. The method of claim 1, further comprising at least one of the following steps:

creating a graphical rendering of the virtual object;

storing a graphical rendering of the virtual object;

displaying a graphical rendering of the virtual object;

printing a graphical rendering of the virtual object; and

modifying a graphical rendering of the virtual object.

4. The method of claim 1, wherein the graphical rendering data comprises at least one of the following:

at least a portion of a depth buffer;

at least a portion of a feedback buffer;

at least a portion of a color buffer;

at least a portion of a selection buffer;

at least a portion of an accumulation buffer;

at least a portion of a texture map;

at least a portion of a fat framebuffer;

data from a pixel shading program;

data from a vertex shading program;

rasterization primitives;

application programming interface input data; and

state data.

5. The method of claim 1, wherein the graphical rendering data comprises a transformation matrix for mapping between object space and screen space.

6. The method of claim 1, wherein the graphical rendering data comprises at least a portion of a depth buffer.

7. The method of claim 6, wherein the accessing step comprises accessing at least a portion of the depth buffer from a graphics card.

8. The method of claim 6, wherein the interpreting step comprises locating a point on a surface of a virtual object in the virtual environment using the depth buffer data.

9. The method of claim 8, wherein locating the point on the surface of the virtual object comprises:

determining an intersection point and an intersection normal in screen space; and

transforming coordinates of at least one of the intersection point and the intersection normal from screen space to object space.

10. The method of claim 6, wherein the interpreting step comprises resolving an occlusion.

11. The method of claim 6, wherein the depth buffer comprises a 2D buffer of depth values.

12. The method of claim 6, wherein the graphical rendering data comprises at least a portion of at least one of the following:

a color buffer, a stencil buffer, and an accumulation buffer.

13. The method of claim 12, wherein the interpreting step comprises using the color buffer and the depth buffer to haptically interpret smooth shading along a surface of the virtual object in the virtual environment.

14. The method of claim 12, wherein the interpreting step comprises using the stencil buffer and the depth buffer to haptically interpret a variation along a surface of the virtual object in the virtual environment.

15. The method of claim 14, wherein the variation comprises at least one of a friction variation, a stiffness variation, and a damping.

16. The method of claim 1, wherein the graphical rendering data in the graphics pipeline comprises data resulting from a hardware-accelerated geometry modification.

17. The method of claim 16, wherein the hardware-accelerated geometry modification comprises at least one of a view-dependent adaptive subdivision, a view-dependent tessellation, a displacement mapping, and a normal mapping.

18. The method of claim 1, wherein the graphical rendering data comprises at least a portion of a feedback buffer.

19. The method of claim 1, wherein the graphical rendering data comprises rasterization primitives.

20. The method of claim 18, wherein the feedback buffer excludes geometric elements that lie outside a selected view volume.

21. The method of claim 18, wherein the interpreting step comprises using the feedback buffer to haptically interpret a non-uniform surface.

22. The method of claim 1, wherein the accessing step comprises sizing a view volume to exclude geometric elements that lie beyond a desired distance from a haptic interface location.

23. The method of claim 1, further comprising the step of culling at least a portion of the data in the graphics pipeline to exclude data corresponding to geometric primitives that lie outside a view volume.

24. The method of claim 23, wherein the culling step is performed using the graphics processor.

25. The method of claim 23, wherein the culling step is performed using a spatial partition.

26. The method of claim 25, wherein the spatial partition comprises a hierarchical data structure.

27. The method of claim 26, wherein the hierarchical data structure comprises at least one of an octree data structure and a BSP data structure.

28. The method of claim 23, wherein the culling step is performed using the graphics processor and a spatial partition.

29. The method of claim 1, wherein the graphical rendering data in the graphics pipeline comprises at least a portion of a fat framebuffer.

30. The method of claim 29, wherein the fat framebuffer comprises at least one member of the group consisting of:

vertex positions;

normals;

color;

texture;

normal maps;

bump maps; and

depth data.

31. The method of claim 1, wherein the graphical rendering data comprises application programming interface data.

32. The method of claim 31, wherein the application programming interface data comprises state data.

33. The method of claim 32, wherein the state data comprises at least one of the following:

an operator, an operand, a resultant state, a stencil buffer, a raster position, a lighting variable, a line width, an enabling status, and a disabling status.

34. The method of claim 1, wherein the accessing step comprises intercepting at least a portion of the data using a pass-through dynamic link library.

35. The method of claim 1, wherein the accessing step comprises writing at least a subset of the data in the graphics pipeline to a memory buffer.

36. The method of claim 35, wherein the accessing step further comprises reading at least a subset of the data that is written to the memory buffer.

37. The method of claim 1, wherein the interpreting step comprises determining a height map using the accessed graphical rendering data.

38. The method of claim 37, wherein the height map describes at least a portion of a surface of the virtual object in the virtual environment.

39. The method of claim 37, wherein the interpreting step further comprises generating a mesh from the height map.

40. The method of claim 1, wherein the interpreting step comprises:

performing an

intersection test to determine at least one intersection point and at least one intersection normal in screen space; and

transforming coordinates of the at least one intersection point and the at least one intersection normal from screen space to object space.

41. The method of claim 40, wherein the transforming step comprises defining for each intersection point a plane tangent to the surface of a virtual object at the intersection point.

42. The method of claim 40, wherein the transforming step comprises performing a projection test to determine a geometric feature nearest a haptic interface location.

43. The method of claim 40, wherein the interpreting step comprises concatenating a plurality of transforms used by the 3D graphics processor into a single transformation.

44. The method of claim 43, wherein the plurality of transforms comprises at least one of a world-view transformation, a view-clip transformation, a clip-window transformation, a view-touch transformation, and a touch-workspace transformation.

45. The method of claim 40, wherein the interpreting step comprises correcting perspective foreshortening.

46. The method of claim 1, wherein the interpreting step comprises transforming intersection line segments from object space to screen space and transforming resultant intersection points and intersection normals from screen space to object space.

47. The method of claim 1, further comprising the step of haptically rendering the virtual environment using the interpreted data.

48. The method of claim 47, wherein the step of haptically rendering the virtual environment comprises providing a representation of a surface of the virtual object in the virtual environment.

49. The method of claim 47, wherein the step of haptically rendering the virtual environment comprises:

determining a haptic interface location in the virtual environment corresponding to a location of the haptic interface device in real space;

determining a location of at least one point on a surface of the virtual object in the virtual environment; and

determining an interaction force based at least in part on the haptic interface location and the location of the at least one point of the surface of the virtual object.

50. The method of claim 49, further comprising the step of delivering the interaction force to the user through the haptic interface device.

51. The method of claim 1, comprising the step of using the interpreted data to haptically render the virtual environment in real-time during the operation of the 3D graphics processor.

52. The method of claim 1, comprising the step of generating user interface input for use by the 3D graphics processor.

53. The method of claim 52, wherein the step of generating user interface input comprises converting a 3D position of a haptic interface device into a 2D cursor position.

54. The method of claim 1, further comprising the step of haptically rendering a user interface menu.

55. The method of claim 1, wherein the graphics pipeline comprises a first camera positioned at a first location within the virtual environment.

56. The method of claim 55, wherein the graphics pipeline comprises a second camera positioned substantially at a haptic interface location within the virtual environment.

57. The method of claim 56, wherein the accessing step comprises accessing data corresponding to a view from the first camera and accessing data corresponding to a view from the second camera.

58. The method of claim 57, wherein the method further comprises the step of haptically rendering the virtual environment using the interpreted data.

59. An apparatus for providing haptic feedback to a user of a 3D graphics application, the apparatus comprising:

a user-controlled haptic interface device adapted to provide a user input to a computer and to transmit force to the user; and

computer software that, when operating with the computer and the user input, is adapted to determine force transmitted to the user via a haptic interface device by (i) accessing graphical rendering data from a graphics pipeline of a 3D graphics processor, the graphical rendering data having been collected from a virtual camera and comprising data generated to produce a 2D screen view of a 3D virtual object in the virtual environment and (ii) haptically rendering a 3D virtual object in a virtual environment using the accessed graphical rendering data, where the force is determined according to the position of a haptic interface location in relation to a surface of the virtual object.

60. The apparatus of claim 59, wherein the haptic interface device has at least two degrees of freedom.

61. The apparatus of claim 59, wherein the haptic interface device has six degrees of freedom.

62. The apparatus of claim 59, wherein the haptic interface device comprises at least one button input.

63. The apparatus of claim 59, wherein the haptic interface device comprises at least one member of the group consisting of a button, a joystick, a stylus, a wheel, a mouse, a gripper, a pressure pad, a toggle switch, and a pressure switch.

64. The apparatus of claim 59, wherein the 3D processor comprises at least one graphics tool.

65. The apparatus of claim 64, wherein the at least one graphics tool comprises at least one member selected from the group consisting of a drawing tool, an editing tool, a control point editing tool, a sculpting tool, a pressure-sensitive sculpting tool, a painting tool, a pressure-sensitive painting tool, an airbrush, and an angle-sensitive airbrush.

66. The apparatus of claim 59, wherein force transmitted to the user is a function of an action of a graphics tool of the 3D graphics processor.

67. The apparatus of claim 59, wherein the haptic interface device is adapted to receive force from the user.

68. The apparatus of claim 67, wherein the user input comprises user-imposed force, and wherein the software is adapted to generate input for use by the 3D graphics processor based at least in part on the user-imposed force.

69. The apparatus of claim 59, wherein the virtual environment comprises at least one animated virtual object.