SYSTEM WITH 3D USER INTERFACE INTEGRATION

• US 20130257718A1
• Filed: 12/05/2011
• Published: 10/03/2013
• Est. Priority Date: 12/06/2010
• Status: Active Grant

• Current Assignee: 3shape A/S
• Original Assignee: 3shape A/S
• Inventors: jelund, Henrik, Fischer, David, Hollenbeck, Karl-Josef
• Granted Patent: US 9,329,675 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A61B 5/0088 for oral or dental tissue
• CPC Class Codes: A61C 9/004 Means or methods for taking digitized impressions
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Disclosed is a system comprising a handheld device and at least one display, where the handheld device is adapted for performing at least one action in a physical 3D environment, where the at least one display is adapted for visually representing the physical 3D environment, and where the handheld device is adapted for remotely controlling the view with which the 3D environment is represented on the display.

Claims

1. A system comprising a handheld device and at least one display, where the handheld device is adapted for switching between

performing at least one action in a physical 3D environment, where the at least one display is adapted for visually representing the physical 3D environment, and

remotely controlling the view with which the 3D environment is represented on the display.

2. A system according to any one or more of the preceding claims, wherein the handheld device is adapted to record the 3D geometry of the 3D environment.

3. A system according to any one or more of the preceding claims, wherein means for manually switching between performing the at least one action and remotely controlling the view is provided on the handheld device.

4. A system according to claim 3, wherein the means for manually switching is an actuator, such as a button, a switch or a contact.

5. The system according to any one or more of the preceding claims, wherein the view is defined as viewing angle and/or viewing position.

6. The system according to any one or more of the preceding claims, wherein the handheld device is adapted for remotely controlling the magnification with which the 3D environment is represented on the display.

7. The system according to any one or more of the preceding claims, wherein the handheld device is adapted for changing the rendering of the 3D environment on the display.

8. The system according to any one or more of the preceding claims, wherein the at least one action comprises one or more of:

measuring,

recording,

scanning,

manipulating, and/or

modifying.

9. The system according to any one or more of the preceding claims, wherein the 3D environment comprises one or more 3D objects.

10. The system according to any one or more of the preceding claims, wherein the handheld device is adapted to be held in one hand by an operator.

11. The system according to any one or more of the preceding claims, wherein the display is adapted to represent the 3D environment from multiple views.

12. The system according to any one or more of the preceding claims, wherein the handheld device comprises at least one motion sensor.

13. The system according to any one or more of the preceding claims, wherein the view of the 3D environment represented in the at least one display is at least partly determined by the at least one motion sensor.

14. The system according to any one or more of the preceding claims, wherein the magnification represented in the at least one display is at least partly determined by the at least one motion sensor.

15. The system according to any one or more of the preceding claims, wherein the handheld device is adapted to record the 3D geometry of the 3D environment.

16. The system according to any one or more of the preceding claims, wherein the 3D geometry of the 3D environment is known a-priori.

17. The system according to any one or more of the preceding claims, wherein the handheld device comprises at least one user-interface element.

18. The system according to any one or more of the preceding claims, wherein the at least one user-interface element is at least one motion sensor.

19. The system according to any one or more of the preceding claims, wherein the handheld device comprises at least two motion sensors providing sensor fusion.

20. The system according to any one or more of the preceding claims, wherein the user interface functionality comprises the use of gestures.

21. The system according to any one or more of the preceding claims, wherein the gestures are detected by the at least one motion sensor.

22. The system according to any one or more of the preceding claims, wherein the handheld device comprises at least one user-interface element other than the at least one motion sensor.

23. The system according to any one or more of the preceding claims, wherein the at least one other user-interface element is a touch-sensitive element.

24. The system according to any one or more of the preceding claims, wherein the at least one other user-interface element is a button.

25. The system according to any one or more of the preceding claims, wherein the at least one other user-interface element is a scroll wheel.

26. The system according to any one or more of the preceding claims, wherein the handheld device is adapted to change a viewing angle with which the 3D environment is represented on the at least one display.

27. The system according to any of the preceding claims, wherein the handheld device is adapted to change a magnification factor with which the 3D environment is represented on the at least one display.

28. The system according to any one or more of the preceding claims, wherein the handheld device is adapted to change a viewing position with which the 3D environment is represented on the at least one display.

29. The system according to any one or more of the preceding claims, wherein the view of the 3D environment comprises a viewing angle, a magnification factor, and/or a viewing position.

30. The system according to any one or more of the preceding claims, wherein the view of the 3D environment comprises rendering of texture and/or shading.

31. The system according to any one or more of the preceding claims, wherein the at least one display is divided into multiple regions, each showing the 3D environment with a different view.

32. The system according to any one or more of the preceding claims, wherein the 3D geometry comprises a 3D surface of the environment.

33. The system according to any one or more of the preceding claims, wherein the 3D geometry comprises a 3D volumetric representation of the environment.

34. The system according to any one or more of the preceding claims, wherein the handheld device is an intra-oral 3D scanner.

35. The system according to any one or more of the preceding claims, wherein the handheld device is a surgical instrument.

36. The system according to any one or more of the preceding claims, wherein the handheld device is a mechanical tool.

37. The system according to any one or more of the preceding claims, wherein the handheld device is an in-ear 3D scanner.

38. The system according to any one or more of the preceding claims, wherein the at least one display is arranged separate from the handheld device.

39. The system according to any one or more of the preceding claims, wherein the at least one display is arranged on a cart.

40. The system according to any one or more of the preceding claims, wherein the at least one display is defined as a first display, and where the system further comprises a second display.

41. The system according to any one or more of the preceding claims, wherein the second display is arranged on the handheld device.

42. The system according to any one or more of the preceding claims, wherein the second display is arranged on the handheld device in a position such that the display is adapted to be viewed by the operator, while the operator is operating the handheld device.

43. The system according to any one or more of the preceding claims, wherein the second display indicates where the handheld device is positioned relative to the 3D environment.

44. The system according to any one or more of the preceding claims, wherein the first display and/or the second display provides instructions for the operator.

45. The system according to any one or more of the preceding claims, wherein visual information is provided to the operator on one or more means other than the first display.

46. The system according to any one or more of the preceding claims, wherein audible information to the operator is provided to the operator.

47. The system according to any one or more of the preceding claims, wherein the scanning is performed by means of LED scanning, laser light scanning, white light scanning, X-ray scanning, and/or CT scanning.

48. The system according to any one or more of the preceding claims, wherein a graphical pointer is provided on the display for representing the movement of the handheld device when the handheld device is used to remotely controlling the view with which the 3D environment is represented on the display.

49. The system according to any one or more of the preceding claims, wherein the handheld device is an intra-oral 3D scanner and the at least one action performed in the physical 3D environment is scanning and that the view is remotely controlled by at least one motion sensor arranged in the handheld device, and wherein an actuator provided on the handheld device switches between performing the at least one action and remotely controlling the view.

51. A computer program product comprising program code means for causing a data processing system to perform the method of claim 44, when said program code means are executed on the data processing system.

52. A computer program product according to the previous claim, comprising a computer-readable medium having stored there on the program code means.

50. A method of interaction between a handheld device and at least one display, where the method comprises the steps of:

performing at least one action in a physical 3D environment by means of the handheld device;

visually representing the physical 3D environment by the at least one display; and

remotely controlling the view of the represented 3D environment on the display by means of the handheld device.