Apparatus and method for determining an absolute pose of a manipulated object in a real three-dimensional environment with invariant features

• US 7,826,641 B2
• Filed: 09/03/2009
• Issued: 11/02/2010
• Est. Priority Date: 01/30/2004
• Status: Active Grant

• Current Assignee: Electronic Scripting Products, Inc.
• Original Assignee: Electronic Scripting Products, Inc.
• Inventors: Alboszta, Marek, Mandella, Michael J., Gonzalez-Banos, Hector H.
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US12/584,402
• Publication Number:
  US 20100001998A1
• Time in Patent Office: 425 Days
• Field of Search: 382/103, 382/106, 382/107, 382/108, 382/113, 382120-123, 382/140, 382/168, 382175-180, 382181-189, 382/201, 382/232, 382/255, 382/274, 382/276, 382286-298, 382/305, 382312-321, 382/193, 345/179, 345/166, 345/156, 345/180
• US Class Current: 382/106
• CPC Class Codes: G01B 11/002 for measuring two or more coordinates
• CPC Class Codes: G01B 11/03 by measuring coordinates of points
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/17 using hand-held instruments

Abstract

An apparatus and method for optically inferring an absolute pose of a manipulated object in a real three-dimensional environment from on-board the object with the aid of an on-board optical measuring arrangement. At least one invariant feature located in the environment is used by the arrangement for inferring the absolute pose. The inferred absolute pose is expressed with absolute pose data (φ, θ, ψ, x, y, z) that represents Euler rotated object coordinates expressed in world coordinates (X_o, Y_o, Z_o) with respect to a reference location, such as, for example, the world origin. Other conventions for expressing absolute pose data in three-dimensional space and representing all six degrees of freedom (three translational degrees of freedom and three rotational degrees of freedom) are also supported. Irrespective of format, a processor prepares the absolute pose data and identifies a subset that may contain all or fewer than all absolute pose parameters. This subset is transmitted to an application via a communication link, where it is treated as input that allows a user of the manipulated object to interact with the application and its output.

Claims

1. An apparatus for processing absolute pose data derived from an absolute pose of a manipulated object in a real three-dimensional environment, said apparatus comprising:

a) at least one invariant feature in said real three-dimensional environment;

b) an optical measuring means for optically inferring said absolute pose from on-board said manipulated object using said at least one invariant feature and expressing said inferred absolute pose with absolute pose data (φ

, θ

, ψ

, x, y, z) representing Euler rotated object coordinates expressed in world coordinates (X_o, Y_o, Z_o) with respect to a reference location;

c) a processor for preparing said absolute pose data and identifying a subset of said absolute pose data; and

d) a communication link for transmitting said subset to an application.

2. The apparatus of claim 1, wherein said reference location is a world origin (0, 0, 0) of said world coordinates (X_o, Y_o, Z_o).

3. The apparatus of claim 1, wherein said at least one invariant feature comprises at least one high optical contrast feature.

4. The apparatus of claim 3, wherein said at least one high optical contrast feature comprises at least one predetermined light source.

5. The apparatus of claim 4, wherein said at least one predetermined light source is selected from the group consisting of screens, display pixels, fiber optical waveguides and light-emitting diodes.

6. The apparatus of claim 5, wherein said light-emitting diode is an infrared emitting diode.

7. The apparatus of claim 3, wherein said at least one high optical contrast feature comprises a retro-reflector.

8. The apparatus of claim 1, wherein said application comprises an output displayed to a user and said subset comprises input for interacting with said output.

9. The apparatus of claim 8, wherein said output comprises at least one visual element.

10. The apparatus of claim 9, wherein said at least one visual element is selected from the group consisting of images, graphics, text and icons.

11. The apparatus of claim 9, further comprising a display for displaying said visual elements.

12. The apparatus of claim 11, wherein said display comprises a touch sensitive display.

13. The apparatus of claim 8, wherein said output comprises audio elements.

14. The apparatus of claim 13, wherein said audio elements are selected from the group consisting of tones, tunes, musical compositions and alert signals.

15. The apparatus of claim 13, further comprising a speaker for playing said audio elements.

16. The apparatus of claim 1, wherein said optical measuring means comprises a light-measuring component having a lens and an optical sensor.

17. The apparatus of claim 16, wherein said optical sensor is a photodetector for sensing light from said at least one invariant feature.

18. The apparatus of claim 17, wherein said at least one invariant feature comprises at least one predetermined light source and said photodetector is a position-sensing device for determining a centroid of light flux from said at least one predetermined light source.

19. The apparatus of claim 18, wherein said at least one predetermined light source comprises a plurality of infrared emitting diodes.

20. The apparatus of claim 19, wherein said plurality of infrared emitting diodes is modulated in a predetermined modulation pattern.

21. The apparatus of claim 1, wherein said optical measuring means comprises an active illumination component for projecting a predetermined light into said real three-dimensional environment and for receiving a scattered portion of said predetermined light from said at least one invariant feature.

22. The apparatus of claim 21, wherein said active illumination component comprises a scanning unit having a scanning mirror for directing said predetermined light into said real three-dimensional environment.

23. The apparatus of claim 1, further comprising an auxiliary motion detection component.

24. The apparatus of claim 23, wherein said auxiliary motion detection component is an inertial sensing device.

25. The apparatus of claim 24, wherein said inertial sensing device is selected from the group consisting of gyroscopes and accelerometers.

26. The apparatus of claim 23, wherein said auxiliary motion detection component is selected from the group consisting of an optical flow measuring unit, a magnetic field measuring unit and an acoustic field measuring unit.

27. The apparatus of claim 1, wherein said manipulated object is selected from the group consisting of pointers, wands, remote controls, three-dimensional mice, game controls, gaming objects, jotting implements, surgical implements, three-dimensional digitizers, digitizing styluses, hand-held tools and utensils.

28. The apparatus of claim 27, wherein said gaming objects are selected from the group consisting of golfing clubs, tennis rackets, squash rackets, guitars, guns, knives, swords, spears, balls, clubs, bats, steering wheels, joysticks and flying controls.

29. The apparatus of claim 1, wherein said application is selected from the group consisting of reality simulations, remote control applications, cyber games, virtual worlds, searches, photography applications, audio applications and augmented realities.

30. A method for processing absolute pose data derived from an absolute pose of a manipulated object, said method comprising:

a) placing said manipulated object in a real three-dimensional environment with at least one invariant feature;

b) optically inferring said absolute pose from on-board said manipulated object using said at least one invariant feature and expressing said inferred absolute pose with absolute pose data (φ

, θ

, Ψ

, x, y, z) representing Euler rotated object coordinates expressed in world coordinates (X_o, Y_o, Z_o) with respect to a reference location;

c) preparing said absolute pose data using a processor;

d) identifying a subset of said absolute pose data; and

e) transmitting said subset to an application using a communication link.

31. The method of claim 30, wherein said manipulated object is undergoing a motion and said optical inferring is performed periodically at times t_i, such that said absolute pose data describes said motion.

32. The method of claim 30, wherein said at least one invariant feature comprises at least one predetermined light source and said method comprises modulating said at least one predetermined light source.

33. The method of claim 30, wherein said subset comprises at least one orientation parameter.

34. The method of claim 33, wherein said at least one orientation parameter comprises at least one Euler angle.

35. The method of claim 30, wherein said subset comprises at least one position parameter.

36. The method of claim 35, wherein said at least one position parameter comprises at least one Cartesian coordinate.

37. The method of claim 30, wherein said subset comprises all absolute pose data (φ

, θ

, ψ

, x, y, z) and said application comprises one-to-one motion mapping between said real three-dimensional space and a cyberspace of said application.

38. The method of claim 30, wherein said application provides an output and said subset is used for a user interaction selected from the group consisting of text input, modification of said output, re-arrangement of said output.

39. The method of claim 38, wherein said output comprises at least one element selected from the group consisting of audio elements and visual elements.

40. The method of claim 30, wherein said step of optically inferring comprises receiving light from said at least one invariant feature by an optical measuring component having an optical sensor and a lens.

41. The method of claim 30, wherein said step of optically inferring comprises projecting a predetermined light into said real three-dimensional environment containing said at least one invariant feature and receiving a scattered portion of said predetermined light beam from said at least one invariant feature.

42. The method of claim 41, wherein said step of projecting further comprises directing said predetermined light into said real three-dimensional environment by scanning.

43. The method of claim 30, further comprising interpolating said absolute pose with an auxiliary motion detection component.

44. The method of claim 43, wherein said auxiliary motion detection component comprises an inertial sensing device and said step of interpolating is performed when not optically inferring said absolute pose.

45. The method of claim 43, wherein said auxiliary motion detection component is selected from the group consisting of optical flow measuring units, magnetic field measuring units and acoustic field measuring units, and said step of interpolating is performed when not optically inferring said absolute pose.

46. The method of claim 30, wherein said application comprises representing a user by an avatar having a token corresponding to said manipulated object.

47. The method of claim 30, wherein said application superposes at least one virtual element onto said real three-dimensional environment.

48. The method of claim 47, wherein said at least one virtual element is rendered interactive with said manipulated object by said application.

49. The method of claim 47, wherein said virtual element comprises data about at least one aspect of said real three-dimensional environment.