Computer interface for manipulated objects with an absolute pose detection component

• US 10,191,559 B2
• Filed: 03/07/2018
• Issued: 01/29/2019
• Est. Priority Date: 01/30/2004
• Status: Active Grant

• Current Assignee: Electronic Scripting Products, Inc.
• Original Assignee: Electronic Scripting Products, Inc.
• Inventors: Mandella, Michael J., Gonzalez-Banos, Hector H., Alboszta, Marek
• Primary Examiner(s): Bhatnagar, Anand P
• Application Number:
  US15/914,797
• Publication Number:
  US 20180217673A1
• Time in Patent Office: 328 Days
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: G01B 21/04 by measuring coordinates of points
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/17 : using hand-held instruments

Abstract

A manipulated object using high optical contrast features and an on-board photodetector that detects light produced, reflected or scattered by the features and outputs data indicative thereof. The manipulated object uses one or more controllers to determine its position and/or orientation, including full pose, in the real three-dimensional environment based on data from the photodetector. Data from one or more auxiliary motion sensing devices, e.g., a relative motion sensor such as an inertial device or other auxiliary motion device relying on acoustics, optics or electromagnetic waves within or outside the visible spectrum, can be used to supplement the position and/or orientation data from the photodetector. The manipulated object can be embodied by any suitable device manipulated by a user, including a tablet computer or a phone.

Claims

1. A manipulated object cooperating with a first plurality of high optical contrast features disposed in a real three-dimensional environment, said manipulated object comprising:

a) a photodetector configured to detect said first plurality of high optical contrast features and generate photodetector data representative of the positions of said first plurality of high optical contrast features;

b) a controller configured to identify a derivative pattern of said first plurality of high optical contrast features from said photodetector data, wherein said derivative pattern is indicative of the position of said photodetector; and

c) at least one component selected from the group consisting of an auxiliary motion detection component, an active illumination component and a scanning component.

2. The manipulated object of claim 1, wherein said first plurality of high optical contrast features are invariant features within said real three-dimensional environment.

3. The manipulated object of claim 2, wherein said invariant features are disposed in a known pattern within said real three-dimensional environment.

4. The manipulated object of claim 1, wherein said derivative pattern is computed with at least one absolute pose recovery technique selected from the group consisting of geometric invariance, triangulation, ranging, path integration, detection of features in images and motion analysis.

5. The manipulated object of claim 4, wherein said at least one absolute pose recovery technique comprises geometric invariance computed under application of the rules of perspective geometry.

6. The manipulated object of claim 1, wherein said manipulated object is selected from the group consisting of a pointer, a wand, a remote control, a three-dimensional mouse, a game control, a gaming object, a jotting implement, a surgical implement, a three-dimensional digitizer, a digitizing stylus, a hand-held tool, a utensil, a phone, a tablet computer, a portable device, an ultra-mobile device, a PDA and a smart object.

7. The manipulated object of claim 6 deployed in an application selected from the group consisting of virtual reality and augmented reality.

8. The manipulated object of claim 1, wherein said auxiliary motion detection component is selected from the group consisting of accelerometers, gyroscopes, magnetic sensing elements, acoustic sensors and optical flow measuring units.

9. The manipulated object of claim 1, wherein said photodetector is selected from the group consisting of optical sensors, CCD sensors, CMOS sensors, sensors with an array of pixels and position-sensitive diodes.

10. A manipulated object cooperating with a first plurality of high optical contrast features disposed in a real three-dimensional environment, said manipulated object comprising:

a) a photodetector configured to detect said first plurality of high optical contrast features and generate photodetector data representative of the positions of said first plurality of high optical contrast features;

b) a controller configured to identify a derivative pattern of said first plurality of high optical contrast features from said photodetector data, wherein said derivative pattern is indicative of the orientation of said photodetector; and

c) at least one component selected from the group consisting of an auxiliary motion detection component, an active illumination component and a scanning component.

11. The manipulated object of claim 10, wherein said first plurality of high optical contrast features are invariant features within said real three-dimensional environment.

12. The manipulated object of claim 11, wherein said invariant features are disposed in a known pattern within said real three-dimensional environment.

13. The manipulated object of claim 10, wherein said derivative pattern is computed with at least one absolute pose recovery technique selected from the group consisting of geometric invariance, triangulation, ranging, path integration, detection of features in images and motion analysis.

14. The manipulated object of claim 13, wherein said at least one absolute pose recovery technique comprises geometric invariance computed under application of the rules of perspective geometry.

15. The manipulated object of claim 10, wherein said manipulated object is selected from the group consisting of a pointer, a wand, a remote control, a three-dimensional mouse, a game control, a gaming object, a jotting implement, a surgical implement, a three-dimensional digitizer, a digitizing stylus, a hand-held tool, a utensil, a phone, a tablet computer, a portable device, an ultra-mobile device, a PDA and a smart object.

16. The manipulated object of claim 15, deployed in an application selected from the group consisting of virtual reality and augmented reality.

17. The manipulated object of claim 10, wherein said auxiliary motion detection component is selected from the group consisting of accelerometers, gyroscopes, magnetic sensing elements, acoustic sensors and optical flow measuring units.

18. The manipulated object of claim 10, wherein said photodetector is selected from the group consisting of optical sensors, CCD sensors, CMOS sensors, sensors with an array of pixels and position-sensitive diodes.

19. A manipulated object cooperating with a first plurality of high optical contrast features disposed in a real three-dimensional environment, said manipulated object comprising:

a) an on-board photodetector configured to detect said first plurality of high optical contrast features and generate photodetector data representative of the positions of said first plurality of high optical contrast features;

b) a controller configured to identify a derivative pattern of said first plurality of high optical contrast features from said photodetector data, wherein said derivative pattern is indicative of the pose of said photodetector; and

c) at least one component selected from the group consisting of an auxiliary motion detection component, an active illumination component and a scanning component.

20. The manipulated object of claim 19, wherein said first plurality of high optical contrast features are invariant features within said real three-dimensional environment.

21. The manipulated object of claim 20, wherein said invariant features are disposed in a known pattern within said real three-dimensional environment.

22. The manipulated object of claim 19, wherein said derivative pattern is computed with at least one absolute pose recovery technique selected from the group consisting of geometric invariance, triangulation, ranging, path integration, detection of features in images and motion analysis.

23. The manipulated object of claim 22, wherein said at least one absolute pose recovery technique comprises geometric invariance computed under application of the rules of perspective geometry.

24. The manipulated object of claim 19, wherein said manipulated object is selected from the group consisting of a pointer, a wand, a remote control, a three-dimensional mouse, a game control, a gaming object, a jotting implement, a surgical implement, a three-dimensional digitizer, a digitizing stylus, a hand-held tool, a utensil, a phone, a tablet computer, a portable device, an ultra-mobile device, a PDA and a smart object.

25. The manipulated object of claim 24, deployed in an application selected from the group consisting of virtual reality and augmented reality.

26. The manipulated object of claim 19, wherein said auxiliary motion detection component is selected from the group consisting of accelerometers, gyroscopes, magnetic sensing elements, acoustic sensors and optical flow measuring units.

27. The manipulated object of claim 19, wherein said on-board photodetector is selected from the group consisting of optical sensors, CCD sensors, CMOS sensors, sensors with an array of pixels and position-sensitive diodes.