Method and apparatus for individualized three dimensional display calibration

• US 10,368,059 B2
• Filed: 10/02/2015
• Issued: 07/30/2019
• Est. Priority Date: 10/02/2015
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Fateh, Sina, Abercrombie, Nathan, Itani, Sleiman
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Dang, Philip P.
• Application Number:
  US14/874,313
• Publication Number:
  US 20170099482A1
• Time in Patent Office: 1,397 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/002 Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Abstract

A target is outputted to an ideal position in 3D space. A viewer indicates the apparent position of the target, and the indication is sensed. An offset between the ideal and apparent positions is determined, and an adjustment determined from the offset such that the apparent position of the ideal position with the adjustment matches the ideal position without the adjustment. The adjustment is made to the first entity and/or a second entity, such that the entities appear to the viewer in the ideal position. The indication may be monocular with a separate indication for each eye, or binocular with a single viewer indication for both eyes. The indication also may serve as communication, such as a PIN input, so that calibration is transparent to the viewer. The method may be continuous, intermittent, or otherwise ongoing over time.

Claims

1. A method, comprising:

defining, by a processor, a first coordinate in three-dimensional (3D) space to display a virtual object to a viewer, wherein the three-dimensional space is an augmented reality environment or a virtual reality environment;

displaying, by a display, the virtual object at the first coordinate in the 3D space;

sensing, by a sensor, an end-effector interacting with the virtual object when the virtual object is displayed at the first coordinate;

determining, by the processor, that the end-effector is pointing to a second coordinate in the 3D space that is different than the first coordinate where the virtual object is displayed;

determining, by the processor, an offset value between the first coordinate where the virtual object is located and the second coordinate where the end-effector is pointing to, wherein the offset value indicates a difference in a first coordinate and the second coordinate;

determining, by the processor, a third coordinate in the 3D space, wherein the third coordinate is the first coordinate of the first coordinate adjusted by the offset value so that the viewer perceives the virtual object as being located at the first coordinate in the 3D space;

displaying, by the display, the virtual object at the third coordinate in the 3D space;

sensing, by the sensor, the end-effector interacting with the virtual object when the virtual object is displayed at the third coordinate;

determining, by the processor, whether the end-effector is pointing to the first coordinate in the 3D space when the virtual object is displayed at the third coordinate;

in response to the end-effector pointing to the first coordinate in the 3D space, continuing to display the virtual object at the third coordinate; and

in response to the end-effector pointing to a coordinate in the 3D space that is not the first coordinate, iteratively adjusting, by the processor, a coordinate that the virtual object is displayed at until the end-effector points to the first coordinate.

2. The method of claim 1, wherein displaying the virtual object as the first coordinate further comprises displaying the virtual object to a first display and a second display of a stereo display pair.

3. The method of claim 1, wherein displaying the virtual object in the first coordinate further comprises:

displaying the virtual object at a first location at a first display of a stereo display pair; and

displaying the virtual object in a second location at a second display of the stereo display pair.

4. The method of claim 2, wherein the first coordinate is a first two-dimensional coordinate and the second coordinate is a second two dimensional coordinate.

5. The method of claim 1, wherein displaying the first virtual object at the first coordinate comprises:

displaying the virtual object at a first location to a first eye of the viewer but not a second eye of the viewer; and

displaying the virtual object at a second location to the second eye of the viewer but not the first eye of the viewer.

6. The method of claim 5, wherein displaying the virtual object at the first coordinate further comprises displaying the virtual object substantially simultaneously to the first eye and the second eye.

7. The method of claim 5, wherein displaying the virtual object at the first coordinate further comprises displaying the virtual object sequentially to the first eye and the second eye.

8. The method of claim 7, wherein displaying the virtual object at the first coordinate further comprises:

displaying substantially nothing to the second eye while displaying the virtual object to the first eye; and

displaying substantially nothing to the first eye while displaying the virtual object to the second eye.

9. The method of claim 7, wherein displaying the virtual object at the first coordinate further comprises:

substantially obstructing the second eye while displaying the virtual object to the first eye; and

substantially obstructing the first eye while displaying the virtual object to the second eye.

10. The method of claim 1, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises sensing at least one of a posture or a gesture of the viewer.

11. The method of claim 1, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises sensing, by the sensor, the viewer substantially aligning an end-effector with the second location of the virtual object.

12. The method of claim 6, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises:

sensing, by the sensor, the viewer substantially aligning a first end-effector with the first coordinate of the virtual object; and

sensing, by the sensor, the viewer substantially aligning the first end-effector with the second coordinate of the virtual object, substantially simultaneously to the viewer substantially aligning the first end-effector with the first coordinate of the virtual object.

13. The method of claim 7, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises:

sensing, by the sensor, the viewer substantially aligning a first end-effector with the first coordinate of the virtual object; and

sensing, by the sensor, the viewer substantially aligning the first end-effector with the second coordinate of the virtual object.

14. The method of claim 7, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises:

sensing, by the sensor, the viewer substantially aligning a first end-effector with the first coordinate of the virtual object; and

sensing, by the sensor, the viewer substantially aligning a second end-effector with the second coordinate of the virtual object.

15. The method of claim 1, wherein the virtual object comprises at least one of a virtual reality object or an augmented reality object.

16. The method of claim 1, comprising outputting the virtual object to a stereo display.

17. The method of claim 1, comprising outputting the virtual object to a see-through display.

18. The method of claim 1, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space through stereo imaging.

19. The method of claim 1, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space through depth imaging.

20. The method of claim 1, wherein determining that the viewer perceives the virtual object as being located at the second coordinate that is different than the first coordinate in the 3D space further comprises receiving a viewer communication.

21. The method of claim 20, wherein the viewer communication comprises at least one of:

activating a device in communication with the processor;

activating a data entity comprising executable instructions instantiated by the processor;

waking the device in communication with the processor;

waking the data entity comprising executable instructions instantiated by the processor;

unlocking the device in communication with the processor;

unlocking the data entity comprising executable instructions instantiated by the processor;

addressing the device in communication with the processor;

addressing the data entity comprising executable instructions instantiated by the processor;

identifying a user of the device in communication with the processor;

identifying the user of the data entity comprising executable instructions instantiated by the processor;

entering a security verification for the device in communication with the processor; and

entering the security verification for the data entity comprising executable instructions instantiated by the processor.

22. A method, comprising:

defining, by a processor, a first coordinate in an augmented reality environment or a virtual reality environment to display a first virtual object to a viewer;

displaying, by a display, the first virtual object at the first coordinate in the augmented reality environment or the virtual reality environment;

sensing, by a sensor, an end-effector interacting with the virtual object when the virtual object is displayed at the first coordinate;

determining, by the processor, that the end-effector is pointing to a second coordinate in the 3D space that is different than the first coordinate where the virtual object is displayed;

determining, by the processor, an offset value between the first coordinate where the virtual object is located and the second coordinate where the end-effector is pointing to, wherein the offset value indicates a difference in a first coordinate and the second coordinate

determining, by the processor, a third coordinate in the augmented reality environment or the virtual reality environment, wherein the third coordinate is the first coordinate of the first coordinate adjusted by the offset value so that the viewer perceives the virtual object as being located at the first coordinate in the augmented reality environment or the virtual reality environment;

generating a second virtual object that is different than the first virtual object;

defining, by the processor, a fourth coordinate in the augmented reality environment or the virtual reality environment to display the second virtual object to the viewer;

adjusting the fourth coordinate that the second virtual object is displayed at based on the offset value;

displaying, by the display, the first virtual object at the first coordinate in the augmented reality environment or the virtual reality environment; and

displaying the second virtual object at the adjusted fourth coordinate.

23. The method of claim 22, comprising sequentially displaying the first virtual object and the second virtual object.

24. The method of claim 22, comprising:

displaying the first virtual object by a first display while outputting substantially nothing by a second display; and

displaying the second virtual object by the second display while outputting substantially nothing by the first display.

25. The method of claim 22, comprising:

displaying the first virtual object by a first display while substantially obstructing a second display; and

displaying the second virtual object by the second display while substantially obstructing the first display.

26. The method of claim 22, further comprising sensing a viewer first indication of the first virtual object and a viewer second indication of the second virtual object.

27. The method of claim 22, wherein displaying the first virtual object and the second virtual object at the third coordinate further comprises displaying the first virtual object by a first display and substantially simultaneously displaying the second virtual object by a second display.

28. The method of claim 22, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises sensing substantially simultaneous a perceived location of the first virtual object and the second virtual object.

29. The method of claim 22, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises sensing at least one of a viewer posture or a viewer gesture.

30. The method of claim 22, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises sensing a first end-effector aligning with the second coordinate of the first virtual object.

31. The method of claim 30, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises sensing the viewer substantially aligning the first end-effector with the second coordinate of the first virtual object and substantially simultaneously sensing the viewer substantially aligning the first end-effector with the first coordinate of the first virtual object.

32. The method of claim 30, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises:

sensing the viewer substantially aligning the first end-effector with the first coordinate of the first virtual object; and

sensing the viewer substantially aligning the first end-effector with the second coordinate of the first virtual object.

33. The method of claim 30, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises:

sensing the viewer substantially aligning the first end-effector with the first coordinate of the first virtual object; and

sensing the viewer substantially aligning a second end-effector with the second coordinate of the first virtual object.

34. The method of claim 30, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate further comprises:

sensing the viewer substantially aligning the first end-effector with the first coordinate of the first virtual object; and

sensing the viewer substantially aligning the first end-effector with the second coordinate of the first virtual object substantially simultaneously to the viewer substantially aligning the first end-effector with the first coordinate of the first virtual object.

35. The method of claim 22, wherein the first virtual object comprises at least one of a virtual reality object or an augmented reality object.

36. The method of claim 22, wherein the display comprises a see-through display.

37. The method of claim 22, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate that is different than the first coordinate in the augmented reality environment or the virtual reality environment further comprises determining that the viewer perceives the first virtual object as being located at the second coordinate that is different than the first coordinate in the augmented reality environment or the virtual reality environment through stereo imaging.

38. The method of claim 22, wherein determining that the viewer perceives the first virtual object as being located at the second coordinate that is different than the first coordinate in the augmented reality environment or the virtual reality environment further comprises determining that the viewer perceives the first virtual object as being located at the second coordinate that is different than the first coordinate in the augmented reality environment or the virtual reality environment through depth imaging.

39. An apparatus comprising:

a first three-dimensional (3D) display operable to output a virtual object at a first coordinate in three-dimensional (3D) space to a first eye of a viewer;

a second 3D display operable to output the virtual object at a second coordinate in the 3D space to a second eye of the viewer, wherein the first coordinate is different than the first coordinate;

a first sensor configured to measure the first coordinate of an end-effector interacting with the virtual object relative to the first eye when the virtual object is displayed at the first coordinate;

a second sensor configured to measure the second coordinate of the end-effector interacting with the virtual object relative to the second eye when the virtual object is displayed at the second coordinate;

a processor coupled to the first 3D display, the second 3D display, the first sensor, and the second sensor, wherein the processor is configured to;

determine that the end-effector is pointing to a third coordinate on the first 3D display that is different than the first coordinate where the virtual object is displayed;

determine that the end-effector is pointing to a fourth coordinate on the second 3D display that is different than the second coordinate where the virtual object is displayed;

determine a first offset value between the first coordinate where the virtual object is located and the third coordinate where the end-effector is pointing to on the first display, wherein the first offset value indicates a difference in the first coordinate and the third coordinate;

determine a second offset value between the second coordinate where the virtual object is located and the fourth coordinate where the end-effector is pointing to on the second display, wherein the second offset value indicates a difference in the second coordinate and the fourth coordinate;

determine a fifth coordinate in the 3D space, wherein;

the fifth coordinate is the first coordinate adjusted by the first offset value so that the viewer perceives the virtual object as being located at the first coordinate on the first 3D display;

the first 3D display is to display the virtual object at the fifth coordinate on the first 3D display; and

determine a sixth coordinate in the 3D space, wherein;

the sixth coordinate is the second coordinate adjusted by the second offset value so that the viewer perceives the virtual object as being located at the second coordinate on the second 3D display; and

the second 3D display is to display the virtual object at the sixth coordinate on the second 3D display.

40. The apparatus of claim 39, wherein the first 3D display comprises a stereo display.

41. The apparatus of claim 39, wherein the first 3D display comprises a see-through display.

42. The apparatus of claim 39, wherein the first sensor or the second sensor comprises an imager.

43. The apparatus of claim 39, wherein the first sensor or the second sensor comprises a stereo imager pair.

44. The apparatus of claim 39, wherein the first sensor or the second sensor comprises a depth imager.

45. The apparatus of claim 39, wherein the processor, the display, and the sensor are disposed on a head-mounted display.

46. An apparatus, comprising:

a stereo display pair comprising a first display to display a virtual object to a first eye of a viewer at a first coordinate of the first display and a second display to display the virtual object to a second eye of the viewer at a second coordinate of the second display;

a sensor configured to;

measure the first coordinate an end-effector interacting with the virtual object relative to the first eye of when the virtual object is displayed at the first coordinate;

measure the second coordinate of the end-effector interacting with the virtual object relative to the second eye when the virtual object is displayed at the second coordinate; and

a processor coupled to the stereo display and the sensor, wherein the processor is configured to;

determine that the end-effector is pointing to a third coordinate on the first 3D display that is different than the first coordinate where the virtual object is displayed;

determine that the end-effector is pointing to a fourth coordinate on the second 3D display that is different than the second coordinate where the virtual object is displayed;

determine a first offset value between the first coordinate where the virtual object is located and the third coordinate where the end-effector is pointing to on the first display, wherein the first offset value indicates a difference in the first coordinate and the third coordinate;

determine a second offset value between the second coordinate where the virtual object is located and the fourth coordinate where the end-effector is pointing to on the second display, wherein the second offset value indicates a difference in the second coordinate and the fourth coordinate, and

simultaneously adjust the first coordinate by the first offset value and the second coordinate by the second offset value such that the first display and the second display simultaneously display the virtual object at the adjusted first coordinate and the adjusted second coordinate, respectively.

47. The apparatus of claim 46, wherein the first display and the second display comprise see-through displays.

48. The apparatus of claim 46, wherein the sensor comprises an imager.

49. The apparatus of claim 46, wherein the sensor comprises a stereo imager pair.

50. The apparatus of claim 46, wherein the sensor comprises a depth imager.

51. The apparatus of claim 46, wherein at least one of the virtual object is a calibration object.

52. The apparatus of claim 46, wherein the processor is further to combine the virtual object to form a 3D object.