RECALIBRATION OF A FLEXIBLE MIXED REALITY DEVICE

• US 20130120224A1
• Filed: 11/11/2011
• Published: 05/16/2013
• Est. Priority Date: 11/11/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cajigas, Elmer S., O'Brien, Paul M., Tantos, Andras
• Granted Patent: US 9,311,883 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

The technology provides embodiments for recalibration of outward facing cameras supported by a see-through, head mounted, mixed reality display system having a flexible portion between see-through displays for the eyes. Each outward facing camera has a fixed spatial relationship with a respective or corresponding see-through display positioned to be seen through by a respective eye. For front facing cameras, the fixed spatial relationship allows a predetermined mapping between positions on an image sensor of each camera and positions on the respective display. The mapping may be used to register a position of a virtual object to a position of a real object. A change in a first flexible spatial relationship between the outward facing cameras can be automatically detected. A second spatial relationship between the cameras is determined. A registration of a virtual object to a real object may be updated based on the second spatial relationship.

Claims

1. A method of recalibrating a see-through, head mounted, mixed reality display system having a flexible frame portion between see-through displays comprising:

detecting a change in a first spatial relationship between outward facing cameras supported by the see-through, head mounted, mixed reality display system, each outward facing camera having a fixed spatial relationship with respect to a respective see-through display;

determining a second spatial relationship between the outward facing cameras based on the change in the first spatial relationship between the outward facing cameras; and

automatically updating processing of image data based on the second spatial relationship.

2. The method of claim 1 wherein detecting a change in the first spatial relationship between the outward facing cameras of the see-through, head mounted mixed reality display system is based on performing photogrammetry on overlapping image data from the outward facing cameras.

3. The method of claim 1 wherein detecting a change in the first spatial relationship between the outward facing cameras of the see-through, head mounted, mixed reality display system is based on motion sensor data indicating different changes in position of one camera from that of another beyond a threshold.

4. The method of claim 1 wherein determining the second spatial relationship between the outward facing cameras based on the change in the first spatial relationship further comprises determining a scale for image data based on a predetermined geometrical measurement of an object in a field of view of the see-through, head mounted, mixed reality display system.

5. The method of claim 1 wherein determining the second spatial relationship between the outward facing cameras based on the change in the first spatial relationship further comprises determining a geometrical measurement of a real object in a field of view of the see-through, head mounted, mixed reality display system based on depth data.

6. The method of claim 1 wherein the change in the first spatial relationship between the outward facing cameras includes a rotational change.

7. The method of claim 1 wherein the change in the first spatial relationship between the outward facing cameras includes a translational change.

8. A see-through, head mounted, mixed reality display system comprising:

a frame including a flexible frame portion positioned between see-through displays positioned in the frame and outward facing cameras supported by the frame;

each see-through display being positioned in the frame to be seen through by a respective eye;

each outward facing camera for capturing image data of a real object in a field of view of the respective see-through display and having a fixed spatial relationship with respect to its respective see-through display for providing a fixed mapping from position coordinates on an image sensor of the outward facing camera to positions coordinates on the respective see-through display;

the outward facing cameras having a first spatial relationship;

a memory for storing the image data of the real object and the first spatial relationship and the fixed mapping for each outward facing camera and its respective see-through display; and

one or more processors having access to the memory and being communicatively coupled to the cameras and see-through displays for automatically updating a registration between a position of the real object and a position of a virtual object responsive to a change in the first spatial relationship between the outward facing cameras.

9. The see-through, head mounted, mixed reality display system of claim 8 wherein the flexible frame portion is a flexible nose bridge section capable of rotating and translating.

10. The see-through, head mounted, mixed reality display system of claim 8 further comprising:

one or more motion sensing units for automatically detecting a change in the first spatial relationship between the outward facing cameras of the see-through, head mounted, mixed reality display.

11. The see-through, head mounted mixed reality display system of claim 8 further comprising:

the one or more processors automatically determining a change in the first spatial relationship between the outward facing cameras based on detecting an area size change in overlapping image data from each of the outward facing cameras.

12. The see-through, head mounted, mixed reality display of claim 11 further comprising:

a depth camera for providing depth data of a real object wherein the one or more processors determine a fixed geometrical measurement with respect to the real object for defining a scale for image data; and

the one or more processors perform photogrammetry based on the scale determined using the fixed geometrical measurement.

13. The see-through, head mounted, mixed reality display system of claim 8 further comprising:

a network interface communicatively coupled to the one or more processors for accessing a computer system with access to one or more databases of images of objects of standard size over a network;

the one or more processors sending image data from the outward facing cameras to the computer system and requesting an identification of any real object in the image data which matches an object of standard size; and

responsive to receiving an identification of a standard size real object in the image data, the one or more processors perform photogrammetry using a predetermined fixed geometrical measurement of the standard size real object.

14. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method of recalibrating a see-through, head mounted, mixed reality display system having a flexible frame portion between see-through displays, the method comprising:

automatically detecting a change in a first spatial relationship between outward facing cameras supported by the see-through, head mounted, mixed reality display system, each outward facing camera having a fixed spatial relationship with respect to a corresponding see-through display for providing a fixed mapping from position coordinates on an image sensor of the outward facing camera to position coordinates on the corresponding see-through display;

determining a second spatial relationship between the outward facing cameras based on photogrammetry with respect to overlapping image data captured by the outward facing cameras and a scale based on a predetermined geometrical measurement in a field of view of the outward facing cameras; and

updating a registration between a position of a real object and a position of a virtual object on at least one see-through display based on the second spatial relationship.

15. The one or more processor readable storage devices of claim 14 wherein determining a second spatial relationship between the outward facing cameras based on photogrammetry with respect to a overlapping image data captured by the outward facing cameras and a scale based on a predetermined geometrical measurement in a field of view of the see-through displays further comprises:

identifying one or more predetermined lengths in image data captured by the outward facing cameras;

determining a current scale for the image data based on the one or more predetermined lengths;

determining the second spatial relationship based on photogrammetry using triangulation based on image data including overlapping image data from the outward facing cameras; and

determining any translation change and any rotation change of each camera based on a change of the second spatial relationship from the first spatial relationship.

16. The one or more processor readable storage devices of claim 15 wherein automatically detecting a change in a first spatial relationship between outward facing cameras supported by the see-through, head mounted, mixed reality display system further comprises:

performing a calibration check based on triangulation with respect to overlapping image data captured by the outward facing cameras; and

determining whether the calibration check indicates a change in the first spatial relationship between the outward facing cameras.

17. The one or more processor readable storage devices of claim 14 wherein updating a registration between a position of a real object and a position of a virtual object on at least one see-through display based on the second spatial relationship further comprises:

identify a position of a real object to which a virtual object is registered on an image sensor of a respective outward facing camera;

map the position of the real object on the image sensor to a position in display coordinates on the corresponding see-through display using the fixed mapping;

based on any translation change of the first spatial relationship between the cameras and motion sensor data for the respective outward facing camera, determine the translation change for the respective outward camera in a coordinate system for the position of the display;

adjust the display coordinates of the real object in one or more directions to accommodate for movement of the display by the translation change for the respective outward camera; and

update display coordinates of a virtual object registered to the real object based on the adjusted display coordinates of the real object.

18. The one or more processor readable storage devices of claim 14 wherein updating a registration between a position of a real object and a position of a virtual object on at least one see-through display based on the second spatial relationship further comprises:

determining a rotation change for the respective outward camera based on any rotation change of the first spatial relationship between the cameras and motion sensor data for the respective outward facing camera;

determining a skew angle with respect to each axis defined in the coordinate system for the position of the display caused by the rotation change for the respective outward camera; and

updating display coordinates of the virtual object registered to the real object based on the skew angles.

19. The one or more processor readable storage devices of claim 14 wherein the method further comprises selecting a real object as a calibration object based on depth image data by identifying a real object in an overlap area of depth image data having one or more edges which maintain size over a time period for the same focal length;

and

obtaining a length measurement of one or more edges of the real object in any of three dimensions.

20. The one or more processor readable storage devices of claim 14 wherein the method further comprises

identifying one or more real objects in image data;

performing pattern recognition for identifying whether any of the one or more real objects is a standard size object; and

responsive to identifying a real object as a standard size object, selecting a predetermined length of the standard size object for use in determining the second spatial relationship.