Stereo-based calibration apparatus

• US 10,127,732 B1
• Filed: 09/29/2017
• Issued: 11/13/2018
• Est. Priority Date: 11/13/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Katz, Dov, Dolson, Jennifer Leigh, Hallam, Simon, Levin, Kieran Tobias, Vaughan, Eric Loren, Ivmark, Klas Petter, Melim, Andrew
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US15/721,603
• Time in Patent Office: 410 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A virtual reality (VR) headset calibration system calibrates a VR headset, which includes a plurality of locators and an inertial measurement unit (IMU) generating output signals indicative of motion of the VR headset. The system comprises a calibration controller configured to receive a headset model of the VR headset that identifies expected positions of each of the locators. The controller controls cameras to capture images of the VR headset while the headset is moved along a predetermined path. The images detect actual positions of the locators during the movement along the predetermined path. Calibration parameters for the locators are generated based on differences between the actual positions and the expected positions. Calibration parameters for the IMU are generated based on the calibration parameters for the locators and differences between expected and actual signals output by the IMU. The calibration parameters are stored to the VR headset.

Claims

1. A method comprising:

capturing a plurality of images of a virtual reality (VR) headset in a plurality of orientations;

for each orientation of the plurality of orientations of the VR headset;

receiving a signal output by an inertial measurement unit (IMU),

calculating expected orientation of the VR headset based on the received signal output by the IMU,

determining an actual orientation of the VR headset based on the plurality of captured images, and

comparing the actual orientation of the VR headset and the expected orientation of the VR headset;

generating calibration parameters for the plurality of orientations based on the comparison; and

storing the calibration parameters for the plurality of orientations of the VR headset, the calibration parameters usable to identify orientations of the VR headset during use of the VR headset.

2. The method of claim 1, wherein determining an actual orientation of the VR headset comprises:

determining positions of a plurality of locators of the VR headset using at least one of the captured images; and

determining the actual orientation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

3. The method of claim 2, wherein determining an actual orientation of the VR headset based on the plurality of captured images further comprises:

translating the determined positions of the plurality of locators of the VR headset based on translation calibration parameters.

4. The method of claim 3, further comprising:

accessing a memory that stores a set of expected positions of the plurality of locators on the VR headset;

capturing a second plurality of images of the VR headset in a plurality of positions as the VR headset is moved along a predetermined path;

using the second plurality of images captured at each position of the VR headset;

identifying a respective position of the VR headset,

determining positions of at least a subset of the locators based on the second plurality of images,

computing a difference between an actual position of each locator in the subset on the VR headset and the expected position of the respective locator based on the determined position of the locator and the identified position of the VR headset; and

generating the translation calibration parameters for the plurality of locators based on the computed differences.

5. The method of claim 1, further comprising:

determining expected accelerometer outputs based on the plurality of captured images; and

determining the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

6. The method of claim 5, further comprising:

determining angles between accelerometer channels; and

comparing the determined angles and the actual orientation of the VR headset.

7. The method of claim 1, wherein the plurality of images are captured simultaneously by two or more cameras.

8. A non-transitory computer readable storage medium storing instructions, the instructions, when executed by a processor, cause the processor to:

capture a plurality of images of a virtual reality (VR) headset in a plurality of orientations;

for each orientation of the plurality of orientations of the VR headset;

receive a signal output by an inertial measurement unit (IMU),

calculate expected orientation of the VR headset based on the received signal output by the IMU,

determine an actual orientation of the VR headset based on the plurality of captured images, and

compare the actual orientation of the VR headset and the expected orientation of the VR headset;

generate calibration parameters for the plurality of orientations based on the comparison; and

store the calibration parameters for the plurality of orientations of the VR headset, the calibration parameters usable to identify orientations of the VR headset during use of the VR headset.

9. The non-transitory computer readable storage medium of claim 8, wherein the instructions for determining an actual orientation of the VR headset cause the processor to:

determine positions of a plurality of locators of the VR headset using at least one of the captured images; and

determine the actual orientation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

10. The non-transitory computer readable storage medium of claim 9, wherein the instructions for determining an actual orientation of the VR headset based on the plurality of captured images further cause the processor to:

translate the determined positions of the plurality of locators of the VR headset based on translation calibration parameters.

11. The non-transitory computer readable storage medium of claim 10, wherein the instructions further cause the processor to:

access a memory that stores a set of expected positions of the plurality of locators on the VR headset;

capture a second plurality of images of the VR headset in a plurality of positions as the VR headset is moved along a predetermined path;

using the second plurality of images captured at each position of the VR headset;

identify a respective position of the VR headset,

determine positions of at least a subset of the locators based on the second plurality of images,

compute a difference between an actual position of each locator in the subset on the VR headset and the expected position of the respective locator based on the determined position of the locator and the identified position of the VR headset; and

generate the translation calibration parameters for the plurality of locators based on the computed differences.

12. The non-transitory computer readable storage medium of claim 8, wherein the instructions further cause the processor to:

determine expected accelerometer outputs based on the plurality of captured images; and

determine the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

13. The non-transitory computer readable storage medium of claim 12, wherein the instructions further cause the processor to:

determine angles between accelerometer channels; and

compare the determined angles and the actual orientation of the VR headset.

14. The non-transitory computer readable storage medium of claim 8, wherein the plurality of images are captured simultaneously by two or more cameras.

15. A system comprising:

two or more cameras; and

a calibration controller configured to;

capture a plurality of images of a virtual reality (VR) headset in a plurality of orientations;

for each orientation of the plurality of orientations of the VR headset;

receive a signal output by an inertial measurement unit (IMU),

calculate expected orientation of the VR headset based on the received signal output by the IMU,

determine an actual orientation of the VR headset based on the plurality of captured images, and

compare the actual orientation of the VR headset and the expected orientation of the VR headset;

generate calibration parameters for the plurality of orientations based on the comparison; and

store the calibration parameters for the plurality of orientations of the VR headset, the calibration parameters usable to identify orientations of the VR headset during use of the VR headset.

16. The system of claim 15, wherein determining an actual orientation of the VR headset comprises:

determining positions of a plurality of locators of the VR headset using at least one of the captured images; and

determining the actual orientation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

17. The system of claim 16, wherein determining an actual orientation of the VR headset based on the plurality of captured images further comprises:

translating the determined positions of the plurality of locators of the VR headset based on translation calibration parameters.

18. The system of claim 17, wherein the calibration controller is further configured to:

access a memory that stores a set of expected positions of the plurality of locators on the VR headset;

capture a second plurality of images of the VR headset in a plurality of positions as the VR headset is moved along a predetermined path;

using the second plurality of images captured at each position of the VR headset;

identify a respective position of the VR headset,

determine positions of at least a subset of the locators based on the second plurality of images,

compute a difference between an actual position of each locator in the subset on the VR headset and the expected position of the respective locator based on the determined position of the locator and the identified position of the VR headset; and

generate the translation calibration parameters for the plurality of locators based on the computed differences.

19. The system 15, wherein the calibration controller is further configured to:

determine expected accelerometer outputs based on the plurality of captured images; and

determine the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

20. The system of claim 19, wherein the calibration controller is further configured to:

determine angles between accelerometer channels; and

compare the determined angles and the actual orientation of the VR headset.