Stereo-based calibration apparatus

• US 10,290,154 B1
• Filed: 09/12/2018
• Issued: 05/14/2019
• Est. Priority Date: 11/13/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Katz, Dov, Dolson, Jennifer Leigh, Hallam, Simon, Levin, Kieran Tobias, Vaughan, Eric Loren, Ivmark, Klas Petter, Melim, Andrew
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US16/128,527
• Time in Patent Office: 244 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A virtual reality (VR) headset calibration system calibrates a VR headset, which includes a plurality of locators and an inertial measurement unit (IMU) generating output signals indicative of motion of the VR headset. The system comprises a calibration controller configured to receive a headset model of the VR headset that identifies expected positions of each of the locators. The controller controls cameras to capture images of the VR headset while the headset is moved along a predetermined path. The images detect actual positions of the locators during the movement along the predetermined path. Calibration parameters for the locators are generated based on differences between the actual positions and the expected positions. Calibration parameters for the IMU are generated based on the calibration parameters for the locators and differences between expected and actual signals output by the IMU. The calibration parameters are stored to the VR headset.

Claims

1. A method comprising:

capturing a plurality of images of a virtual reality (VR) headset while the VR headset is rotated around one or more axes;

receiving a signal output by a inertial measurement unit (IMU) while the VR headset is rotated around the one or more axes;

calculating an expected rotation of the VR headset based on the received signal output by the IMU;

determining an actual rotation of the VR headset based on the plurality of captured images;

comparing the actual rotation of the VR headset and the expected rotation of the VR headset;

generating one or more dynamic calibration parameters based on the comparison; and

storing the dynamic calibration parameters for the VR headset, the calibration parameters usable to identify rotation of the VR headset during use of the VR headset.

2. The method of claim 1, wherein determining an actual rotation of the VR headset comprises:

determining positions of a plurality of a plurality of locators of the VR headset using at least one of the captured images; and

determining the actual rotation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

3. The method of claim 2, wherein determining an actual rotation of the VR headset based on the plurality of captured images further comprises:

determining a plurality of orientations of the VR headset, each orientation determined based on at least one of the captured images; and

determining the actual rotation of the VR headset based on the determined orientations.

4. The method of claim 2, wherein determining an actual rotation of the VR headset based on the plurality of captured images further comprises:

determining a first position of the plurality of locators on the VR headset based on a first subset of captured images;

determining a second position of the plurality of locators on the VR headset based on a second subset of captured images;

determining a difference between the first position and the second position; and

determining the actual rotation of the VR headset based on the determined difference.

5. The method of claim 1, further comprising:

determining expected accelerometer outputs based on the plurality of captured images; and

determining the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

6. The method of claim 5, further comprising:

determining angles between accelerometer channels; and

comparing the determined angles and the actual rotation of the VR headset.

7. The method of claim 1, wherein the plurality of images are captured by two or more cameras.

8. A non-transitory computer readable storage medium storing instructions, the instructions, when executed by a processor, cause the processor to:

capture a plurality of images of a virtual reality (VR) headset while the VR headset is rotated around one or more axes;

receive a signal output by a inertial measurement unit (IMU) while the VR headset is rotated around the one or more axes;

calculate an expected rotation of the VR headset based on the received signal output by the IMU;

determine an actual rotation of the VR headset based on the plurality of captured images;

compare the actual rotation of the VR headset and the expected rotation of the VR headset;

generate one or more dynamic calibration parameters based on the comparison; and

store the dynamic calibration parameters for the VR headset, the calibration parameters usable to identify rotation of the VR headset during use of the VR headset.

9. The non-transitory computer readable storage medium of claim 8, wherein the instructions for determining an actual orientation of the VR headset cause the processor to:

determine positions of a plurality of a plurality of locators of the VR headset using at least one of the captured images; and

determine the actual rotation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

10. The non-transitory computer readable storage medium of claim 9, wherein the instructions for determining an actual orientation of the VR headset based on the plurality of captured images further cause the processor to:

determine a plurality of orientations of the VR headset, each orientation determined based on at least one of the captured images; and

determine the actual rotation of the VR headset based on the determined orientations.

11. The non-transitory computer readable storage medium of claim 9, wherein the instructions for determining an actual orientation of the VR headset based on the plurality of captured images further cause the processor to:

determine a first position of the plurality of locators on the VR headset based on a first subset of captured images;

determine a second position of the plurality of locators on the VR headset based on a second subset of captured images;

determine a difference between the first position and the second position; and

determine the actual rotation of the VR headset based on the determined difference.

12. The non-transitory computer readable storage medium of claim 8, wherein the instructions further cause the processor to:

determine expected accelerometer outputs based on the plurality of captured images; and

determine the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

13. The non-transitory computer readable storage medium of claim 12, wherein the instructions further cause the processor to:

determine angles between accelerometer channels; and

compare the determined angles and the actual rotation of the VR headset.

14. The non-transitory computer readable storage medium of claim 8, wherein the plurality of images are captured by two or more cameras.

15. A system comprising:

two or more cameras; and

a calibration controller configured to;

capture a plurality of images of a virtual reality (VR) headset while the VR headset is rotated around one or more axes;

receive a signal output by a inertial measurement unit (IMU) while the VR headset is rotated around the one or more axes;

calculate an expected rotation of the VR headset based on the received signal output by the IMU;

determine an actual rotation of the VR headset based on the plurality of captured images;

compare the actual rotation of the VR headset and the expected rotation of the VR headset;

generate one or more dynamic calibration parameters based on the comparison; and

store the dynamic calibration parameters for the VR headset, the calibration parameters usable to identify rotation of the VR headset during use of the VR headset.

16. The system of claim 15, wherein determining an actual orientation of the VR headset comprises:

determining positions of a plurality of a plurality of locators of the VR headset using at least one of the captured images; and

determining the actual rotation of the VR headset based on the determined positions of the plurality of locators of the VR headset.

17. The system of claim 16, wherein determining an actual orientation of the VR headset comprises:

determining a plurality of orientations of the VR headset, each orientation determined based on at least one of the captured images; and

determining the actual rotation of the VR headset based on the determined orientations.

18. The system of claim 16, wherein determining an actual rotation of the VR headset based on the plurality of captured images further comprises:

determining a first position of the plurality of locators on the VR headset based on a first subset of captured images;

determining a second position of the plurality of locators on the VR headset based on a second subset of captured images;

determining a difference between the first position and the second position; and

determining the actual rotation of the VR headset based on the determined difference.

19. The system of claim 15, wherein the calibration controller is further configured to:

determine expected accelerometer outputs based on the plurality of captured images; and

determine the calibration parameters based on a difference between the expected accelerometer outputs and the received signal output of the IMU.

20. The system of claim 19, wherein the calibration controller is further configured to:

determining angles between accelerometer channels; and

comparing the determined angles and the actual rotation of the VR headset.