Systems and Methods for Generating Accurate Sensor Corrections Based on Video Input

• US 20150002663A1
• Filed: 04/10/2014
• Published: 01/01/2015
• Est. Priority Date: 06/28/2013
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Pan, Weibin, Hu, Liang
• Application Number:
  US14/250,193
• Publication Number:
  US 20150002663A1
• Time in Patent Office: Days
• US Class Current: 348/142
• CPC Class Codes: G01C 25/00 Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass testing, calibrating or compensating compasses G01C17/38
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

A portable device includes a sensor, a video capture module, a processor, and a computer-readable memory that stores instructions. When executed on the processor, the instructions operate to cause the sensor to generate raw sensor data indicative of a physical quantity, cause the video capture module to capture video imagery of a reference object concurrently with the sensor generating raw sensor data when the portable device is moving relative to the reference object, and cause the processor to calculate correction parameters for the sensor based on the captured video imagery of the reference object and the raw sensor data.

Claims

1. A portable device comprising:

a sensor;

a video capture module;

a processor; and

a computer-readable memory that stores instructions thereon, wherein the instructions, when executed by the processor, operate to;

cause the sensor to generate raw sensor data indicative of a physical quantity,

cause the video capture module to capture video imagery of a reference object concurrently with the sensor generating raw sensor data when the portable device is moving relative to the reference object, and

cause the processor to calculate correction parameters for the sensor based on the captured video imagery of the reference object and the raw sensor data.

2. The portable device of claim 1, wherein the instructions, when executed by the processor, further operate to identify the reference object as a standard real-world object having known geometric properties.

3. The portable device of claim 2, wherein the standard real-world object is a two-dimensional (2D) image on a 2D surface.

4. The portable device of claim 1, wherein the instructions, when executed by the processor, further operate to match the captured video imagery to a digital 3D model of the reference object, wherein:

the digital 3D model is stored in a database to which the portable device is coupled via a communication network, and

the digital 3D model specifies geometric properties of the reference object.

5. The portable device of claim 4, wherein to match the captured video imagery to the digital 3D model of the reference object, the instructions operate to transmit at least part of the captured video imagery to a reference object server coupled to the database, via the communication network.

6. The portable device of claim 4, wherein the instructions, when executed by the processor, further operate to generate an approximate position fix of the portable device for matching with a geolocation data of the digital 3D model.

7. The portable device of claim 1, wherein the sensor is one of:

(i) an accelerometer,

(ii) a gyroscope, or

(iii) a magnetometer.

8. The portable device of claim 1, wherein the instructions, when executed by the processor, further cause the processor to apply the correction parameters to subsequent raw sensor data output of the sensor.

9. The portable device of claim 1, wherein to calculate the correction parameters, the instructions operate to:

obtain geometric properties of the reference object,

apply a 3D reconstruction technique to the captured video imagery using the geometric properties of the reference object, and

calculate a plurality of position and orientation fixes of the portable device at respective times based on the captured video imagery.

10. The portable device of claim 9, wherein to calculate the correction parameters, the instructions operate to determine vector a and matrix C in x_raw=a+Cx, wherein:

vector x_raw represents raw sensor data,

the vector a represents to drift errors,

the matrix C represents cross-axis errors, and

x represents corrected raw sensor data;

wherein the instructions operate to determine the vector a and the matrix C using the plurality of position and orientation fixes of the portable device.

11. The portable device of claim 1, wherein the instructions, when executed by the processor, further operate to update the correction parameters periodically at a regular interval.

12. The portable device of claim 1, wherein the video capture module is configured to capture video imagery continuously while the portable device is operational.

13. A method implemented on one or more processors for efficiently developing sensor error corrections in a portable device having a sensor and a camera, the method comprising:

while the portable device is moving relative to a reference object, causing the sensor to generate raw sensor data indicative of a physical quantity;

causing the camera to capture a plurality of images of the reference object concurrently with the sensor generating the raw sensor data;

determining a plurality of position and orientation fixes of the portable device based on the plurality of images and geometric properties of the reference object; and

calculating correction parameters for the sensor using the plurality of position and orientation fixes and the raw sensor data.

14. The method of claim 13, the method further comprising transmitting the plurality of images to a reference object server via a communication network, wherein the reference object server matches the plurality of images to the reference object.

15. The method of claim 14, the method further comprising transmitting the raw sensor data and sensor information to the reference object server.

16. The method of claim 13, further comprising identifying the reference object as a standard real-world object having known geometric properties.

17. The method of claim 13, further comprising matching the plurality of images to a digital 3D model of the reference object, wherein the digital 3D model is stored in a database.

18. The method of claim 13, wherein matching the plurality of images to a digital 3D model of the reference object includes:

generating a set of one or more approximate positioning fixes of the portable device,

transmitting the set of one or more approximate positioning fixes to a reference object server via a communication network, and

receiving a geolocated digital 3D model of the reference object from the reference object server, wherein the geolocated digital 3D model is indicative of geometric properties of the reference object.

19. A tangible computer-readable medium storing thereon instructions that, when executed on or more processors, cause the one or more processors to:

receive raw sensor data generated by a sensor operating in a portable device;

receive video imagery of a reference object captured by a video capture module operating in the portable device, wherein the raw sensor data and the video imagery are captured concurrently while the portable device is moving relative to the reference object;

calculate correction parameters for the sensor using the captured video imagery of the reference object and the raw sensor data.

20. The computer-readable medium of claim 19, wherein to calculate the correction parameters, the instructions cause the one or more processors to:

determine geometric properties of the reference object,

determine position and orientation fixes of the portable device based on the geometric properties of the reference object and the video imagery,

determine correct sensor data corresponding to the raw sensor data based on the determined position and orientation fixes, and

calculate the correction parameters based on a difference between the correct sensor data and the raw sensor data.

21. The computer-readable medium of claim 20, wherein:

the sensor is an accelerometer, and

to calculate the correction parameters, the instructions cause the one or more processors to calculate average acceleration based on the plurality of position fixes.

22. The computer-readable medium of claim 20, wherein:

the sensor is a gyroscope, and

to calculate the correction parameters, the instructions cause the one or more processors to calculate a numerical derivative of a time-dependent rotation matrix associated with the plurality of orientation fixes.

23. The computer-readable medium of claim 20, wherein to determine the position and orientation fixes of the portable device, the instructions cause the one or more processors to apply 3D reconstruction.

24. The computer-readable medium of claim 19, wherein the movement of the portable device relative to the reference object includes a change in at least one of position and orientation relative to the reference object.