Registration for vehicular augmented reality using auto-harmonization

• US 9,766,075 B2
• Filed: 05/01/2015
• Issued: 09/19/2017
• Est. Priority Date: 05/02/2014
• Status: Active Grant

• Current Assignee: Thales Visionix Incorporated (Thales SA)
• Original Assignee: Thales Visionix Incorporated (Thales SA)
• Inventors: Foxlin, Eric
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US14/702,402
• Publication Number:
  US 20150317838A1
• Time in Patent Office: 872 Days
• Field of Search: 345633
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/06 with moving picture effect
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, a system, and a computer program product for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference. The system includes An inertial sensor for tracking the object relative to the external reference frame, a non-inertial sensor for tracking the object relative to the moving platform, and a processor to perform sensor fusion of the inertial and non-inertial measurements in order to accurately track the object and concurrently estimate the misalignment of the non-inertial sensor'"'"'s reference frame relative to the moving platform.

Claims

1. A method for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, the method comprising:

updating an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtaining, from an inertial navigation system (INS) mounted to the moving platform, measurements of the orientation of a platform coordinate frame with respect to said external frame of reference;

at certain measurement times, obtaining measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the moving platform, combining said platform-referenced non-inertial orientation measurements with the INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fusing said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference; and

combining said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

2. The method of claim 1, wherein the external frame of reference is an external world frame that does not rotate with said moving platform.

3. The method of claim 1, wherein the measurements are obtained from the INS mounted to the moving platform at a lower rate than the sensor unit fixed to the object.

4. The method of claim 1, wherein said fusion of said indirect measurement with said inertial orientation estimate is performed using a Kalman filter.

5. The method of claim 4, wherein a state vector of said Kalman filter includes three states representing orientation errors and a measurement model includes a measurement residual with a linear relationship to those states.

6. The method of claim 5, wherein the measurement residual is of the form

7. The method of claim 5, wherein a Jacobian of the measurement residual with respect to the three orientation error states comprises an identity matrix.

8. The method of claim 1, wherein combining said platform-referenced non-inertial orientation measurements with the INS measurement of platform orientation further comprises:

forward-predicting said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

9. The method of claim 8, wherein the prediction interval tau is an estimate of the effective time of the platform-referenced non-inertial orientation measurement minus the effective time of the INS measurement of platform orientation.

10. The method of claim 9, wherein the estimation of tau includes:

timestamping the arrival of the INS measurements and the platform-referenced non-inertial orientation measurement;

calculating the difference of arrival times for the INS measurements and the platform-referenced non-inertial orientation measurement;

subtracting an estimate of an age of the platform-referenced non-inertial orientation measurement when it arrived; and

adding an estimate of an age of the INS measurement when it arrived.

11. The method of claim 10, wherein the estimate of the age of the INS measurement at arrival is formed using an algorithm operating on the data during run time.

12. The method of claim 11, wherein said algorithm includes a Kalman filter with an added state for at least one of the prediction interval, a portion of the prediction interval, and the prediction interval error.

13. The method of claim 1, wherein at least one aspect of a misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is not known prior to commencement of tracking operation.

14. The method of claim 13, further comprising:

using an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

15. The method of claim 14, wherein said algorithm comprises:

comparing inertial angular rates from said sensor unit, converted into the measurement reference frame, with angular rates of the platform orientation frame from the INS mounted to the moving platform.

16. The method of claim 14, wherein said algorithm comprises:

comparing object orientation from integration of inertial angular rates from said sensor unit, converted into the measurement reference frame, with the platform orientation from the INS mounted to the moving platform.

17. The method of claim 14, wherein said tracking comprises using a Kalman filter whose states include one of:

at least one aspect of object orientation relative to said external frame of reference, and

at least one aspect of an error in said object orientation relative to said external frame of reference.

18. The method of claim 17, wherein said states of said Kalman filter further includes at least one of said unknown aspect and aspects of said coordinate frame misalignment.

19. A system for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, comprising:

means for updating an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

means for obtaining, from an inertial navigation system (INS) mounted to the moving platform, measurements of the orientation of a platform coordinate frame with respect to said external frame of reference;

means for obtaining measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the moving platform;

means for combining said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference and fusing said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference; and

means for combining said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

20. The system of claim 19, wherein the means for combining said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation forward-predicts said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

21. The system of claim 19, wherein at least one aspect of a misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the system further comprising means for using an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

22. An apparatus for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, comprising:

a memory; and

at least one processor coupled to the memory and configured to;

update an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtain measurements from an inertial navigation system (INS) mounted to the moving platform of the orientation of a platform coordinate frame with respect to said external frame of reference;

obtain measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the moving platform, combine said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fuse said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference; and

combine said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

23. The apparatus of claim 22, wherein the processor is further configured to forward-predict said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

24. The apparatus of claim 22, wherein at least one aspect of the misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the processor further configured to use an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

25. A computer program product for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, the computer program product comprising a computer usable medium having control logic stored therein for causing a computer to:

update an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtain measurements from an inertial navigation system (INS) mounted to the moving platform of the orientation of a platform coordinate frame with respect to said external frame of reference;

obtain measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the moving platform, combine said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fuse said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference; and

combine said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

26. The computer program product of claim 25, further comprising code for causing a computer to forward-predict said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

27. The computer program product of claim 25, wherein at least one aspect of the misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the computer program product further comprising code for causing a computer to use an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.