REGISTRATION FOR VEHICULAR AUGMENTED REALITY USING AUTO-HARMONIZATION

• US 20150317838A1
• Filed: 05/01/2015
• Published: 11/05/2015
• Est. Priority Date: 05/02/2014
• Status: Active Grant

• Current Assignee: Thales Visionix Incorporated (Thales SA)
• Original Assignee: Thales Visionix Incorporated (Thales SA)
• Inventors: FOXLIN, Eric
• Granted Patent: US 9,766,075 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/06 with moving picture effect
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, a system, and a computer program product for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference. The system includes An inertial sensor for tracking the object relative to the external reference frame, a non-inertial sensor for tracking the object relative to the moving platform, and a processor to perform sensor fusion of the inertial and non-inertial measurements in order to accurately track the object and concurrently estimate the misalignment of the non-inertial sensor'"'"'s reference frame relative to the moving platform.

Claims

1. A method for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, the method comprising:

updating an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtaining measurements from a platform-mounted inertial navigation system (INS) of the orientation of a platform coordinate frame with respect to said external frame of reference; and

at certain measurement times, obtaining measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the platform, combining said platform-referenced non-inertial orientation measurements with the INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fusing said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference.

2. The method of claim 1, wherein the external frame of reference is an external world frame that does not rotate with said moving platform.

3. The method of claim 1, wherein the measurements are obtained from the platform-mounted INS at a lower rate than the sensor unit fixed to the object.

4. The method of claim 1, further comprising:

combining said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

5. The method of claim 1, wherein said fusion of said indirect measurement with said inertial orientation estimate is performed using a Kalman filter.

6. The method of claim 5, wherein a state vector of said Kalman filter includes three states representing orientation errors and a measurement model includes a measurement residual with a linear relationship to those states.

7. The method of claim 6, wherein the measurement residual is of the form zS⁻

1({tilde over (C)}_bⁱĈ

_b^iT−

I).

8. The method of claim 6, wherein a Jacobian of the measurement residual with respect to the three orientation error states comprises an identity matrix.

9. The method of claim 1, wherein combining said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation further comprises:

forward-predicting said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

10. The method of claim 9, wherein the prediction interval tau is an estimate of the effective time of the platform-referenced non-inertial orientation measurement minus the effective time of the INS platform orientation measurement.

11. The method of claim 10, wherein the estimation of tau includes:

timestamping the arrival of the INS measurements and the platform-referenced non-inertial orientation measurement;

calculating the difference of arrival times for the INS measurements and the platform-referenced non-inertial orientation measurement;

subtracting an estimate of an age of the non-inertial measurement when it arrived; and

adding an estimate of an age of the INS measurement when it arrived.

12. The method of claim 11, wherein the estimate of the age of the INS measurement at arrival is formed using an algorithm operating on the data during run time.

13. The method of claim 12, wherein said algorithm includes a Kalman filter with an added state for at least one of the prediction interval, a portion of the prediction interval, and the prediction interval error.

14. The method of claim 1, wherein at least one aspect of a misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is not perfectly known prior to commencement of tracking operation.

15. The method of claim 14, further comprising:

using an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

16. The method of claim 15, wherein said algorithm comprises:

comparing the inertial angular rates from said sensor unit, converted into the measurement reference frame, with the angular rates of the platform frame from the platform INS.

17. The method of claim 15, wherein said algorithm comprises:

comparing the orientation from integration of inertial angular rates from said sensor unit, converted into the measurement reference frame, with the platform orientation from the platform INS.

18. The method of claim 15, wherein said tracking comprises using a Kalman filter whose states include one of:

at least one aspect of the object orientation relative to said external frame of reference, and

at least one aspect of the error in said object orientation relative to said external frame of reference.

19. The method of claim 18, wherein said Kalman filter'"'"'s state vector further includes at least one of said unknown aspect and aspects of said coordinate frame misalignment.

20. A system for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, comprising:

means for updating an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

means for obtaining measurements from a platform-mounted inertial navigation system (INS) of the orientation of a platform coordinate frame with respect to said external frame of reference;

means for obtaining measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the platform; and

means for combining said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference and fusing said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference.

21. The system of claim 20, further comprising:

means for combining said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

22. The system of claim 20, wherein the means for combining said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation forward-predicts said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

23. The system of claim 20, wherein at least one aspect of the misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the system further comprising means for using an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

24. An apparatus for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, comprising:

a memory; and

at least one processor coupled to the memory and configured to;

update an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtain measurements from a platform-mounted inertial navigation system (INS) of the orientation of a platform coordinate frame with respect to said external frame of reference; and

obtain measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the platform, combine said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fuse said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference.

25. The apparatus of claim 24, wherein the processor is further configured to combine said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

26. The apparatus of claim 24, wherein the processor is further configured to forward-predict said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

27. The apparatus of claim 24, wherein at least one aspect of the misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the processor further configured to use an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.

28. A computer program product for tracking an object moving relative to a moving platform, the moving platform moving relative to an external frame of reference, the computer program product comprising a computer usable medium having control logic stored therein for causing a computer to:

update an inertial orientation estimate of the object relative to the external frame of reference using at least one angular rate sensor comprised in a sensor unit fixed to the object;

obtain measurements from a platform-mounted inertial navigation system (INS) of the orientation of a platform coordinate frame with respect to said external frame of reference; and

obtain measurements from a non-inertial sensor of the orientation of the sensor unit relative to a measurement reference coordinate frame fixed with respect to the platform, combine said platform-referenced non-inertial orientation measurements with the most recent INS measurement of platform orientation to form an indirect measurement of the orientation of said object with respect to said external frame of reference, and fuse said indirect measurement with said inertial orientation estimate of the object to form an improved estimate of object orientation relative to said external frame of reference.

29. The computer program product of claim 28, further comprising:

code for causing a computer to combine said improved estimate of object orientation relative to said external frame of reference with said INS measurement of platform orientation relative to said external frame of reference to form an estimate of relative orientation of said object with respect to said moving platform.

30. The computer program product of claim 28, further comprising code for causing a computer to forward-predict said INS measurement through a prediction interval tau to the effective time of the platform-referenced non-inertial orientation measurement.

31. The computer program product of claim 28, wherein at least one aspect of the misalignment of said measurement reference coordinate frame relative to said platform coordinate frame is unknown prior to commencement of tracking operation, the computer program product further comprising code for causing a computer to use an algorithm to estimate said unknown aspect of the coordinate frame misalignment concurrently during tracking.