Integrated photogrammetric light communications positioning and inertial navigation system positioning

• US 9,590,728 B2
• Filed: 09/29/2012
• Issued: 03/07/2017
• Est. Priority Date: 09/29/2012
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Roberts, Richard D., Yang, Xue
• Primary Examiner(s): Lau, Tung
• Application Number:
  US13/631,909
• Publication Number:
  US 20140093249A1
• Time in Patent Office: 1,620 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: H04B 10/116 Visible light communication

Abstract

A mobile device includes an inertial navigation system (INS) to measure inertial quantities associated with movement of the device, and estimate a kinematic state associated with the movement based on the measured inertial quantities. The device includes a light receiver to record light beams originating from lights at respective image positions in a sequence of images. The device photogrammetrically determines its position relative to the originating lights based on predetermined real-world positions and corresponding image positions of the lights. The device corrects the estimated kinematic state based on the photogrammetrically determined position, to produce a corrected estimated kinematic state.

Claims

1. A non-transitory computer readable medium encoded with a computer program that includes instructions to cause a processor to:

receive measured inertial quantities resulting from movement of an apparatus;

estimate a kinematic state associated with the movement based on the measured inertial quantities;

record a sequence of images of an array of spatially positioned lights, wherein each image includes images of modulated light beams emitted from respective lights of the array at respective 2-dimensional coordinates of the image;

demodulate identifiers from the images of modulated light beams;

retrieve predetermined physical positions of the lights of the array, within a 3-dimensional coordinate frame, based on the respective demodulated identifiers;

photogrammetrically determine a position of the apparatus relative to the array of lights based on the predetermined physical positions of the respective lights within the 3-dimensional coordinate frame, the 2-dimensional coordinates of the images of the light beams within the images, and the estimated kinematic state; and

correct the estimated kinematic state based on the photogrammetrically determined position to produce a corrected estimated kinematic state.

2. The non-transitory computer readable medium of claim 1, further including instructions to cause the processor to:

determine the 2-dimensinal coordinates of the images of the light beams relative to a 2-dimensional image plane coordinate frame of an imaging device;

transform the 2-dimensinal coordinates of the images of the light beams from the 2-dimesional image plane coordinate frame to the 3-dimensional coordinate frame based on the estimated kinematic state; and

compute the position of the apparatus within the 3-dimensional coordinate frame based on the transformed coordinates of the images of the light beams and the predetermined physical positions of the lights within the 3-dimensional coordinate frame.

3. The non-transitory computer readable medium of claim 1, further including instructions to cause the processor to:

estimate positions and corresponding orientations of the apparatus based on the measured inertial quantities;

transform the estimated positions to the 3-dimensional coordinate frame based on the corresponding estimated orientations;

photogrammetrically determine a position of the apparatus for each of multiple sequences of images;

compute a difference between each photogrammetrically determined position and a corresponding transformed estimated position;

filter the differences over time to reduce noise generated by sensors of the inertial quantities;

adjust the transformed estimated positions based on the filtered differences; and

output the adjusted transformed estimated positions as error-corrected positions of the apparatus within the 3-dimesnaion coordinate frame.

4. The non-transitory computer readable medium of claim 2, further including instructions to cause the processor to:

estimate orientations of the apparatus based on the measured inertial quantities;

transform the estimated orientations from a kinetic sensor coordinate frame to the 2-dimensional image plane coordinate frame;

predict the 2-dimensional coordinates of the images of the light beams over the sequence of images based on the transformed estimated orientations;

demodulate the identifiers from the images of the light beams based on the predicted 2-dimensional coordinates of the images of the light beams; and

retrieve the predetermined physical positions of the lights within the 3-dimensional coordinate frame based on the identifiers.

5. The non-transitory computer readable medium of claim 3, further including instructions to cause the processor to:

estimate the positions and corresponding orientations of the apparatus at a first rate; and

photogrammetrically determine the positions of the apparatus for the respective images at a second rate that is lower than the first rate.

6. The non-transitory computer readable medium of claim 3, further including instructions to cause the processor to:

estimate the positions and corresponding orientations of the apparatus based on outputs of an accelerometer, a gyroscope, and a magnetometer; and

estimate a travel distance of the apparatus based on the estimated orientation, the output of the accelerometer, and gravity compensation.

7. The non-transitory computer readable medium of claim 3, further including instructions to cause the processor to:

estimate a length of a walking step of a user of the apparatus;

detect walking steps of the user based on the measured inertial quantities; and

compute a travel distance of the user based on output of an accelerometer, the estimated length of a walking step, and detected walking steps of the user.

8. An apparatus, comprising, a controller configured to:

estimate a kinematic state associated with movement of the apparatus based on measured inertial quantities associated with the movement;

access a sequence of images of an array of spatially positioned lights, recorded from a perspective of the apparatus, wherein each image includes images of light beams emitted from respective lights of the array at respective 2-dimensional coordinates of the image;

demodulate identifiers from the images of modulated light beams;

retrieve predetermined physical positions of the lights of the array, within a 3-dimensional coordinate frame, based on the respective demodulated identifiers;

photogrammetrically determine a position of the apparatus relative to the array of lights based on the predetermined physical positions of the lights within the 3-dimensional coordinate frame, the 2-dimensional coordinates of the images of the light beams within the images, and the estimated kinematic state; and

correct the estimated kinematic state based on the photogrammetrically determined position to provide a corrected estimated kinematic state.

9. The apparatus of claim 8, wherein the controller is further configured to:

determine the 2-dimensional coordinates of the images of the light beams relative to a 2-dimensional image plane coordinate frame of an imaging device;

transform the 2-dimensional coordinates of the images of the light beams from the 2-dimesional image plane coordinate frame to the 3-dimensional coordinate frame based on the estimated kinematic state; and

compute the position of the apparatus within the 3-dimensional coordinate frame based on the transformed coordinates of the images of the light beams and the predetermined physical positions of the lights within the 3-dimensional coordinate frame.

10. The apparatus of claim 8, wherein the controller is further configured to:

estimate positions and corresponding orientations of the apparatus based on the measured inertial quantities;

transform the estimated positions to the 3-dimensional coordinate frame based on the corresponding estimated orientations;

photogrammetrically determine a position of the apparatus for each of multiple sequences of images;

compute a difference between each photogrammetrically determined position and a corresponding transformed estimated position;

filter the differences over time to reduce noise generated by sensors of the inertial quantities;

adjust the transformed positions based on the filtered differences; and

output the adjusted transformed estimated positions as error-corrected positions of the apparatus within the 3-dimesnaion coordinate frame.

11. The apparatus of claim 9, wherein the controller is further configured to:

estimate orientations of the apparatus based on the measured inertial quantities;

transform the estimated orientations from a kinetic sensor coordinate frame to the 2-dimesional image plane coordinate frame;

predict the 2-dimensional coordinates of the images of the light beams over the sequence of images based on the transformed estimated orientations;

demodulate the identifiers from the images of the light beams based on the predicted 2-dimensional coordinates of the images of the light beams; and

retrieve the predetermined physical positions of the lights within the 3-dimensional coordinate frame based on the identifiers.

12. The apparatus of claim 10, wherein the controller is further configured to:

estimate the positions and corresponding orientations of the apparatus at a first rate; and

photogrammetrically determine the positions of the apparatus for the respective images at a second rate that is lower than the first rate.

13. The apparatus of claim 10, wherein the controller is further configured to:

estimate the positions and corresponding orientations of the apparatus based on outputs of an accelerometer, a gyroscope, and a magnetometer; and

estimate travel distance of the apparatus based on the estimated orientation, the output of the accelerometer, and gravity compensation.

14. The apparatus of claim 10, wherein the controller is further configured to:

estimate a length of a walking step of a user of the apparatus;

detect walking steps of the user based on the measured inertial quantities; and

compute a travel distance of the apparatus based on output of an accelerometer, the estimated length of a walking step, and detected walking steps of the user.

15. A method, comprising:

measuring inertial quantities resulting from movement of an apparatus;

estimating a kinematic state associated with the movement based on the measured inertial quantities;

recording a sequence of images of an array of spatially positioned lights, from a perspective of the apparatus, wherein each image includes images of light beams emitted from respective lights of the array at respective 2-dimensional coordinates of the image;

demodulating identifiers from the images of modulated light beams;

retrieving predetermined physical positions of the lights of the array, within a 3-dimensional coordinate frame, based on the respective demodulated identifiers;

photogrammetrically determining a position of the apparatus relative to the array of lights based on the predetermined physical positions of the respective lights within the 3-dimensional coordinate frame, the 2-dimensional coordinates of the images of the light beams within the images, and the estimated kinematic state; and

correcting the estimated kinematic state based on the photogrammetrically determined position to provide a corrected estimated kinematic state.

16. The method of claim 15, wherein the photogrammetrically determining includes:

determining the 2-dimensinal coordinates of the images of the light beams relative to a 2-dimesional image plane coordinate frame of an imaging device;

transforming the 2-dimensinal coordinates of the images of the light beams from the 2-dimesional image plane coordinate frame to the 3-dimensional coordinate frame based on the estimated kinematic state; and

computing the position of the apparatus within the 3-dimensional coordinate frame based on the transformed coordinates of the images of the light beams and the predetermined physical positions of the lights within the 3-dimensional coordinate frame.

17. The method of claim 15, wherein:

the estimating a kinematic state includes estimating positions and corresponding orientations of the apparatus based on the measured inertial quantities, and transforming the estimated positions to the 3-dimensional coordinate frame based on the corresponding estimated orientations;

the photogrammetrically determining includes photogrammetrically determining a position of the apparatus for each of multiple sequences of images; and

the correcting includes computing a difference between each photogrammetrically determined position and a corresponding transformed estimated position, filtering the transformed differences over time to reduce noise generated by sensors of the inertial quantities, adjusting the transformed estimated positions based on the filtered differences, and outputting the adjusted transformed estimated positions as error-corrected positions of the apparatus within the 3-dimensional coordinate frame.

18. The method of claim 16, wherein the estimating a kinematic state includes estimating orientations of the apparatus based on the measured inertial quantities, and wherein the photogrammetrically determining further includes:

transforming the estimated orientations from a kinetic sensor coordinate frame to the 2-dimensional image plane coordinate frame;

predicting the 2-dimensional coordinates of the images of the light beams over the sequence of images based on the transformed estimated orientations;

demodulating the identifiers from the images of the light beams based on the predicted 2-dimensional coordinates of the images of light; and

retrieving the predetermined physical positions of the lights within the 3-dimensional coordinate frame based on the identifiers.

19. The method of claim 17, further including:

performing the estimating positions and corresponding orientations of the apparatus at a first rate; and

performing the photogrammetrically determining a position of the apparatus for each of multiple sequences of images at a second rate that is lower than the first rate.

20. The method of claim 17, further including:

performing the estimating positions and corresponding orientations of the apparatus based on outputs of an accelerometer, a gyroscope, and a magnetometer; and

estimating a travel distance of the apparatus based on the estimated orientation, the output of the accelerometer, and gravity compensation.

21. The method of claim 17, further including:

estimating a length of a walking step of a user of the apparatus;

detecting walking steps of the user based on the measured inertial quantities; and

computing a travel distance of the apparatus based on output of an accelerometer, the estimated length of a walking step, and detected walking steps of the user.