Video augmented orientation sensor

• US 7,071,970 B2
• Filed: 03/10/2003
• Issued: 07/04/2006
• Est. Priority Date: 03/10/2003
• Status: Expired due to Fees

• Current Assignee: CBIP, LLC
• Original Assignee: Charles Benton
• Inventors: Benton, Charles
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): JONES, HEATHER RAE
• Application Number:
  US10/385,119
• Publication Number:
  US 20040179107A1
• Time in Patent Office: 1,212 Days
• Field of Search: 348/169, 348/148, 348/208.14, 348/218.1, 701/28, 396/13
• US Class Current: 348/208.14
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence

Abstract

An orientation sensor utilizing an inexpensive but relatively slow base orientation sensor, augmented by capture of a video images, identifying reference objects or regions therein, and calculating displacement of those objects to derive supplementary orientation information. Recurrent base orientation sensor results are used to correct drift of the video based technique. Augmenting the slow but accurate base orientation sensor with the consecutive images derived orientation data, provides a fast and inexpensive orientation sensor.

Claims

1. A video augmented orientation sensor comprising:

a base sensor adapted to measure and provide orientation information;

a processor coupled to said base sensor, and adapted to;

receive a current set of orientation information from said base sensor;

receive a first captured image from an image sensor at a known orientation to said base sensor;

identify at least two reference image components in said first image;

receive a subsequently captured image;

compute displacement of said reference image components between said first and subsequent images;

using said displacement, compute and update said current orientation information; and

,

upon reception of a subsequent set of orientation information from said base sensor, utilizing data from said subsequent set to correct or replace said current orientation information.

2. A video augmented orientation sensor according to claim 1, wherein said step of utilizing occurs after a plurality of acquisitions of a valid orientation resolution by said base sensor.

3. A video augmented orientation sensor according to claim 1 wherein said first and subsequent image comprises infrared or ultra-violet data.

4. A video augmented orientation sensor according to claim 1 further comprising an image sensor to capture said images.

5. A video augmented orientation sensor according to claim 1, further comprising a video camera.

6. A video augmented orientation sensor according to claim 5 wherein said video camera is sensitive to infrared or ultraviolet data.

7. A video augmented orientation sensor according to claim 1 wherein said base sensor is further adapted to provide location information.

8. A video augmented orientation sensor according to claim 7, wherein said current orientation information is combined with said location information to create a current location and orientation information.

9. A video augmented orientation sensor according to claim 1 wherein said base sensor is coupled directly or indirectly to a video camera, for capturing said first and subsequent images.

10. A video augmented orientation sensor according to claim 9, wherein said camera is coupled to said base sensor indirectly, and wherein said processor is farther adapted to receive information representative of the relative orientation between said base sensor and said camera.

11. A video augmented orientation sensor according to claim 10, wherein the orientation of said camera is variable.

12. A video augmented orientation sensor according to claim 1 wherein said processor is adapted to replace said first image with a subsequent image, so as to have the processor compare consecutive frames from a video image.

13. A video augmented orientation sensor according to claim 1 wherein said base sensor further comprises a location sensor, and wherein said processor is further adapted to receive speed and heading information of a platform to which the sensor is coupled, and to compute a dead reckoning location information from said speed and heading.

14. A video augmented orientation sensor according to claim 1 wherein said base sensor further comprises a location sensor, and wherein said processor is further adapted to receive ranging information, and to compute a location information from said ranging information.

31. A computer-readable medium storing a computer program that when executed on a computer coupled to a base sensor, will cause said computer to operate substantially as the video augmented location and orientation sensor of claim 1.

32. A computer program that when executed on a computer coupled to a base sensor, will cause said computer to operate substantially as the video augmented location and orientation sensor of claim 7.

15. A method of deriving orientation data comprising the steps of:

sensing a first set of orientation information from a base orientation sensor;

capturing a first image from an image sensor at a known orientation to said base sensor;

identifying at least two reference image components in said first image;

capturing a subsequent image;

computing a displacement of said reference image components between said first image and said subsequent image;

updating current orientation information utilizing said displacement; and

,

receiving a subsequent set of orientation information from said base sensor, and utilizing said second set of information to correct or replace said current orientation parameters after it was updated at least once by said step of updating.

16. A method of deriving orientation data according to claim 15 wherein said base sensor further comprises a location sensor for providing location data.

17. A method of deriving orientation data according to claim 16, wherein said step of computing comprises the steps of computing yaw displacement, pitch displacement and roll displacement.

18. A method of deriving orientation data according to claim 16, wherein said step of receiving is initiated when said reference image components are unidentifiable or fall outside pre-determined parameters.

19. A method of deriving orientation data according to claim 16, further comprising the steps of receiving external motion inputs;

and, performing dead reckoning computation to derive current location information from previously known location information and said motion inputs.

20. A method of deriving orientation data according to claim 19, wherein said motion inputs are selected from a group comprising course, heading, speed, altitude, or a combination thereof.

21. A method of deriving orientation data according to claim 16, further comprising the step of ranging distance and direction information to a known reference point, and utilizing said data to compute current location parameters between successive steps of receiving location information from said base sensor.

22. A method of deriving orientation data according to claim 15, wherein said step of receiving is carried out in response to said base sensor obtaining a valid orientation resolution.

23. A method of deriving orientation data according to claim 15, wherein said first and subsequent image comprises infrared or ultra-violet data.

24. A method of deriving orientation data according to claim 15, further comprising the step of replacing said first image with said subsequent image, to cause said step of computing the displacement to occur between two consecutive images.

25. A method of deriving orientation data according to claim 15, further comprising the step of replacing said first image with any image subsequent thereto, to cause said step of computing the displacement to occur between any two captured images.

26. A method of deriving orientation data according to claim 15, further comprising the step of replacing said first image with any subsequent image, to cause said step of computing the displacement to occur between any two captured images.

27. A virtual reality or an augmented reality system comprising:

a computer image generation system coupled to said sensor, the image generating system having a virtual Reality Database comprising a 3 dimensional virtual environment model, and a processor adapted to generating images from the database;

a display system adapted to display said generated images

a video augmented orientation sensor coupled to said image generation system, said orientation sensor comprising;

a base sensor adapted to measure and provide at least orientation information;

a processor coupled to said base sensor, and adapted to;

receive a first set of orientation information from said base sensor;

receive a first captured image from an image sensor at a known orientation to said base sensor;

identify at least two reference image components in said first image;

receive a subsequently captured image;

compute displacement of said reference image components between said first and subsequent images;

using said displacement, compute current orientation information; and

,

upon reception of a subsequent set of orientation information from said base sensor, utilizing data from said subsequent set to correct or replace said current orientation information;

wherein said orientation sensor is coupled to said image generating system for providing orientation information thereto.

28. A virtual reality or an augmented reality system according to claim 27, further comprising a location sensor for providing location and orientation information to said computer image generation system.

29. A virtual reality or an augmented reality system according to claim 28, wherein said augmented reality system is adapted to be worn by a person.

30. A virtual reality or an augmented reality system according to claim 27, further comprising a video combiner to combine images captured from a camera coupled to said base sensor at a known orientation, and images generated by said computer image generating system.

33. A video camera stabilizer comprising:

a suspension constructed to support a camera and allow said camera controlled freedom of movement in a plurality of axes;

a plurality of actuators, each adapted to control movement of said camera in at least one axis;

a controller coupled to said actuators;

a video augmented orientation sensor comprising;

a base sensor adapted to measure and provide at least orientation information;

a processor coupled to said base sensor, and adapted to;

receive a first set of orientation information from said base sensor;

receive a first captured image from an image sensor at a known orientation to said base sensor;

identify at least two reference rage components in said first image;

receive a subsequently captured image;

compute displacement of said reference image components between said first and subsequent images;

using said displacement, compute current orientation information; and

,

upon reception of a subsequent set of orientation information from said base sensor, utilizing data from said subsequent set to correct or replace said current orientation information;

wherein said video augmented orientation sensor is coupled to said control unit, and the control unit is adapted to operate said plurality of actuators to maintain said camera at a controlled orientation, responsive to orientation data received from said video augmented orientation sensor.

34. A video camera stabilizer according to claim 33, wherein said camera is used to capture said first and subsequent images.

35. A video camera stabilizer according to claim 33 wherein said processor is adapted to replace said first image with a subsequent image, so as to have the processor compare consecutive frames from a video image.

36. A video camera stabilizer according to claim 33, wherein said camera is used to capture said first and subsequent images;

and wherein said processor is adapted to replace said first image with a subsequent image, so as to have the processor compare consecutive frames from a video image.

37. A method for providing Location and orientation (LOR) data to a virtual reality or an augmented reality system, the method comprising the steps of:

receiving a first set of LOR parameters from a base LOR sensor, said LOR parameters comprising a first set of orientation parameters and a first set of location parameters;

capturing a first image from an image sensor at a known orientation to said base sensor;

identifying a plurality of reference image components in said first image;

capturing a subsequent image;

computing a displacement of said reference image components between said first image and said subsequent image;

updating current orientation parameters using said displacement; and

,

correcting said current LOR parameters after they were updated at least once by said step of updating using an updated or replaced set of LOR parameters received from said base sensor, for generating a computer image or components thereof.

38. A method for providing Location and orientation (LOR) data to a virtual reality or an augmented reality system according to claim 37, further comprising the step of replacing said first image with said subsequent image, to cause said step of computing the displacement to occur between two consecutive images.