Systems, apparatus and fast methods for aligning images to external markers in near-to-eye display systems

• US 8,362,973 B2
• Filed: 05/19/2009
• Issued: 01/29/2013
• Est. Priority Date: 05/19/2009
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Cernasov, Andrei, Cupero, Frank P.
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Sharifi-Tafreshi, Koosha
• Application Number:
  US12/468,601
• Publication Number:
  US 20100295754A1
• Time in Patent Office: 1,351 Days
• Field of Search: 345 7- 9, 345/633, 382/103
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G09G 2380/12 Avionics applications
• CPC Class Codes: G09G 3/001 using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP

Abstract

Provided are systems, apparatus, and methods for registering an image within a defined area within the confines of a user. The area is defined by one or more information markers that may be active or passive markers. A sensor operable to detect the information markers is rigidly attached to a motion tracker which is aligned and calibrated relative to the information markers and by detecting at least one of the information markers causes the virtual object image to appear to remain within the defined area.

Claims

1. A system for registering an image to a defined external area as perceived through a near-to-eye (NTE) raster scan video display worn by a user, comprising:

at least one active marker disposed remote from the user, wherein each of the at least one active marker is configured to emit a uniquely distinguishing feature;

a marker sensor configured to detect the uniquely distinguishing feature of each of the at least one active marker and supply marker identification data;

a video display configured to render an image, the video display being genlocked with the marker sensor and the uniquely distinguishing feature; and

a processor in operable communication with the marker sensor and the video display, the processor operable to, upon receipt of motion data and the marker identification data, cause the image to appear in the defined external area.

2. The system of claim 1 further comprising a motion tracker rigidly affixed to the sensor and configured to sense user motion and supply user motion data representative thereof.

3. The system of claim 1 wherein the uniquely distinguishing feature is an information pattern.

4. The system of claim 1 wherein the information pattern is emitted at a periodicity that is genlocked with the raster scan of the video display and the raster scan of the sensor.

5. The system of claim 2 wherein the processor contains instructions that when executed perform the steps of:

identifying the marker detected by the sensor;

calculating a sensor display offset between an actual position of the identified marker and a predicted position derived from the user motion data;

converting the sensor display offset to a pixel address offset within the video display; and

determining a pixel address within the video display from which to continue rendering the virtual object image.

6. The system of claim 1 wherein the sensor comprises a dual linear array.

7. The system of claim 1 wherein the sensor is a complementary metal-oxide semiconductor (CMOS) array.

8. A method for registering a image to a defined external area as perceived through a near-to-eye (NTE) video display worn by a user, comprising:

affixing a first marker to a known location, wherein the marker exhibits a uniquely distinguishing feature;

aligning and calibrating a marker sensor operable to detect the uniquely distinguishing feature of the marker to a motion tracker that is operable to discern movements of the NTE video display, each of the sensor and the tracker being rigidly connected together;

electronically synchronizing a raster scan periodicity of the of the marker sensor and a separate raster scan periodicity of the NTE video display, wherein the periodicity of the marker sensor raster scan is equal to or faster than the periodicity of the NTE video display raster scan; and

executing commands by a computing device that is in operable communication with the marker sensor, the motion tracker and the NTE video display, whereby the commands cause the computing device to;

identify the first marker detected by the sensor;

determine the known location of the first marker;

calculate a sensor display offset between the known location of the first marker based on the user motion data;

convert the sensor display offset to a pixel address offset within the NTE video display; and

determine a pixel address within the NTE video display from which to continue to render the virtual object image.

9. The method of claim 8 further comprising affixing at least a second information marker.

10. The method of claim 8 wherein the first and second information markers are passive information markers.

11. The method of claim 8 wherein the first and the at least second information markers are active information markers.

12. The method of claim 11 wherein the first information marker emits light information at a different frequency than the at least second information marker.

13. The method of claim 11 wherein the first information marker emits light information at the same frequency but in a different timing sequence than the at least second information marker.

14. The method of claim of claim 11 wherein the first and the at least second information markers emit at least one of visible light, infrared light and ultraviolet light.

15. The method of claim 13 wherein the periodicity of the timing sequences of the first information marker and the at last second information marker are genlocked with the raster scan of the NTE video display and the raster scan of the sensor.

16. The method of claim 13 wherein the periodicity of the timing sequences of the first information marker and the at last second information marker are genlocked with the periodicity of at least some of the pixels of the sensor.

17. The method of claim 8, wherein the raster scan of the NTE video display is genlocked with the raster scan of the marker sensor.

18. An apparatus for registering an image to a defined external area as perceived through a near-to-eye (NTE) raster scan video display worn by a user, comprising:

a marker sensor configured to detect at least one active marker configured to emit an information pattern with a periodicity and to supply marker identification data based on the information pattern;

a motion tracker rigidly affixed to the marker sensor and configured to sense motion and supply motion data representative thereof;

a video display configured to render an image, the video display being genlocked with the marker sensor and the information pattern of the marker; and

a processor in operable communication with the marker sensor, the motion tracker and the video display, the processor operable upon receipt of the motion data and the marker identification data, to cause the virtual object image to appear in the defined external area.

19. The apparatus of claim 18 wherein the marker sensor, the motion tracker, the video display and the processor are rigidly mounted on a headset of the user.

20. The apparatus of claim 18, wherein the marker sensor is rigidly attached to the front of the motion tracker.