Method and apparatus for marking a position of a real world object in a see-through display

• US 8,397,181 B2
• Filed: 11/17/2008
• Issued: 03/12/2013
• Est. Priority Date: 11/17/2008
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Hartman, Randolph G., Buster, Duke
• Primary Examiner(s): Kim, Matt
• Assistant Examiner(s): Nguyen, Tuan S
• Application Number:
  US12/272,395
• Publication Number:
  US 20100125812A1
• Time in Patent Office: 1,576 Days
• Field of Search: 345/156, 345/8, 345/419, 348/239
• US Class Current: 715/848
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A method for marking a position of a real world object on a see-through display is provided. The method includes capturing an image of a real world object with an imaging device. A viewing angle and a distance to the object are determined. A real world position of the object is calculated based on the viewing angle to the object and the distance to the object. A location on the see-through display that corresponds to the real world position of the object is determined. A mark is then displayed on the see-through display at the location that corresponds to the real world object.

Claims

1. A method for marking a position of a real world object on a see-through display comprising:

capturing an image of a real world object with an imaging device;

identifying a pointing device within the field of view of the imaging device;

determining a viewing angle along a line from a see-through display to a tip of the pointing device to the real world object based on an azimuth angle and an elevation angle between locations of the real world object and the pointing device within the field of view of the imaging device;

steering a ranging device toward the real world object based on the determined viewing angle;

determining a real world distance between the ranging device and the real world object with the ranging device, wherein the ranging device determines the real world distance based on an amount of time it takes a signal to travel from the ranging device to the real world object and back to the ranging device;

determining an orientation of the see-through display;

calculating a real world position of the real world object based on the viewing angle to the real world object, the real world distance between the ranging device and the real world object, and the orientation of the see-through display;

determining a first location on the see-through display that corresponds to the calculated real world position of the real world object;

displaying a mark on the see-through display at the first location that corresponds to the calculated real world position of the real world object;

tracking movement of the see-through display relative to the calculated real world position of the real world object using an inertial sensor; and

adjusting the first location of the mark as displayed on the see-through display to account for the movement of the see-through display relative to the calculated real world position of the real world object.

2. The method of claim 1, further comprising:

correlating a location of the pointing device relative to the field of view of the imaging device to a second location on the see-through display; and

placing a mark on the see-through display at the correlated second location on the see-through display.

3. The method of claim 2, further comprising:

updating the correlation of the pointing device relative to the field of view of the imaging device; and

updating the second location on the see-through display in which the mark is placed.

4. The method of claim 2, further comprising:

updating at least one of the orientations or a position of the see-through display;

updating the viewing angle of the see-through display;

determining a second updated location on the see-through display based on the updated viewing angle; and

changing the second location of the mark on the see-through display.

5. The method of claim 1, wherein the see-through display is located within the field of view of a user such that determining a viewing angle determines an angle from an eye of the user to the first location in the see-through display.

6. The method of claim 1, wherein determining a real world position determines a position in terms of an earth coordinate system.

20. The method of claim 1, wherein the ranging device is a laser ranging device;

and wherein the signal is a laser beam.

23. The method of claim 1, further comprising:

communicating the real world position of the real world object to a remote device.

7. A system for marking a position of a real world object on a see-through display comprising:

a processor;

a see-through display communicatively coupled to the processor;

an imaging device communicatively coupled to the processor; and

a ranging device communicatively coupled to the processor;

wherein the imaging device is configured to;

capture an image of a real world object;

wherein the processor is configured to;

identify a pointing device within the field of view of the imaging device;

determine a viewing angle along a line from a see-through display to a tip of the pointing device to the real world object based on an azimuth angle and an elevation angle between locations of the real world object and the pointing device within the field of view of the imaging device;

steer the ranging device toward the real world object based on the determined viewing angle;

determine a real world distance between the ranging device and the real world object with the ranging device, wherein the ranging device determines the real world distance based on an amount of time it takes a signal to travel from the ranging device to the real world object and back to the ranging device;

calculate a real world position of the real world object based on the viewing angle to the real world object, the real world distance between the ranging device and the real world object, and an orientation of the see-through display;

determine a first location on the see-through display that corresponds to the calculated real world position of the real world object;

display a mark on the see-through display at the first location that corresponds to the calculated real world position of the real world object;

track movement of the see-through display relative to the calculated real world position of the real world object using an inertial sensor; and

adjusting the first location of the mark as displayed on the see-through display to account for the movement of the see-through display relative to the real world position of the real world object.

8. The system of claim 7, wherein the processor is further configured to:

correlate a location of the pointing device relative to the field of view of the imaging device to a second location on the see-through display; and

place a mark on the see-through display at the correlated second location on the see-through display.

9. The system of claim 8, wherein the processor is further configured to:

update the correlation of the pointing device relative to the field of view of the imaging device; and

update the second location on the see-through display in which the mark is placed.

10. The system of claim 8, wherein the processor is further configured to:

update at least one of the orientations or a position of the see-through display;

update the viewing angle of the see-through display;

determine an updated second location on the see-through display based on the updated viewing angle; and

change the second location of the mark on the see-through display.

11. The system of claim 7, wherein the see-through display is located within a field of view of a user such that determining a viewing angle determines an angle from an eye of the user to the first location in the see-through display.

12. The system of claim 7, wherein the processor is configured to determine a relative position in term of an earth coordinate system.

21. The system of claim 7, wherein the ranging device is a laser ranging device;

and wherein the signal is a laser beam.

24. The system of claim 7, wherein the processor is further configured to:

cause the system to communicate the real world position of the real world object to a remote device.

13. A program product comprising a non-transitory processor readable medium on which program instructions are embodied, wherein the program instructions are operable to:

capture an image of a real world object with the imaging device;

identify a pointing device within the field of view of the imaging device;

determine a viewing angle along a line from the see-through display to a tip of the pointing device to the real world object based on an azimuth angle and an elevation angle between locations of the real world object and the pointing device within the field of view of the imaging device;

steer a ranging device toward the real world object based on the determined viewing angle;

determine a real world distance between the ranging device and the real world object with the ranging device, wherein the ranging device determines the real world distance based on an amount of time it takes a signal to travel from the ranging device to the real world object and back to the ranging device;

determining an orientation of the see-through display;

calculate a real world position of the real world object based on the viewing angle to the real world object, the real world distance between the ranging device and the real world object and the orientation of the see-through display;

determine a first location on the see-through display that corresponds to the calculated real world position of the real world object;

display a mark on the see-through display at the first location that corresponds to the calculated real world position of the real world object;

track movement of the see-through display relative to the calculated real world position of the real world object using an inertial sensor; and

adjust the first location of the mark as displayed on the sec-through display to account for the movement of the see-through display relative to the real world position of the real world object.

14. The program product of claim 13, wherein the program instructions are further operable to:

correlate a location of the pointing device relative to the field of view of the imaging device to a second location on the see-through display; and

place a mark on the see-through display at the correlated second location on the see-through display.

15. The program product of claim 14, wherein the program instructions are further operable to:

update the correlation of the location of the pointing device relative to the field of view of the imaging device; and

update the second location on the see-through display in which the mark is placed.

16. The program product of claim 14, wherein the processor is further configured to:

update at least one of the orientations or a position of the see-through display;

update the viewing angle of the see-through display;

determine an updated second location on the see-through display based on the updated viewing angle; and

change the second location of the mark on the see-through display.

17. The program product of claim 13, wherein the program instructions are further operable to:

calculate the real world position of the real world object in terms of an earth coordinate system.

22. The program product of claim 13, wherein the ranging device is a laser ranging device;

and wherein the signal is a laser beam.

25. The program product of claim 13 wherein the program instructions are further operable to:

cause the system to communicate the real world position of the real world object to a remote device.

18. A method of determining a distance to a point comprising:

identifying a point within a field of view of an imaging device, the point corresponding to a real world object within the field of view of the imaging device;

determining an orientation between the imaging device and the point within the field of view of the imaging device based on a position of the point within the field of view of the imaging device, wherein determining the orientation includes determining an azimuth angle for the point based on the position of the point within the field of view of the imaging device and determining the elevation angle for the point based on the position of the point within the field of view of the imaging device;

steering a laser ranging device such that a laser beam from the laser ranging device propagates at the determined orientation towards the real world object, wherein steering the laser ranging device includes orienting the laser ranging device based on the azimuth angle and the elevation angle;

determining a distance from the laser ranging device to the real world object located at the determined orientation based on an amount of time it takes the laser beam to travel from the laser ranging device to the real world object and back to the laser ranging device;

calculating a real world position of the real world object based on the determined distance from the laser ranging device to the real world object and the determined orientation;

determining a first location on a see-through display that corresponds to the calculated real world position of the real world object;

displaying a mark on the see-through display at the first location that corresponds to the calculated real world position of the real world object;

tracking movement of the see-through display relative to the calculated real world position of the real world object using an inertial sensor; and

adjusting the first location of the mark as displayed on the see-through display to account for the movement of the see-through display relative to the calculated real world position of the real world object.

26. The method of claim 18, further comprising:

communicating the real world position of the real world object to a remote device; and

identifying the real world position of the real world object to a user at the remote device.

19. An apparatus comprising:

a processing unit;

a see-through display communicatively coupled to the processing unit;

a laser ranging device coupled to the processing unit, wherein the laser ranging device is steerable, such that a laser beam from the laser ranging device is capable of being directed towards a particular orientation; and

an imaging device coupled to the processing unit;

wherein the processing unit is configured to;

identify a point within a field of view of an imaging device, the point corresponding to a real world object within the field of view of the imaging device;

determine an orientation between the imaging device and the point within the field of view of the imaging device based on a position of the point within the field of view of the imaging device, wherein the processing unit is configured to determine an orientation by being configured to determine an azimuth angle for the point based on the position of the point within the field of view of the imaging device, determine the elevation angle of the point based on the position of the point within the field of view of the imaging device;

provide the determined orientation to the laser ranging device by being configured to provide the azimuth angle and the elevation angle to the laser ranging device, wherein the laser ranging device is configured to be steered such that a laser beam from the laser ranging device propagates at the determined orientation towards the real world object;

determine a distance from the laser ranging device to an object located at the determined orientation based on an amount of time it takes the laser beam to travel from the laser ranging device to the object and back to the laser ranging device;

calculate a real world position of the real world object based on the determined distance from the laser ranging device to the real world object and the determined orientation;

determine a first location on the see-through display that corresponds to the calculated real world position of the real world object;

display a mark on the see-through display at the first location that corresponds to the calculated real world position of the real world object;

track movement of the see-through display relative to the calculated real world position of the real world object using an inertial sensor; and

adjusting the first location of the mark as displayed on the see-through display to account for the movement of the see-through display relative to the real world position of the real world object.

27. The apparatus of claim 19, wherein the processing unit is further configured to:

cause the apparatus to communicate the real world position of the real world object to a remote device so that the real world position of the real world object can be identified to a user of the remote device.