Head mountable display system

• US 9,335,545 B2
• Filed: 01/14/2014
• Issued: 05/10/2016
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Johnson, Lonny E., Braunstein, Michael D.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): CHEN, FRANK S
• Application Number:
  US14/154,619
• Publication Number:
  US 20150199847A1
• Time in Patent Office: 847 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: E02F 9/261 Surveying the work-site to be treated
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

A head mountable display system includes a visual image system, a head mountable display device including a display screen, and a headset pose sensor system. A controller is configured to generate a camera image based upon the image data, generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site, determine an image to be rendered on the head mountable display based upon the camera image and the augmented reality overlay, and render the image on the display screen.

Claims

1. A head mountable display system for use at a work site, comprising:

a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine;

a head mountable display device configured to be mounted on an operator'"'"'s head, the head mountable display device including a display screen;

a headset pose sensor system associated with the head mountable display device for generating headset pose signals indicative of a position and an orientation of the head mountable display device relative to the work site;

a sensor system associated with the head mountable display device for determining the position and the orientation of the head mountable display device relative to the interior of the operator cab;

a pose sensing system for determining the position and the orientation of the machine relative to the work site; and

a controller configured to;

receive the image data from the visual image system;

generate a camera image based upon the image data;

generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site;

determine an image to be rendered on the head mountable display device based upon the camera image and the augmented reality overlay; and

render the image on the display screen;

determine whether a portion of the image is within a no draw zone based upon a position and an orientation of the head mountable display device relative to an interior of an operator cab;

determine the position and the orientation of the head mountable display device relative to the interior of the operator cab;

determine the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon the position and the orientation of the head mountable display device relative to the work site and the position and the orientation of the machine relative to the work site.

2. The head mountable display system of claim 1, wherein the controller is further configured to determine a percentage of obstruction of an operator'"'"'s view of the work site and the camera image is rendered if the percentage of obstruction is greater than a predetermined percentage.

3. The head mountable display system of claim 1, wherein the controller is further configured to modify the image so that the image is not rendered within the no draw zone.

4. The head mountable display system of claim 1, further including a mapping system associated with the head mountable display device for determining a position and an orientation of the head mountable display device relative to the interior of the operator cab.

5. The head mountable display system of claim 4, wherein the mapping system is configured to determine a range and direction to objects within the interior of the operator cab.

6. The head mountable display system of claim 4, wherein the mapping system includes a computer vision system.

7. The head mountable display system of claim 1, further including a machine sensor associated with the machine for sensing a state of the machine, and wherein the controller is further configured to sense a state of the machine based upon the machine sensor, and the camera image is generated based upon the state of the machine and any objects in proximity to the machine.

8. The head mountable display system of claim 7, further including an object detection system for detecting objects in proximity to the machine, and the controller is further configured to identify objects in proximity to the machine based upon the object detection system.

9. The head mountable display system of claim 8, wherein the controller is configured to determine the augmented reality overlay based upon on a type of object identified.

10. The head mountable display system of claim 1, wherein the camera image is generated by combining the image data from the plurality of points of view.

11. A controller-implemented method of rendering images on a head mountable display system of an operator at a work site, comprising:

receiving image data from a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine;

generating a camera image based upon the image data;

generating an augmented reality overlay based upon a position and an orientation of an operator'"'"'s head relative to the work site;

determining an image to be rendered on a head mountable display based upon the camera image and the augmented reality overlay;

determining whether a portion of the image is within a no draw zone based upon a position and an orientation of the head mountable display device relative to an interior of an operator cab, and modifying the image so that the image is not rendered within the no draw zone;

determining the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon the position and the orientation of the head mountable display device relative to the work site and the position and the orientation of the machine relative to the work site; and

rendering the image on a display screen of the head mountable display device.

12. The method of claim 11, further including determining a percentage of obstruction of an operator'"'"'s view of the work site and rendering the camera image if the percentage of obstruction is greater than a predetermined percentage.

13. The method of claim 12, further including determining the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon a sensor system associated with the head mountable display device.

14. The method of claim 11, further including determining a position and an orientation of the head mountable display device relative to the interior of the operator cab based upon a mapping system associated with the head mountable display device.

15. A machine comprising:

a propulsion system;

a visual image system mounted on the machine for generating image data of a work site from a plurality of points of view relative to the machine;

a head mountable display device configured to be mounted on an operator'"'"'s head, the head mountable display device including a display screen;

a headset pose sensor system associated with the head mountable display device for generating headset pose signals indicative of a position and an orientation of the head mountable display device relative to the work site;

a sensor system associated with the head mountable display device for determining the position and the orientation of the head mountable display device relative to the interior of the operator cab;

a pose sensing system for determining the position and the orientation of the machine relative to the work site; and

a controller configured to;

receive the image data from the visual image system;

generate a camera image based upon the image data;

generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site;

determine an image to be rendered on the head mountable display based upon the camera image and the augmented reality overlay; and

render the image on the display screen;

determine whether a portion of the image is within a no draw zone based upon a position and an orientation of the head mountable display device relative to an interior of an operator cab;

determine the position and the orientation of the head mountable display device relative to the interior of the operator cab;

determine the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon the position and the orientation of the head mountable display device relative to the work site and the position and the orientation of the machine relative to the work site.