Head Mountable Display System

• US 20150199847A1
• Filed: 01/14/2014
• Published: 07/16/2015
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Johnson, Lonny E., Braunstein, Michael D.
• Granted Patent: US 9,335,545 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: E02F 9/261 Surveying the work-site to be treated
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

A head mountable display system includes a visual image system, a head mountable display device including a display screen, and a headset pose sensor system. A controller is configured to generate a camera image based upon the image data, generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site, determine an image to be rendered on the head mountable display based upon the camera image and the augmented reality overlay, and render the image on the display screen.

Claims

1. A head mountable display system for use at a work site, comprising:

a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine;

a head mountable display device configured to be mounted on an operator'"'"'s head, the head mountable display device including a display screen;

a headset pose sensor system associated with the head mountable display device for generating headset pose signals indicative of a position and an orientation of the head mountable display device relative to the work site; and

a controller configured to;

receive the image data from the visual image system;

generate a camera image based upon the image data;

generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site;

determine an image to be rendered on the head mountable display device based upon the camera image and the augmented reality overlay; and

render the image on the display screen.

2. The head mountable display system of claim 1, wherein the controller is further configured to determine a percentage of obstruction of an operator'"'"'s view of the work site and the camera image is rendered if the percentage of obstruction is greater than a predetermined percentage.

3. The head mountable display system of claim 1, wherein the controller is further configured to determine whether a portion of the image is within a no draw zone based upon a position and an orientation of the head mountable display device relative to an interior of an operator cab.

4. The head mountable display system of claim 3, wherein the controller is further configured to modify the image so that the image is not rendered within the no draw zone.

5. The head mountable display system of claim 3, further including a sensor system associated with the head mountable display device for determining the position and the orientation of the head mountable display device relative to the interior of the operator cab and the controller is further configured to determine the position and the orientation of the head mountable display device relative to the interior of the operator cab.

6. The head mountable display system of claim 5, further including a pose sensing system for determining the position and the orientation of the machine relative to the work site, and the controller is further configured to determine the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon the position and the orientation of the head mountable display device relative to the work site and the position and the orientation of the machine relative to the work site.

7. The head mountable display system of claim 5, further including a mapping system associated with the head mountable display device for determining a position and an orientation of the head mountable display device relative to the interior of the operator cab.

8. The head mountable display system of claim 7, wherein the mapping system is configured to determine a range and direction to objects within the interior of the operator cab.

9. The head mountable display system of claim 7, wherein the mapping system includes a computer vision system.

10. The head mountable display system of claim 1, further including a machine sensor associated with the machine for sensing a state of the machine, and wherein the controller is further configured to sense a state of the machine based upon the machine sensor, and the camera image is generated based upon the state of the machine and any objects in proximity to the machine.

11. The head mountable display system of claim 10, further including an object detection system for detecting objects in proximity to the machine, and the controller is further configured to identify objects in proximity to the machine based upon the object detection system.

12. The head mountable display system of claim 11, wherein the controller is configured to determine the augmented reality overlay based upon on a type of object identified.

13. The head mountable display system of claim 1, wherein the camera image is generated by combining the image data from the plurality of points of view.

14. A controller-implemented method of rendering images on a head mountable display system of an operator at a work site, comprising:

receiving image data from a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine;

generating a camera image based upon the image data;

generating an augmented reality overlay based upon a position and an orientation of an operator'"'"'s head relative to the work site;

determining an image to be rendered on a head mountable display based upon the camera image and the augmented reality overlay; and

rendering the image on display screen of the head mountable display device.

15. The method of claim 14, further including determining a percentage of obstruction of an operator'"'"'s view of the work site and rendering the camera image if the percentage of obstruction is greater than a predetermined percentage.

16. The method of claim 14, further including determining whether a portion of the image is within a no draw zone based upon a position and an orientation of the head mountable display device relative to an interior of an operator cab, and modifying the image so that the image is not rendered within the no draw zone.

17. The method of claim 15, further including determining the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon a sensor system associated with the head mountable display device.

18. The method of claim 16, further including determining the position and the orientation of the head mountable display device relative to the interior of the operator cab based upon the position and the orientation of the head mountable display device relative to the work site and the position and the orientation of the machine relative to the work site.

19. The method of claim 16, further including determining a position and an orientation of the head mountable display device relative to the interior of the operator cab based upon a mapping system associated with the head mountable display device.

20. A machine comprising:

a propulsion system;

a visual image system mounted on the machine for generating image data of a work site from a plurality of points of view relative to the machine;

a head mountable display device configured to be mounted on an operator'"'"'s head, the head mountable display device including a display screen;

a headset pose sensor system associated with the head mountable display device for generating headset pose signals indicative of a position and an orientation of the head mountable display device relative to the work site; and

a controller configured to;

receive the image data from the visual image system;

generate a camera image based upon the image data;

generate an augmented reality overlay based upon the position and the orientation of the operator'"'"'s head relative to the work site;

determine an image to be rendered on the head mountable display based upon the camera image and the augmented reality overlay; and

render the image on the display screen.