Augmented reality system for vehicle blind spot prevention

• US 10,373,378 B2
• Filed: 06/26/2015
• Issued: 08/06/2019
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: PACCAR Incorporated
• Original Assignee: PACCAR Incorporated
• Inventors: Harbach, Andrew Paul, Lee, Bruce
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US14/751,891
• Publication Number:
  US 20160379411A1
• Time in Patent Office: 1,502 Days
• Field of Search: None
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/081 avoiding blind spots, e.g. by using a side-by-side association of mirrors
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 11/0235 of flat type, e.g. LCD
• CPC Class Codes: B60R 2001/1215 with information displays
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2011/0017 Head-rests
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: B60R 2300/8026 in addition to a rear-view mirror system
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

The present disclosure relates to systems and methods for providing various types of information to a vehicle driver. Such information can be used by the vehicle driver singularly or in conjunction with other information available to the vehicle driver in order to allow the driver to operate the vehicle in an increasingly safe manner and/or to reduce the likelihood of property damage and/or possible bodily injuries to the driver, etc. In some instances, such information is presented to the driver as an augmented reality environment such that the driver can “see through” objects that may be occluding the driver'"'"'s vision.

Claims

1. In a vehicle platoon including a host vehicle controlled by a first driver and a trailing vehicle that is physically separate from and behind the host vehicle and is controlled by a second driver, the host vehicle having one or more displays, and the trailing vehicle having an information source mounted thereon, a method implemented in computer-executable instructions for displaying information about vehicle surroundings to the first driver, the method comprising:

obtaining vehicle environment data from one or more information sources, the vehicle environment data indicative of at least a part of a scene occluded from view of the first driver when operating the host vehicle, wherein the one or more information sources include the information source mounted to the trailing vehicle, wherein the vehicle environment data includes a real image from the information source mounted to the trailing vehicle, and wherein the real image depicts an area behind the trailing vehicle that is occluded from view of the first driver by the trailing vehicle; and

presenting to the first driver, with the aid of the one or more displays, an augmented reality environment based on the vehicle environment data and representative of an area surrounding the host vehicle but obstructed from view of the first driver;

wherein said presenting includes causing the one or more displays to render at least a portion of the real image of the area occluded from view of the first driver by the trailing vehicle.

2. The method of claim 1, wherein said presenting includes

causing the one or more displays to render one or more virtual design elements representative of one or more objects located in the scene occluded from view of the first driver.

3. The method of claim 1, wherein said presenting includes

generating one or more virtual design elements representative of one or more objects located in the scene occluded from view of the first driver;

obtaining a real image of the scene, the real image based on the point of view of the first driver; and

causing the virtual design elements and the real image to be displayed together by the one or more displays.

4. The method of claim 3, wherein said causing the virtual design elements and the real image to be displayed together by the one or more displays includes

arranging the generated one or more virtual design elements over a preselected area of the real image.

5. The method of claim 1, wherein said obtaining vehicle environment data includes

obtaining additional data from a source discrete from the host vehicle, the external source including one of a beacon and a traffic camera.

6. The method of claim 1, wherein the vehicle environment data is associated with one or more sensing zones that correspond to one or more external areas of the host vehicle, wherein the one or more external areas of the host vehicle represent at least in part areas occluded from view of the first driver when operating the host vehicle.

7. A non-transitory computer-readable medium having modules for conveying information to a first driver of a host vehicle in a vehicle platoon regarding vehicle surroundings, wherein the vehicle platoon includes the host vehicle and a trailing vehicle that is physically separate from and behind the host vehicle and controlled by a second driver, the modules comprising:

an information gathering module configured to collect data from one or more information sources associated with one or more sensing zones, wherein the one or more information sources include an information source mounted to the trailing vehicle, wherein the data includes a real image from the information source mounted to the trailing vehicle, wherein the real image depicts an area behind the trailing vehicle that is occluded from view of the first driver by the trailing vehicle;

an augmented reality image rendering module configured to generate from the collected data one or more virtual design elements representative of objects occluded from view of the first driver; and

a display module configured to;

cause the virtual design elements to be presented to a display; and

cause the real image of the area occluded from view of the first driver by the trailing vehicle to be presented to a display.

8. The computer-readable medium of claim 7, wherein the display module is further configured to cause the real image to be presented with the virtual design elements, wherein the real image and the virtual design elements together form an augmented reality environment.

9. The computer-readable medium of claim 8, wherein the augmented reality image rendering module is further configured to arrange and align the generated one or more virtual design elements over a preselected area of the real image.

10. The computer-readable medium of claim 7, wherein the one or more information sources include information sources selected from the group consisting of a digital camera, a thermal imaging device, radar, and lidar.

11. The computer-readable medium of claim 7, wherein the one or more sensing zones correspond to one or more external areas of the host vehicle.

12. The computer-readable medium of claim 11, wherein the one or more external areas of the host vehicle represent at least in part areas occluded from view of the first driver when operating the host vehicle.

13. A system for providing information to a first driver of a host vehicle of a vehicle platoon, wherein the vehicle platoon includes the host vehicle and a trailing vehicle that is physically separate from and behind the host vehicle and is controlled by a second driver, the system comprising:

one or more displays; and

a display generator communicatively coupled to one or more information sources configured to generate data associated with one or more sensing zones located at one or more areas surrounding the host vehicle, wherein the one or more information sources include an information source mounted to the trailing vehicle, wherein the data includes a real image from the information source mounted to the trailing vehicle, and wherein the real image depicts an area behind the trailing vehicle that is occluded from view of the first driver by the trailing vehicle;

wherein the display generator is configured to;

collect data from the one or more information sources associated with the one or more sensing zones;

generate from the collected data one or more virtual design elements representative of objects occluded from view of the first driver;

cause the virtual design elements to be presented to a display; and

cause at least a portion of the real image of the area occluded from view of the first driver by the trailing vehicle to be presented to a display.

14. The system of claim 13, wherein the one or more displays include a transparent display configured to overlay one of a transparent layer and a reflective layer.

15. The system of claim 14, wherein the transparent layer includes a window and the reflective layer includes a mirror.

16. The system of claim 15, wherein the mirror includes one of a side mirror and a rear view mirror.

17. The system of claim 13, wherein the display generator includes a memory storing one or more modules having program instructions for conveying vehicle information on the display;

and

a processor configured to execute the program instructions of the one or more modules.

18. The system of claim 13, wherein the one or more information sources include information sources selected from the group consisting of a digital camera, a thermal imaging device, radar, and lidar.

19. The system of claim 13, wherein one of the one or more information sources includes a receiver configured to receive data associated with the one or more sensing zones from a source discrete from the host vehicle, wherein the source discrete from the host vehicle is one of a beacon and a traffic camera.