FAULT TOLERANT DISPLAY

• US 20180295261A1
• Filed: 06/14/2018
• Published: 10/11/2018
• Est. Priority Date: 11/08/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Beckert, Richard D., Hays, Phillip G., Maillet, Steven P., Lovitt, Andrew W.
• Granted Patent: US 11,012,593 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 5/213 Circuitry for suppressing or minimising impulsive noise for suppressing or minimising disturbance in image signal generation H04N23/81
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A fault tolerant display displays safety information on a display shared with an infotainment system even if the processor or software of the infotainment system is unresponsive. The display can be a streaming video from a rear view camera mounted on a vehicle. The safety displayed can be triggered by placing the vehicle in reverse. Failure of the infotainment software and/or processor can be detected by a policy enforcing module that switches to a backup display that streams video directly from the camera in the event of failure of the infotainment processor.

Claims

1. A fault tolerant display system for a vehicle, the system comprising:

a video camera;

an application processor;

a display in a cabin of the vehicle; and

a hardware module interposed between the video camera, the application processor, and the display;

the hardware module configured to;

receive streaming video from the video camera;

merge the streaming video with overlays from the application processor to create an overlaid streaming video;

display the overlaid streaming video on the display in the cabin of the vehicle;

detect failure of the application processor; and

in response to detecting failure of the application processor, switch to display on the display in the cabin of the vehicle the streaming video from the video camera and at least one overlay generated by the hardware module, the streaming video from the video camera and the at least one overlay generated by the hardware module displayed essentially simultaneously on the display.

2. The system of claim 1, wherein failure of the application processor is detected by incorrect formatting of the display information generated by the application processor.

3. The system of claim 2, wherein the incorrect formatting comprises at least one of:

incorrect frame rate, incorrect line rate, average brightness of the display lower than a provided threshold for average brightness of the display, a quantity of non-black pixels lower than a provided threshold for non-black pixels, or placement of features generated by the application processor failing to correspond to objects in the streaming video.

4. The system of claim 1, wherein failure of the application processor is detected by a watchdog timer.

5. The system of claim 1, wherein the overlaid streaming video is displayed within one second of the vehicle shifting into reverse.

6. The system of claim 1, wherein the at least one overlay generated by the hardware module comprises a dynamic overlay of a backup path based on a steering wheel position of the vehicle.

7. The system of claim 1, wherein the at least one overlay generated by the hardware module comprises a dynamic overlay based on hazard detection information.

8. The system of claim 1, wherein the at least one overlay generated by the hardware module comprises a dynamic overlay based on navigation information.

9. The system of claim 1, wherein failure of the application processor is detected when the application processor is detected to be unresponsive.

10. The system of claim 1, wherein failure of the application processor is detected when graphical information received from the application processor fails a quality check.

11. The system of claim 1, wherein the hardware module is further configured to:

insert a check field into the streaming video sent to the application processor; and

verify a modified check field received from the application processor, the modified check field being returned by the application processor responsive to the check field;

wherein failure of the application processor is detected when the modified check field fails verification.

12. The system of claim 1, wherein the hardware module is one of an application specific integrated circuit (ASIC), a field programmable gateway array (FPGA), or a processor separate from the application processor that executes a fault tolerant operating system and application.

13. The system of claim 1, wherein the application processor is an in-vehicle infotainment system.

14. The system of claim 1, wherein the hardware module is further configured to:

receive status information of the vehicle from a vehicle sensor, the status information specifying a gear in which the vehicle is operating; and

display the streaming video from the video camera on the display in response to the vehicle being shifted into reverse gear when failure of the application processor is detected.

15. A method, comprising:

receiving streaming video from a video camera of a vehicle;

merging the streaming video with overlays from an application processor to create an overlaid streaming video by a hardware module;

displaying the overlaid streaming video on a display in a cabin of the vehicle, the hardware module interposed between the video camera, the application processor, and the display;

monitoring the application processor to detect failure of the application processor; and

in response to detecting failure of the application processor, switching to displaying on the display in the cabin of the vehicle, the streaming video from the video camera and at least one overlay generated by the hardware module, the streaming video from the video camera and the at least one overlay generated by the hardware module displayed essentially simultaneously.

16. The method of claim 15, wherein the application processor is an in-vehicle infotainment system.

17. The method of claim 15, wherein the hardware module is one of an application specific integrated circuit (ASIC), a field programmable gateway array (FPGA), or a processor separate from the application processor that executes a fault tolerant operating system and application.

18. A computer-readable storage medium comprising computer-executable instructions which when executed cause at least one processor of a computing device, when operably connected to a video camera, an application processor, and a display, to:

receive streaming video from a video camera of a vehicle;

merge the streaming video with overlays from an application processor to create an overlaid streaming video by a hardware module;

display the overlaid streaming video on a display in a cabin of the vehicle, the hardware module interposed between the video camera, the application processor, and the display;

monitor the application processor to detect failure of the application processor; and

in response to detecting failure of the application processor, switch to displaying on the display in the cabin of the vehicle, the streaming video from the video camera and at least one overlay generated by the hardware module, the streaming video from the video camera and the at least one overlay generated by the hardware module displayed essentially simultaneously.

19. The computer-readable storage medium of claim 18, wherein the application processor is an in-vehicle infotainment system.

20. The computer-readable storage medium of claim 18, wherein the hardware module is one of an application specific integrated circuit (ASIC), a field programmable gateway array (FPGA), or a processor separate from the application processor that executes a fault tolerant operating system and application.