System And Method For Assessing The Interior Of An Autonomous Vehicle

• US 20180225890A1
• Filed: 02/03/2017
• Published: 08/09/2018
• Est. Priority Date: 02/03/2017
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Jales Costa, Bruno Sielly, Goh, Madeline J., Jain, Jinesh J.
• Granted Patent: US 10,290,158 B2
• Time in Patent Office: Days
• CPC Class Codes: B60S 3/00 Vehicle cleaning apparatus not integral with vehicles cleaning in general B08B; cleaning peculiar to waterborne vessels B63B57/00, B63B59/00; ground equipment for cleaning aircraft B64F5/30
• CPC Class Codes: B60S 5/00 Servicing, maintaining, repairing, or refitting of vehicles vehicles adapted to carry a workshop for servicing or maintenance B60P3/14; servicing rail locomotives B61K
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G07C 5/006 Indicating maintenance
• CPC Class Codes: G07C 5/0891 Video recorder in combination with video camera
• CPC Class Codes: G08G 1/202 Dispatching vehicles on the basis of a location, e.g. taxi dispatching
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

An autonomous vehicle includes interior sensors including an IR camera and a visible light camera. Images of an interior of the vehicle are captured using the cameras both before and after a passenger rides in the vehicle. The IR images from before and after are subtracted to obtain a difference image. Pixels above a threshold intensity a clustered. Clusters having an above-threshold size are determined to be anomalies. Portions of images from the visible light camera corresponding to the anomalies are sent to a dispatcher, who may then clear the vehicle to pick up another passenger or proceed to a cleaning station. Anomalies may be identified based on a combination of the IR images and visible light images.

Claims

1. A system for a vehicle comprising:

an infrared (IR) camera sensing an interior of the vehicle;

a controller coupled to the IR camera and programmed to—

receive a first output of the IR camera;

receive a second output of the IR camera subsequent to the first output; and

if a change between the first output and the second output indicates an anomaly, invoke a remedial action.

2. The system of claim 1, further comprising:

a visible light camera operably coupled to the controller and having a field of view of the vehicle in the field of view thereof;

wherein the controller is further programmed to;

receive a third output of the visible light camera;

if the change between the first output and the second output indicates the anomaly, (a) identify a region of interest (ROI) in the third output corresponding to the anomaly and (b) transmit the ROI to a dispatcher.

3. The system of claim 2, wherein the controller is further programmed to:

receive an input from the dispatcher;

if the input is an instruction to proceed to a cleaning station, autonomously drive the vehicle to the cleaning station.

4. The system of claim 3, wherein the controller is further programmed to:

if the input is an instruction to proceed to a next pickup location, autonomously drive the vehicle to the next pickup location.

5. The system of claim 2, wherein the controller is further programmed to:

receive a fourth output from the visible camera prior to the third output;

evaluate a combination of (a) a difference between the first output and the second output and (b) a difference of the fourth output and the third output; and

if the combination indicates the anomaly, invoke the remedial action.

6. The system of claim 1, wherein the controller is further programmed to capture the first output with the IR camera prior to pickup of a passenger and to capture the second output with the IR camera after drop-off of the passenger.

7. The system of claim 6, wherein the controller is further programmed to:

subtract the first output from the second output to obtain a difference image; and

evaluate the difference image to determine whether the first output and the second output indicate the anomaly.

8. The system of claim 7, wherein the controller is further programmed to:

identify pixels in the difference image that exceed a threshold; and

evaluate whether the pixels in the difference image that exceed the threshold indicate the anomaly.

9. The system of claim 7, wherein the controller is further programmed to:

identify pixels in the difference image that exceed a threshold; and

evaluate whether the pixels in the difference image that exceed the threshold indicate the anomaly.

10. The system of claim 7, wherein the controller is further programmed to:

identify pixels in the difference image that exceed a threshold to obtain above-threshold pixels;

identify clusters of contiguous above-threshold pixels; and

if any of the clusters are larger than a threshold size, identify the clusters that are larger than the threshold size as anomalies.

12. The method of claim 1, further comprising:

receiving a third output from a visible light camera, a field of view of the visible light camera including the interior of the vehicle;

in response to determining that the change between the first output and the second output indicates the anomaly, (a) identifying a region of interest (ROI) in the third output corresponding to the anomaly and (b) transmitting the ROI to a dispatcher.

13. The method of claim 12, further comprising:

receiving an input from the dispatcher including an instruction to proceed to a cleaning station; and

in response to the input, autonomously driving the vehicle to the cleaning station.

14. The method of claim 12, further comprising:

receiving an input from the dispatcher including an instruction to proceed to a next pickup location; and

in response to the input, autonomously driving the vehicle to the next pickup location.

15. The method of claim 12, further comprising:

receiving a fourth output from the visible camera prior to the third output;

evaluating a combination of a difference between the first output and the second output and a difference of the fourth output and the third output; and

determining that the combination indicates the anomaly.

11. A method comprising, by a controller of a vehicle:

receiving a first output of an IR camera having an interior of the vehicle in a field of view thereof;

receiving a second output of the IR camera subsequent to the first output;

determining that a change between the first output and the second output indicates an anomaly; and

in response to determining that the change between the first output and the second output indicates the anomaly, invoking a remedial action.

16. The method of claim 11, further comprising capturing the first output with the IR camera prior to pickup of a passenger and to capturing the second output with the IR camera after drop-off of the passenger.

17. The method of claim 16, further comprising:

subtracting the first output from the second output to obtain a difference image; and

evaluating the difference image to determine whether the first output and the second output indicate the anomaly.

18. The method of claim 17, further comprising:

identifying pixels in the difference image that exceed a threshold; and

evaluating whether the pixels in the difference image that exceed the threshold indicate the anomaly.

19. The method of claim 17, further comprising:

identifying pixels in the difference image that exceed a threshold; and

evaluating whether the pixels in the difference image that exceed the threshold indicate the anomaly.

20. The method of claim 17, further comprising:

identifying pixels in the difference image that exceed a threshold to obtain above-threshold pixels;

identifying clusters of contiguous above-threshold pixels; and

identifying clusters that are larger than the threshold size as anomalies.