Vehicle stain and trash detection systems and methods

• US 10,304,165 B2
• Filed: 05/12/2017
• Issued: 05/28/2019
• Est. Priority Date: 05/12/2017
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Tokatyan, Shant
• Primary Examiner(s): Werner, Brian
• Application Number:
  US15/593,880
• Publication Number:
  US 20180330475A1
• Time in Patent Office: 746 Days
• CPC Class Codes: B60S 1/64 for cleaning vehicle interiors, e.g. built-in vacuum cleaners
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions

Abstract

Example vehicle stain and trash detection systems and methods are described. In one implementation, a method receives a clean image and clean Lidar data associated with a clean vehicle interior. The method also receives a second image and second Lidar data associated with the vehicle interior after a passenger has occupied the vehicle. Differences are identified between the clean image and the second image. Additionally, differences are identified between the clean Lidar data and the second Lidar data. The method determines whether the vehicle interior includes at least one of a stain, dirt, or trash based on the identified differences.

Claims

1. A method comprising:

receiving a clean image and clean Lidar data associated with a clean vehicle interior;

receiving a second image and second Lidar data associated with the vehicle interior after a passenger has occupied the vehicle;

identifying differences between the clean image and the second image;

identifying differences between the clean Lidar data and the second Lidar data; and

determining whether the vehicle interior includes at least one of a stain, dirt, or trash;

wherein determining whether the vehicle interior includes at least one of a stain, dirt, or trash includes classifying the identified differences as two-dimensional or three-dimensional.

2. The method of claim 1, further comprising quantizing the clean image and the second image to reduce noise.

3. The method of claim 1, further comprising identifying at least one contour in the identified differences between the clean image and the second image.

4. The method of claim 3, further comprising associating a bounding box with the identified contour.

5. The method of claim 1, wherein determining whether the vehicle interior includes at least one of a stain, dirt, or trash is based on at least one of differences between the clean image and the second image, and differences between the clean Lidar data and the second Lidar data.

6. The method of claim 1, wherein identifying differences between the clean image and the second image includes subtracting the second image from the clean image.

7. The method of claim 1, wherein identifying differences between the clean Lidar data and the second Lidar data includes identifying differences in the point clouds of the clean Lidar data and the second Lidar data.

8. The method of claim 1, further comprising classifying two-dimensional differences as stains and classifying three-dimensional differences as at least one of dirt or trash.

9. The method of claim 1, further comprising determining a shape associated with the three-dimensional differences.

10. The method of claim 1, further comprising determining dimensions associated with the three-dimensional differences.

11. The method of claim 1, further comprising determining whether the vehicle interior needs to be cleaned based on determining whether the vehicle interior includes at least one of a stain, dirt, or trash.

12. The method of claim 1, wherein the vehicle is an autonomous vehicle.

13. A method comprising:

receiving a clean image and clean Lidar data associated with a clean vehicle interior;

receiving a second image and second Lidar data associated with the vehicle interior after a passenger has occupied the vehicle;

identifying, by a vehicle stain and trash detection system, differences between the clean image and the second image;

identifying, by the vehicle stain and trash detection system, differences between the clean Lidar data and the second Lidar data; and

determining whether the vehicle interior includes at least one of a stain, dirt, or trash based on;

the identified differences between the clean image and the second image; and

the identified differences between the clean Lidar data and the second Lidar data;

wherein determining whether the vehicle interior includes at least one of a stain, dirt, or trash further includes classifying the identified differences as two-dimensional or three-dimensional.

14. The method of claim 13, wherein identifying differences between the clean Lidar data and the second Lidar data includes identifying differences in the point clouds of the clean Lidar data and the second Lidar data.

15. The method of claim 13, further comprising classifying two-dimensional differences as stains and classifying three-dimensional differences as at least one of dirt or trash.

16. An apparatus comprising:

a communication manager configured to receive a clean image and clean Lidar data associated with a clean vehicle interior, and configured to receive a second image and second Lidar data associated with the vehicle interior after a passenger has occupied the vehicle;

an image processing module configured to identify differences between the clean image and the second image;

a Lidar processing module configured to identify differences between the clean Lidar data and the second Lidar data; and

a classification module configured to classify an area within the vehicle as one of a stain, dirt, trash, or another item based on at least one of differences between the clean image and the second image, and differences between the clean Lidar data and the second Lidar data;

wherein the classification module is further configured to identify the identified the stain, dirt, trash, or other item as two-dimensional or three-dimensional.

17. The apparatus of claim 16, wherein identifying differences between the clean Lidar data and the second Lidar data includes identifying differences in the point clouds of the clean Lidar data and the second Lidar data.