Object detection and passenger notification

• US 10,303,961 B1
• Filed: 04/13/2017
• Issued: 05/28/2019
• Est. Priority Date: 04/13/2017
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Stoffel, Christopher John, Kentley-Klay, Timothy David
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/486,559
• Time in Patent Office: 775 Days
• Field of Search: 348148
• CPC Class Codes: B60Q 1/507 specific to autonomous vehicles
• CPC Class Codes: B60Q 3/76 for spotlighting, e.g. reading lamps spotlighting on or for specific fittings B60Q3/20; spotlighting in mass transit vehicles B60Q3/44
• CPC Class Codes: B60Q 3/80 Circuits; Control arrangements for dashboards B60Q3/16; for mass transit vehicles B60Q3/47
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: F21V 21/30 Pivoted housings or frames
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3697 Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G08B 21/24 Reminder alarms, e.g. anti-loss alarms
• CPC Class Codes: G08B 25/10 using wireless transmission systems G08B25/009 takes precedence
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements
• CPC Class Codes: H04W 4/38 for collecting sensor information
• CPC Class Codes: H04W 4/44 for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

A system for detecting a classifying objects in a vehicle can comprise one or more sensors for detecting the presence of an object, other than a passenger, on the vehicle. The system can classify the object as, for example, a personal object or trash. The system can provide a series of escalating reminders for the passenger to take the object with them when they exit the vehicle. When a personal object is left in the vehicle after the passenger has exited the vehicle, the system can communicate with a computing device of a central control or the passenger using additional communication channels such as, for example, e-mail, text, or the Internet. When trash is left in the vehicle, the vehicle can automatically contact, or return to, a maintenance facility for servicing. This can prevent the use of unclean vehicles for new passengers, among other things.

Claims

1. A vehicle comprising:

one or more internal sensors to;

generate data associated with an interior portion of the vehicle; and

send the data to an object classification system;

the object classification system in communication with at least the one or more internal sensors to;

receive the data from the at least one of the one or more internal sensors;

input the data into a first machine learned model configured to determine a presence of an object in the vehicle;

input, based at least in part on the determination, a portion of the data associated with the object into a second machine learned model configured to output an object classification;

receive, from the second machine learned model, the object classification;

memory storing instructions that, when executed by one or more processors, cause the one or more processors to;

determine that the vehicle is within a distance from a destination; and

send an indication to a notification system that the vehicle is within the distance from the destination; and

the notification system to;

receive the indication from the navigation system; and

provide one or more of an audio or a visual reminder to a user to remove the object from the vehicle based at least on the object classification and the indication.

2. The vehicle of claim 1, wherein the one or more internal sensors comprise an image sensor;

and

wherein the object classification system further comprises an image processor to interpret images from the image sensor.

3. The vehicle of claim 1, wherein the notification system provides the one or more audio or visual reminder in the interior of the vehicle.

4. The vehicle of claim 1, wherein the one or more internal sensors comprise a weight sensor disposed in at least one of a seat, a floor, or a compartment of the vehicle to detect a weight of the object.

5. The vehicle of claim 1, wherein the notification system comprises:

a plurality of lights, each light of the plurality of lights to illuminate a specific portion of the interior of the vehicle;

wherein a first light of the plurality of lights activates to remind the user to remove the object from the vehicle; and

wherein the first light is a closest light of the plurality of lights to the object.

6. The vehicle of claim 1, wherein the notification system comprises:

an articulating spotlight moveable to illuminate a plurality of locations in the interior of the vehicle;

wherein the articulating spotlight moves to point to an area proximate the object and activates to illuminate the area proximate the object to remind the user to remove the object from the vehicle.

7. The vehicle of claim 1, wherein the object classification system determines a level of reminder associated with the object classification, the level of reminder indicating at least one of a frequency of providing the reminder to the user, a type of reminder to provide to the user, or an event triggering the reminder;

and

wherein the notification system provides the reminder to the user based at least in part on the level of reminder.

8. A method comprising:

detecting, based at least in part on data from one or more internal sensors of a vehicle a presence of an object in the vehicle, the object associated with an object classification;

determining, with a vehicle navigation system, that the vehicle is a first distance from a destination; and

based at least in part on the object classification and determining that the vehicle is the first distance from the destination, providing, with a vehicle notification system via at least one output device disposed within the vehicle, a first reminder in an interior of the vehicle to remind a passenger to remove the object from the vehicle.

9. The method of claim 8, further comprising:

detecting with the one or more internal sensors that the passenger has not retrieved the object;

determining, with the vehicle navigation system, that the vehicle is a second predetermined distance from the destination; and

providing, with the vehicle notification system via at least one output device disposed within the vehicle, a second reminder in the interior of the vehicle to remind the passenger to remove the object from the vehicle.

10. The method of claim 9, wherein the second distance is shorter than then first distance;

and

wherein the second reminder is escalated when compared to the first reminder.

11. The method of claim 9, wherein the first reminder and the second reminder contain an audio component;

and

wherein the second reminder is louder than the first reminder.

12. The method of claim 9, further comprising:

detecting with the one or more internal sensors that the passenger has not retrieved the object;

detecting with the one or more internal sensors that at least one door on the vehicle is open; and

providing, with the vehicle notification system, a third reminder in the interior of the vehicle to remind the passenger to remove the object from the vehicle.

13. The method of claim 12, further comprising:

detecting with the one or more internal sensors that the passenger has not retrieved the object;

detecting with the one or more internal sensors that the passenger has exited the vehicle;

detecting with the one or more internal sensors that the at least one door on the vehicle is closed; and

providing, with the vehicle notification system, a fourth notification on an exterior of the vehicle to remind the passenger to remove the object from the vehicle.

14. The method of claim 8, further comprising:

detecting with the one or more internal sensors that the passenger has not retrieved the object;

detecting with the one or more internal sensors that the passenger has exited the vehicle; and

sending, with the vehicle notification system, a message to at least one of the passenger or a central control indicating that the object has not been retrieved from the vehicle.

15. A method comprising:

detecting with one or more internal sensors a presence of an object in a vehicle;

determining a classification of the object with using an object classification system; and

providing, with a vehicle notification system, a first reminder in an interior of the vehicle to remind a passenger to remove the object from the vehicle, the first reminder based at least in part on the classification of the object.

16. The method of claim 15, further comprising:

determining, with the one or more internal sensors, that the passenger has exited the vehicle;

determining that the classification of the object exceeds a classification; and

sending, from a vehicle communications system to an electronic device associated with the passenger, a message regarding the object.

17. The method of claim 15, further comprising:

navigating, with a vehicle navigation system, from the vehicle'"'"'s current location to a maintenance facility to have the object removed.

18. The method of claim 15, wherein classifying the object with an object classification system comprises determining that the object is trash, and further comprising:

sending, with the vehicle notification system, a message to a service facility regarding the object.

19. The method of claim 15, wherein classifying the object with an object classification system comprises:

receiving image data at an image processor from one or more interior imagers; and

analyzing the image data using one or more machine learning procedures to obtain an image shape profile.

20. The method of claim 19, wherein the one or more machine learning procedures comprise one or more selected from the group consisting of a neural network, a support vector machine, or an analysis involving random decision forest regression.