Detecting and responding to autonomous environment incidents

• US 10,469,282 B1
• Filed: 01/18/2017
• Issued: 11/05/2019
• Est. Priority Date: 01/22/2016
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Konrardy, Blake, Christensen, Scott T., Hayward, Gregory, Farris, Scott
• Primary Examiner(s): Ortiz Rodriguez, Carlos R
• Application Number:
  US15/409,371
• Time in Patent Office: 1,021 Days
• Field of Search: None
• CPC Class Codes: B60L 53/36 by positioning the vehicle
• CPC Class Codes: B60L 58/12 responding to state of charge [SoC]
• CPC Class Codes: B60P 3/12 for salvaging damaged vehicles
• CPC Class Codes: B60R 16/0234 related to maintenance or repairing of vehicles
• CPC Class Codes: B60R 2021/0027 Post collision measures, e.g. notifying emergency services
• CPC Class Codes: B60R 2021/01013 Means for detecting collision, impending collision or roll-over
• CPC Class Codes: B60R 2025/1013 Alarm systems characterised by the type of warning signal, e.g. visual, audible
• CPC Class Codes: B60R 21/0136 responsive to actual contact with an obstacle , e.g. to vehicle deformation, bumper displacement or bumper velocity relative to the vehicle monitoring crash strength in combination with passenger seat detection B60R21/01558
• CPC Class Codes: B60R 21/34 Protecting non-occupants of a vehicle, e.g. pedestrians B60R19/02 takes precedence
• CPC Class Codes: B60R 25/04 operating on the propulsion system, e.g. engine or drive motor
• CPC Class Codes: B60R 25/10 actuating a signalling device
• CPC Class Codes: B60R 25/1001 Alarm systems associated with another car fitting or mechanism, e.g. door lock or knob, pedals features related to the general power supply B60R25/1018
• CPC Class Codes: B60R 25/102 a signal being sent to a remote location, e.g. a radio signal being transmitted to a police station, a security company or the owner
• CPC Class Codes: B60R 25/104 characterised by the type of theft warning signal, e.g. visual or audible signals with special characteristics
• CPC Class Codes: B60R 25/252 Fingerprint recognition
• CPC Class Codes: B60R 25/255 Eye recognition
• CPC Class Codes: B60R 25/305 using a camera
• CPC Class Codes: B60R 25/31 of human presence inside or outside the vehicle
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 : including control of steering systems
• CPC Class Codes: B60W 2420/403 : Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 : Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2530/209 : Fuel quantity remaining in tank
• CPC Class Codes: B60W 2540/229 : Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2552/05 : Type of road, e.g. motorways, local streets, paved or unpaved roads
• CPC Class Codes: B60W 2552/35 : Road bumpiness, e.g. potholes
• CPC Class Codes: B60W 2554/4026 : Cycles
• CPC Class Codes: B60W 2554/4029 : Pedestrians
• CPC Class Codes: B60W 2554/4041 : Position
• CPC Class Codes: B60W 2554/406 : Traffic density
• CPC Class Codes: B60W 2556/10 : Historical data
• CPC Class Codes: B60W 30/0956 : the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 30/12 : Lane keeping
• CPC Class Codes: B60W 30/16 : Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 30/18163 : Lane change; Overtaking manoeuvres
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 60/0023 : in response to energy consumption
• CPC Class Codes: B60W 60/0053 : from vehicle to occupant
• CPC Class Codes: B60W 60/0059 : Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
• CPC Class Codes: G01B 21/00 : Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
• CPC Class Codes: G01C 21/34 : Route searching; Route guidance
• CPC Class Codes: G01C 21/3415 : Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/343 : Calculating itineraries, i.e. routes leading from a starting point to a series of categorical destinations using a global route restraint, round trips, touristic trips travelling salesman problem G06Q10/04; optimisation of routes G06Q10/047
• CPC Class Codes: G01C 21/3438 : Rendez-vous, i.e. searching a destination where several users can meet, and the routes to this destination for these users; Ride sharing, i.e. searching a route such that at least two users can share a vehicle for at least part of the route
• CPC Class Codes: G01C 21/3453 : Special cost functions, i.e. other than distance or default speed limit of road segments
• CPC Class Codes: G01C 21/3461 : Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types, segments such as motorways, toll roads, ferries
• CPC Class Codes: G01C 21/3469 : Fuel consumption; Energy use; Emission aspects
• CPC Class Codes: G01C 21/3617 : using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G01C 21/362 : received from an external device or application, e.g. PDA, mobile phone or calendar application
• CPC Class Codes: G01C 21/3697 : Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G01S 19/13 : Receivers
• CPC Class Codes: G01S 19/42 : Determining position
• CPC Class Codes: G05B 15/02 : electric
• CPC Class Codes: G05B 23/0245 : based on a qualitative model, e.g. rule based; if-then decisions
• CPC Class Codes: G05D 1/0011 : associated with a remote control arrangement
• CPC Class Codes: G05D 1/0055 : with safety arrangements
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0231 : using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0255 : using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0285 : using signals transmitted via a public communication network, e.g. GSM network
• CPC Class Codes: G05D 1/0287 : involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G05D 1/0289 : with means for avoiding collisions between vehicles vehicle fittings for automatically controlling speed including means for detecting potential obstacles B60K31/0008; avoiding obstacles by action on the steering system B62D; radar, sonar, lidar systems designed for anti-collision purposes G01S13/93, G01S15/93, G01S17/93
• CPC Class Codes: G05D 1/0293 : Convoy travelling
• CPC Class Codes: G05D 1/0295 : by at least one leading vehicle of the fleet
• CPC Class Codes: G05D 1/223 : Command input arrangements on the remote controller, e.g. joysticks or touch screens
• CPC Class Codes: G05D 1/227 : Handing over between remote control and on-board control; Handing over between remote control arrangements
• CPC Class Codes: G05D 1/228 : Command input arrangements located on-board unmanned vehicles
• CPC Class Codes: G05D 1/247 : using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
• CPC Class Codes: G05D 1/249 : from positioning sensors located off-board the vehicle, e.g. from cameras
• CPC Class Codes: G05D 1/617 : Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80
• CPC Class Codes: G05D 1/646 : Following a predefined trajectory, e.g. a line marked on the floor or a flight path
• CPC Class Codes: G05D 1/69 : Coordinated control of the position or course of two or more vehicles
• CPC Class Codes: G05D 1/692 : involving a plurality of disparate vehicles
• CPC Class Codes: G05D 1/693 : for avoiding collisions between vehicles
• CPC Class Codes: G05D 1/695 : for maintaining a fixed relative position of the vehicles, e.g. for convoy travelling or formation flight
• CPC Class Codes: G05D 1/697 : for rendezvous of two or more vehicles, e.g. for in-flight refuelling docking at a base station G05D1/661
• CPC Class Codes: G06F 11/3688 : for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 : for test results analysis
• CPC Class Codes: G06F 16/2455 : Query execution
• CPC Class Codes: G06F 16/90335 : Query processing
• CPC Class Codes: G06F 17/00 : Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/55 : Detecting local intrusion or implementing counter-measures
• CPC Class Codes: G06F 2221/034 : Test or assess a computer or a system
• CPC Class Codes: G06F 30/15 : Vehicle, aircraft or watercraft design
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06Q 10/1095 : Meeting or appointment
• CPC Class Codes: G06Q 10/20 : Administration of product repair or maintenance
• CPC Class Codes: G06Q 30/0284 : Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 30/0645 : Rental transactions; Leasing transactions
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06Q 50/163 : Real estate management
• CPC Class Codes: G06Q 50/265 : Personal security, identity or safety
• CPC Class Codes: G06Q 50/40 : Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G07C 5/006 : Indicating maintenance
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0816 : Indicating performance data, e.g. occurrence of a malfunction
• CPC Class Codes: G07C 5/0841 : Registering performance data recording measured values G01D; information storage G11B
• CPC Class Codes: G07C 9/00563 : using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns
• CPC Class Codes: G08B 21/00 : Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
• CPC Class Codes: G08B 21/02 : Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/18 : Status alarms G08B21/02 takes precedence
• CPC Class Codes: G08B 25/00 : Alarm systems in which the location of the alarm condition is signalled to a central station, e.g. fire or police telegraphic systems
• CPC Class Codes: G08B 25/014 : Alarm signalling to a central station with two-way communication, e.g. with signalling back
• CPC Class Codes: G08G 1/017 : identifying vehicles G08G1/015, G08G1/054 take precedence
• CPC Class Codes: G08G 1/0965 : responding to signals from another vehicle, e.g. emergency vehicle
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096783 : where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/146 : where the parking area is a limited parking space, e.g. parking garage, restricted space
• CPC Class Codes: G08G 1/148 : Management of a network of parking areas
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: G08G 1/20 : Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles
• CPC Class Codes: G08G 1/205 : Indicating the location of the monitored vehicles as destination, e.g. accidents, stolen, rental
• CPC Class Codes: G16Y 10/80 : Homes; Buildings
• CPC Class Codes: G16Y 30/00 : IoT infrastructure
• CPC Class Codes: H04L 12/2803 : Home automation networks
• CPC Class Codes: H04L 12/2816 : Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/2825 : Reporting to a device located outside the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; telephonic communication systems adapted for combination with telemetering systems H04M11/002
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: Y02A 10/40 : Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
• CPC Class Codes: Y02T 10/70 : Energy storage systems for electromobility, e.g. batteries
• CPC Class Codes: Y02T 10/7072 : Electromobility specific charging systems or methods for batteries, ultracapacitors, supercapacitors or double-layer capacitors
• CPC Class Codes: Y02T 90/12 : Electric charging stations

Abstract

Methods and systems for assessing, detecting, and responding to malfunctions involving components of autonomous vehicles and/or smart homes are described herein. Autonomous operation features and related components can be assessed using direct or indirect data regarding operation. Vehicle collision and/or smart home incident monitoring, damage detection, and responses are also described, with particular focus on the particular challenges associated with incident response for unoccupied vehicles and/or smart homes. Operating data associated with the autonomous vehicle and/or smart home may be received. Within the operating, an unusual condition indicative of a likelihood of incident may be detected. Based on the unusual condition, it may be determined that the incident occurred. Accordingly, a response to the incident may be determined. The response may be implemented by the autonomous vehicle and/or smart home.

Claims

1. A computer-implemented method for detecting and responding to a smart home incident, the method being implemented at a smart home controller storing a plurality of software routines and comprising:

detecting, by one or more processors, that a smart home is operating in a fully autonomous mode without occupants;

monitoring, by one or more processors executing a first sensor control routine of the plurality of software routines, operation of the smart home by receiving and processing operating data associated with a plurality of autonomous operation features while the smart home is operating;

identifying, by one or more processors, an unusual condition in the operating data, the unusual condition indicative of a likelihood of an incident occurring during operation of the smart home;

determining, by one or more processors, that an incident has occurred based upon a portion of the operating data associated with the unusual condition, wherein determining that an incident has occurred includes determining that damage has occurred to one or more components of the smart home;

selecting, by one or more processors, a response to the incident from a plurality of responses based upon the received operating data;

causing, by one or more processors, the smart home to implement the selected response by calling an autonomous control routine of the plurality of software routines associated with the selected response to automatically control smart equipment disposed in the smart home, wherein implementing the selected response includes continuing to operate the smart home; and

in response to the smart home continuing to operate, further monitoring, by the one or more processors executing a second sensor control routine of the plurality of software routines, further operation of the smart home to identify a further unusual condition, wherein the second sensor control routine configures the one or more processors to exclude further operating data received from the one or more damaged components of the smart home when identifying the further unusual condition.

2. The computer-implemented method of claim 1, wherein the operating data includes sensor data from a plurality of sensors associated with the autonomous operation features and control data generated by the plurality of autonomous operation features.

3. The computer-implemented method of claim 1, further comprising:

transmitting, via a communication network, (i) image data from one or more image capture devices within the smart home and/or (ii) operating data associated with the smart home to a computing device associated with a human reviewer; and

receiving, via the communication network, an indication verifying the determined incident,

wherein the response is selected based upon the received indication verifying the determined incident.

4. The computer-implemented method of claim 3, wherein the indication verifying the determined incident further includes one or more instructions regarding the response to be implemented by the smart home.

5. The computer-implemented method of claim 1, wherein the unusual condition includes a discontinuity in the operating data, a divergence between sensor data from one or more sensors and control data from one or more autonomous operation features, or a plurality of sensor malfunctions.

6. The computer-implemented method of claim 1, further comprising:

determining, by one or more processors, that damage to another object has occurred based upon the received operating data; and

communicating, by one or more processors, information regarding the damage to another object.

7. The computer-implemented method of claim 1, further comprising:

determining, by one or more processors, that the determined incident has resulted in an injury to a person outside the smart home; and

causing, by one or more processors, an emergency action to be implemented.

8. The computer-implemented method of claim 7, wherein the emergency action includes enabling communication between the person and an emergency assistance representative using (i) one or more speakers disposed within the smart home and/or (ii) video equipment disposed within the smart home.

9. The computer-implemented method of claim 1, wherein the response includes establishing a two-way communication link between the smart home and a remote user using input and output components of the smart home.

10. The computer-implemented method of claim 9, wherein:

the remote user is an owner or operator of the smart home,

the input component includes a microphone and/or a camera, and

the output component includes a speaker and/or a display.

11. The computer-implemented method of claim 1, wherein the response includes allowing manual control of the smart home for a limited duration of operation following the incident.

12. The computer-implemented method of claim 1, wherein the response includes granting autonomous control of the smart home to a remote agent following the incident.

13. A computer system configured to detect and respond to a smart home incident, comprising:

one or more processors;

a communication module adapted to communicate with a plurality of components associated with autonomous operation features of the smart home;

a database storing a plurality of software routines; and

a non-transitory program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to;

detect that a smart home is operating in a fully autonomous mode without occupants;

monitor operation of the smart home via a first sensor control routine of the plurality of software routines and by receiving, via the communication module, and processing operating data associated with the plurality of autonomous operation features while the smart home is operating;

identify an unusual condition in the operating data, such unusual condition indicative of a likelihood of an incident occurring during operation of the smart home;

determine that an incident has occurred based upon a portion of the operating data associated with the unusual condition, wherein determining that an incident has occurred includes determining that damage has occurred to one or more components of the plurality of components;

select a response to the incident from a plurality of responses based upon the received operating data;

cause the smart home to implement the selected response by calling an autonomous control routine of the plurality of software routines associated with the selected response to automatically control at least one of the plurality of components, wherein implementing the selected response includes continuing to operate the smart home;

and in response to the smart home continuing to operate, further monitor via a second sensor control routine of the plurality of software routines, further operation of the smart home to identify a further unusual condition, wherein the second sensor control routine excludes analyzing further operating data received from the one or more damaged components of the smart home when identifying the further unusual condition.

14. The computer system of claim 13, wherein, the instructions, when executed by the one or more processors, further cause the computer system to:

transmit, via a communication network, (i) image data from one or more image capture devices within the smart home and/or (ii) operating data associated with the smart home to a computing device associated with a human reviewer; and

receive, via the communication network, an indication verifying the determined incident,

wherein the response is selected based upon the received indication verifying the determined incident.

15. The computer system of claim 14, wherein the indication verifying the determined incident further includes one or more instructions regarding the response to be implemented by the smart home.

16. The computer system of claim 13, wherein the response includes establishing a two-way communication link between the smart home and a remote user using input and output components of the smart home.

17. The computer system of claim 16, wherein:

the remote user is an owner or operator of the smart home,

the input component includes a microphone and/or a camera, and

the output component includes a speaker and/or a display.

18. A non-transitory computer-readable storage medium storing processor-executable instructions, that when executed cause one or more processors to:

detect that a smart home is operating in a fully autonomous mode without occupants;

execute a first sensor control routine to monitor operation of the smart home by receiving, via a communication network, and processing operating data associated with a plurality of autonomous operation features while the smart home is operating;

identify an unusual condition in the operating data, such unusual condition indicative of a likelihood of an incident occurring during operation of the smart home;

determine that an incident has occurred based upon a portion of the operating data associated with the unusual condition, wherein determining that an incident has occurred includes determining that damage has occurred to one or more components of the smart home;

select a response to the incident from a plurality of responses based upon the received operating data;

cause the smart home to implement the selected response by calling an autonomous control routine associated with the selected response to automatically control smart equipment disposed in the smart home, wherein implementing the selected response includes continuing to operate the smart home; and

in response to the smart home continuing to operate, execute a second sensor control routine to further monitor further operation of the smart home to identify a further unusual condition, wherein the second sensor control routine excludes analyzing further operating data received from the one or more damaged components of the smart home when identifying the further unusual condition.