Systems and methods for surveillance of a vehicle using camera images

• US 10,421,436 B2
• Filed: 03/24/2017
• Issued: 09/24/2019
• Est. Priority Date: 03/24/2017
• Status: Active Grant

• Current Assignee: Denso International America Incorporated (DENSO Corporation), Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Gage, Sergei I., Gloomis, Jay Robert, Rios, Isabel
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US15/468,213
• Publication Number:
  US 20180272992A1
• Time in Patent Office: 914 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/8073 for vehicle security, e.g. parked vehicle surveillance, burglar detection
• CPC Class Codes: B60R 25/102 a signal being sent to a remote location, e.g. a radio signal being transmitted to a police station, a security company or the owner
• CPC Class Codes: B60R 25/305 using a camera
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

System, methods, and other embodiments described herein relate to surveillance of a vehicle. In one embodiment, a method includes, in response to detecting a surveillance event associated with the vehicle, collecting information from one or more sensors of the vehicle about the surveillance event. The method includes classifying the information according to a security profile to identify a threat type and a threat level of the surveillance event. The method includes controlling the vehicle to provide a response to the surveillance event according to the threat type and the threat level.

Claims

1. A surveillance system for providing security for a vehicle, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to, in response to detecting a surveillance event associated with the vehicle, collect information including one or more images from one or more sensors of the vehicle and one or more nearby sources about the surveillance event, wherein the monitoring module includes instructions to collect the information including instructions to acquire at least a portion of the information including the one or more images by wirelessly querying the one or more nearby sources; and

a response module including instructions that when executed by the one or more processors cause the one or more processors to classify the information according to a security profile to identify a threat type and a threat level of the surveillance event, wherein the response module includes instructions to identify the threat type and the threat level including instructions to (i) identify, using image recognition to analyze the one or more images of one or more objects, a type of the one or more objects that are proximate to the vehicle, and (ii) identify threat characteristics of the one or more objects indicative of the threat level, wherein the instructions to identify the threat characteristics include instructions to identify a pose of a person in relation to the vehicle, a biometric signature of the person, and a presence of secondary objects held by the person, and

wherein the response module includes instructions to control the vehicle to provide a response to the surveillance event according to the threat type and the threat level.

2. The surveillance system of claim 1, wherein the monitoring module further includes instructions to detect the surveillance event by monitoring the one or more sensors to detect an activating input associated with the surveillance event including monitoring a stream of data from the one or more sensors, and wherein the activating input is defined according to the security profile to include one or more signals from the one or more sensors that are indicative of the surveillance event.

3. The surveillance system of claim 1, wherein the monitoring module further includes instructions to collect the information by acquiring the one or more images from one or more cameras of the vehicle including activating the one or more cameras to provide the one or more images of a surrounding environment of the vehicle, and wherein the monitoring module further includes instructions to analyze the one or more images to identify the one or more objects in the one or more images associated with the surveillance event and to track the one or more objects while the one or more objects move across separate fields-of-view for the one or more cameras.

4. The surveillance system of claim 1,

wherein the response module further includes instructions to classify the information to identify the threat level by assessing the threat characteristics of the surveillance event according to the security profile, and wherein the threat characteristics include at least (i) a proximity of the one or more objects to the vehicle, (ii) the threat type, and (iii) a duration for which the one or more objects is proximate to the vehicle,

wherein the threat type accounts for attributes of the person including the pose in relation to the vehicle, the presence of secondary objects held by the person, a stature of the person, and the biometric signature.

5. The surveillance system of claim 1, wherein the monitoring module further includes instructions to detect the surveillance event by receiving a wireless signal from the one or more nearby sources including a nearby security system indicating an occurrence of the surveillance event,

wherein the instructions to collect the information include instructions to wirelessly query the one or more nearby sources including instructions to transmit a wireless communication that is a beacon message to the one or more nearby sources in response to the surveillance event and requesting sensor data from sensors of the one or more nearby sources about the surveillance event, and wherein the instructions to transmit the wireless communication include instructions to transmit the beacon message as an ad-hoc communication directly to the one or more nearby sources.

6. The surveillance system of claim 1, wherein the security profile indicates different activating inputs from the one or more sensors that correlate with different types of surveillance events, wherein the security profile specifies correlations between threat levels and threat types for which different responses are to be provided by the vehicle,

wherein the threat type indicates a type of an object associated with the surveillance event, an orientation of the object, and a current speed of the object, and

wherein the type of the object specifies at least whether the object is a vehicle.

7. The surveillance system of claim 1, wherein the response module further includes instructions to control the vehicle to provide the response by determining the response from the security profile as a function of the threat type and the threat level,

wherein the response module further includes instructions to control the vehicle to provide the response by transmitting video to a remote device to query the remote device about whether to generate an alarm for the surveillance event when the threat type and the threat level indicate a marginal threat.

8. The surveillance system of claim 1, wherein the response module further includes instructions to control the vehicle to provide the response by generating an alarm when the threat type and the threat level indicate a severe threat, and wherein the response module further includes instructions to control the vehicle to provide the response by logging the information for the surveillance event when the threat type and the threat level indicate a low threat.

9. A non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to:

in response to detecting a surveillance event associated with a vehicle, collect information from one or more sensors of the vehicle and one or more nearby sources about the surveillance event, wherein the instructions to collect the information including instructions to acquire at least a portion of the information including one or more images by wirelessly querying the one or more nearby sources,

classify the information according to a security profile to identify a threat type and a threat level of the surveillance event including instructions to (i) identify, using image recognition to analyze the one or more images of one or more objects, a type of the one or more objects that are proximate to the vehicle, and (ii) identify threat characteristics of the one or more objects indicative of the threat level, wherein identifying the threat characteristics includes identifying a pose of a person in relation to the vehicle, a biometric signature of the person, and a presence of secondary objects held by the person, and

control the vehicle to provide a response to the surveillance event according to the threat type and the threat level.

10. The non-transitory computer-readable medium of claim 9, wherein the instructions to detect the surveillance event include instructions to monitor the one or more sensors to detect an activating input associated with the surveillance event including monitoring a stream of data from the one or more sensors, and wherein the activating input is defined according to the security profile to include one or more signals from the one or more sensors that are indicative of the surveillance event.

11. The non-transitory computer-readable medium of claim 9, wherein the instructions to collect the information include instructions to acquire images from one or more cameras of the vehicle including activating the one or more cameras to provide the images of a surrounding environment of the vehicle, and wherein the instructions to analyze the images include instructions to identify an object in the images associated with the surveillance event and to track the object while the object moves across separate fields-of-view for the one or more cameras.

12. The non-transitory computer-readable medium of claim 9,

wherein the instructions to classify the information include instructions to identify the threat level by assessing the threat characteristics of the surveillance event according to the security profile, and wherein the threat characteristics include at least (i) a proximity of the one or more objects to the vehicle, (ii) the threat type, and (iii) a duration for which the one or more objects is proximate to the vehicle,

wherein the threat type accounts for attributes of the person including the pose in relation to the vehicle, the presence of secondary objects held by the person, a stature of the person, and the biometric signature.

13. The non-transitory computer-readable medium of claim 9, wherein the instructions to detect the surveillance event include instructions to receive a wireless signal from a nearby security system indicating an occurrence of the surveillance event,

wherein the instructions to collect the information include instructions to wirelessly query the one or more nearby sources including instructions to transmit a wireless communication that is a beacon message to the one or more nearby sources in response to the surveillance event and request sensor data from sensors of the one or more nearby sources about the surveillance event, and wherein the instructions to transmit the wireless communication include instructions to transmit the beacon message as an ad-hoc communication directly to the one or more nearby sources.

14. A method of providing security for a vehicle, comprising:

in response to detecting a surveillance event associated with the vehicle, collecting information from one or more sensors of the vehicle and one or more nearby sources about the surveillance event, wherein collecting the information including one or more images includes acquiring at least a portion of the information including the one or more images by wirelessly querying the one or more nearby sources;

classifying the information according to a security profile to identify a threat type and a threat level of the surveillance event including (i) identifying, using image recognition to analyze the one or more images of one or more objects, a type of the one or more objects that are proximate to the vehicle, and (ii) identifying threat characteristics of the one or more objects indicative of the threat level to the vehicle,

wherein identifying the threat characteristics includes identifying a pose of a person in relation to the vehicle, a biometric signature of the person, and a presence of secondary objects held by the person; and

controlling the vehicle to provide a response to the surveillance event according to the threat type and the threat level.

15. The method of claim 14, wherein detecting the surveillance event includes monitoring the one or more sensors to detect an activating input associated with the surveillance event by monitoring a stream of data from the one or more sensors, and wherein the activating input is defined according to the security profile to include one or more signals from the one or more sensors that are indicative of the surveillance event.

16. The method of claim 14, wherein collecting the information includes acquiring the one or more images from one or more cameras of the vehicle by activating the one or more cameras to provide the images of a surrounding environment of the vehicle, and

wherein acquiring the images includes analyzing the images to identify an object in the images associated with the surveillance event and tracking the object while the object moves across separate fields-of-view for the one or more cameras.

17. The method of claim 14, wherein classifying the information to identify the threat level includes assessing the threat characteristics of the surveillance event according to the security profile, wherein the threat characteristics include at least (i) a proximity of the one or more objects to the vehicle, (ii) the threat type, and (iii) a duration for which the one or more objects is proximate to the vehicle, and

wherein the threat type accounts for attributes of the person including the pose in relation to the vehicle, the presence of secondary objects held by the person, a stature of the person, and the biometric signature of the person.

18. The method of claim 14, wherein detecting the surveillance event includes receiving a wireless signal from a nearby security system indicating an occurrence of the surveillance event,

wherein wirelessly querying the one or more nearby sources includes transmitting a wireless communication that is a beacon message to the one or more nearby sources in response to the surveillance event and requesting sensor data from sensors of the one or more nearby sources about the surveillance event, and

wherein transmitting the wireless communication includes transmitting the beacon message as an ad-hoc communication directly to the one or more nearby sources.

19. The method of claim 15, wherein the security profile indicates different activating inputs from the one or more sensors that correlate with different types of surveillance events, wherein the security profile specifies correlations between threat levels and threat types for which different responses are to be provided by the vehicle,

wherein the threat type indicates a type of an object associated with the surveillance event, an orientation of the object, and a current speed of the object, and

wherein the type of the object specifies at least whether the object is a vehicle.

20. The method of claim 14, wherein controlling the vehicle to provide the response includes determining the response from the security profile as a function of the threat type and threat level,

wherein the response includes transmitting video to a remote device to query the remote device about whether to generate an alarm for the surveillance event when the threat type and the threat level indicate a marginal threat,

wherein the response includes generating an alarm when the threat type and the threat level indicate a severe threat, and wherein the response includes logging the information for the surveillance event when the threat type and the threat level indicate a low threat.