OBJECT DETECTION AND ANALYSIS VIA UNMANNED AERIAL VEHICLE

• US 20170053169A1
• Filed: 08/20/2015
• Published: 02/23/2017
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Radical Urban, LLC
• Original Assignee: Motionloft, Inc.
• Inventors: Reitman, Joyce, Pritchard, Jeffrey Orion, Cuban, Mark
• Granted Patent: US 9,989,965 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G05D 1/0202 specially adapted to aircraft
• CPC Class Codes: G05D 1/104 involving a plurality of aircrafts, e.g. formation flying traffic control systems for aircraft G08G5/00
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An unmanned aerial vehicle (UAV) can include one or more cameras for capturing image data within a field of view that depends in part upon the location and orientation of the UAV. At least a portion of the image data can be processed on the UAV to locate objects of interest, such as people or cars, and use that information to determine where to fly the drone in order to capture higher quality image data of those or other such objects. Once identified, the objects of interest can be counted, and the density, movement, location, and behavior of those objects identified. This can help to determine occurrences such as traffic congestion or unusual patterns of pedestrian movement, as well as to locate persons, fires, or other such objects. The data can also be analyzed by a remote system or service that has additional resources to provide more accurate results.

Claims

1. A device for monitoring motion of objects of interest, comprising:

a vehicle body;

a flight mechanism for enabling the vehicle body to be directed along an aerial flight path;

at least one camera;

at least one device sensor including at least one of an orientation sensor or a motion sensor;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the device to;

capture, as the device follows at least a portion of the aerial flight path during the period of time, image data using the at least one camera, the portion of the aerial flight path selected such that an area of interest is within a field of view of the at least one camera during the period of time;

determine device motion data for the device, over the period of time, using data from the at least one device sensor;

analyze, using the at least one processor, the image data to recognize instances of at least one type of object of interest represented in the image data and corresponding to the area of interest, the at least one type of object of interest including at least one of a pedestrian, a motorized vehicle, or a bicycle;

determine a number of the instances of the at least one type of object of interest represented in the image data, each of the number of instances having an associated confidence level for recognition;

determine a respective path of motion for each of at least a subset of the instances, the respective path of motion determined using the device motion data to account for motion effects in the image data resulting from movement of the device; and

transmit information about the number of instances of the at least one type of object of interest and the respective path of motion for the area of interest over the period of time.

2. The device of claim 1, wherein the instructions when executed further cause the device to:

determine a sub-region within the area of interest where at least one object of interest has an associated confidence level that falls below a minimum confidence threshold; and

modify the aerial flight path to cause the camera to capture more detailed image data for the at least one object of interest.

3. The device of claim 1, wherein the instructions when executed further cause the device to:

transfer the image data to a data analysis system for subsequent analysis, the data analysis system configured to use a deep neural network-based analysis process to generate a second determination of the number of instances of the at least one type of object of interest, the data analysis system configured to provide updated training data to the device based at least in part from the second determination.

4. The device of claim 1, further comprising:

a signal sensor capable of detecting signal information from at least a subset of the instances of at least one type of object of interest, the determining of at least one of the number of the instances or the respective path of motion being based at least in part upon the signal information.

5. The device of claim 1, wherein the instructions when executed further cause the device to:

determine, based at least in part upon location data and the respective path of motion for the instances of the objects of interest, an irregular object behavior;

sending a notification regarding the irregular object behavior; and

attempting to capture additional image data regarding the irregular object behavior.

6. The device of claim 5, wherein the irregular object behavior corresponds to at least one of a demonstration, a riot, a flash mob, an accident, an emergency, or a point of congestion.

7. The device of claim 5, wherein the device is a first unmanned aerial vehicle (UAV), and wherein the notification is sent to at least one other UAV in order to cause the at least one other UAV to move toward a location of the irregular object behavior for purposes of capturing additional image data corresponding to the irregular object behavior.

8. A method of determining instances of objects of interest, comprising:

capturing, using a device sensor of an unmanned aerial vehicle (UAV), sensor data for an area of interest;

analyzing, using at least one processor of the UAV, the sensor data to recognize instances of objects of interest represented in the sensor data, the instances recognized using at least one object recognition algorithm executed by the at least one processor of the UAV;

determining at least one of a number of the instances or a respective location of each of the instances of the objects of interest within the area of interest; and

providing summary data including information about at least one of the number of instances or the respective locations within the area of interest.

9. The method of claim 8, further comprising:

capturing the sensor data over a period of time;

determining paths of motion for the objects of interest; and

including information about the paths of motion in the summary data.

10. The method of claim 8, further comprising:

analyzing the one or more paths of motion to determine a location of an occurrence associated with the paths of motion, and

causing the UAV to fly along a flight path enabling the camera to capture additional sensor information corresponding to the location of the occurrence.

11. The method of claim 8, wherein the occurrence is one of a parade, a demonstration, a spontaneous gathering, a traffic flow, traffic congestion, an accident, a flash mob, a concert, a riot, a fire, or a deviation from an expected pattern of object motion.

12. The method of claim 8, further comprising:

sending instruction data to at least a second UAV to cause the second UAV to capture a different set of sensor data with respect to the area of interest.

13. The method of claim 8, further comprising:

capturing additional sensor data using at least a second sensor on the device, the additional sensor data including at least one of heat signature data, radio frequency data, audio data, or motion data; and

using the additional sensor data to determine the at least one of a number of the instances or a respective location of each of the instances of the objects within the area of interest.

14. The method of claim 8, further comprising:

determining, using at least one of an orientation sensor or a motion sensor, motion data for the UAV over a period of time in which the sensor data is captured; and

using the motion data to account for changes in positions of the objects of interest in the sensor data that correspond to movement of the UAV.

15. The method of claim 8, further comprising:

modifying a flight path of the UAV based at least in part upon at least one of a density, a movement, a number, or an unexpected action of at least a subset of the objects of interest.

16. The method of claim 8, wherein the sensor data includes at least one of heat signature data, radio frequency data, audio data, motion data, ambient light image data, infrared image data, or stereoscopic image data.

17. A method of monitoring objects of interest in a designated area, comprising:

determining a regular flight path for an unmanned aerial vehicle (UAV) in order to enable a camera of the UAV to have at least a portion of the designated area in a field of view over a period of time for the monitoring;

capturing, using the camera, image data corresponding to the designated area;

performing a first analysis of the image data to recognize objects of interest located within the designated area;

based at least in part upon at least one of a location or a movement of at least a subset of the objects of interest, modifying the regular flight path to enable capturing of additional image data enabling increased accuracy of recognition of the one or more objects of interest within the area of interest; and

transmitting report data indicating at least one of a number, a location, a density, or a movement of the objects of interest within the designated area over the period of time.

18. The method of claim 17, further comprising:

transmitting the image data to a processing system, whereby the processing system is enabled to perform a second analysis of the image data using a more robust object detection algorithm to generate a more accurate recognition of objects of interest represented in the image data.

19. The method of claim 17, wherein the report data includes at least one of types of traffic, congestion points, patterns of motion, changes in traffic patterns over time, different paths of motion for different types of objects, unusual movements, and unexpected occurrences.

20. The method of claim 17, wherein the report data is updated over time for the designated area as part of a subscription service to which a customer has subscribed, the customer enabled to specify at least one of the designated area, the period of time, a frequency of reporting, and one or more types of the objects of interest.

21. The method of claim 17, wherein image data from a plurality of UAVs is aggregated before analysis, the image data representing different viewpoints and capable of being used to generate three-dimensional representations of at least a subset of the designated region and the objects of interest.

22. The method of claim 17, further comprising:

determining, using the image data, a state of one or more traffic signals in the designated area; and

determining different patterns of motion of different types of objects for each of the states and for each of the traffic signals in the designated areas, the different patterns of motion being included in the report data.