Image creation using geo-fence data

• US 10,313,638 B1
• Filed: 06/12/2015
• Issued: 06/04/2019
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yeturu, Kalidas, Huddleston, Jr., Howard Lee
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US14/738,502
• Time in Patent Office: 1,453 Days
• Field of Search: 348144
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

An unmanned aerial vehicle (UAV) may perform a surveillance action at a property of an authorized party. The property may be defined by a geo-fence, which may be a virtual perimeter or boundary around a real-world geographic area. The UAV may image the property to generate surveillance images, and the surveillance images may include image data of objects inside the geo-fence and image data of objects outside the geo-fence. While gathering surveillance images, or after the surveillance images have been gathered, the geo-fence information may be used to obscure or remove image data referring to objects outside the geo-fence. Geo-clipped surveillance images may be generated by physically constraining a sensor of the UAV, by performing pre-image capture processing, or post-image capture processing. Geo-clipped surveillance images may be limited to authorized property, so privacy is ensured for private persons and property.

Claims

1. A method comprising:

determining, based at least in part on a geo-fence corresponding with a boundary surrounding a surveillance location, geo-fence data corresponding to the surveillance location, the geo-fence data distinguishing between an authorized area and an unauthorized area;

receiving a surveillance image captured by a camera of an unmanned aerial vehicle (UAV), the surveillance image depicting the surveillance location and including first image data of the authorized area within the geo-fence and second image data of the unauthorized area outside the geo-fence;

receiving surveillance image metadata associated with the surveillance image, the surveillance image metadata including at least location data of the UAV and a direction that the camera faces during a capture of the surveillance image;

analyzing, by the UAV, the surveillance image using the geo-fence data and the surveillance image metadata to identify the first image data of the authorized area and the second image data of the unauthorized area;

generating a geo-clipped surveillance image by modifying the surveillance image to at least obscure or remove the second image data depicting the unauthorized area while maintaining the first image data of the authorized area;

detecting, based at least in part on the first image data, an event;

determining, based at least in part on a predefined criterion, that the event qualifies as a surveillance event;

determining a confidence value associated with the surveillance event based in part on the analyzing the surveillance image captured by the UAV;

generating a surveillance alert for the surveillance event based at least in part on the confidence value being above a threshold; and

adding a visual indication of the surveillance alert to the geo-clipped surveillance image.

2. The method of claim 1, wherein the adding the visual indication of the surveillance alert includes at least one of:

changing a color of a portion of the geo-clipped surveillance image to visually communicate the surveillance alert, or

adding a descriptive marker to the surveillance image as the visual indication.

3. The method of claim 1, wherein the geo-fence data further designates publicly accessible land that is permitted to be image-captured, and wherein the geo-clipped surveillance image further depicts at least a portion of the publicly accessible land.

4. The method of claim 1, wherein the surveillance image metadata further includes at least one of:

an altitude of the UAV or the camera;

a heading of the UAV;

or

LIDAR/RADAR data from the UAV.

5. The method of claim 1, wherein the determining the geo-fence data further includes comparing global positioning system (GPS) coordinates to property boundaries to create the geo-fence to define the authorized area that correspond with the property boundaries.

6. A system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory including one or more instructions that when executed by the one or more processors, cause the one or more processors to perform acts comprising;

determining, based at least in part on a geo-fence corresponding with a boundary surrounding a surveillance location, geo-fence data corresponding to the surveillance location, the geo-fence data distinguishing between an authorized area and an unauthorized area;

receiving a surveillance image captured by a camera of an unmanned aerial vehicle (UAV), the surveillance image depicting the surveillance location and including first image data of the authorized area within the geo-fence and second image data of the unauthorized area outside the geo-fence;

receiving surveillance image metadata associated with the surveillance image, the surveillance image metadata including at least location data of the UAV and a direction that the camera faces during a capture of the surveillance image;

analyzing, by the UAV, the surveillance image using the geo-fence data and the surveillance image metadata to identify the first image data of the authorized area and the second image data of the unauthorized area;

generating a geo-clipped surveillance image by modifying the surveillance image to at least obscure or remove the second image data depicting the unauthorized area while maintaining the first image data of the authorized area;

detecting, based at least in part on the first image data, an event;

determining, based at least in part on a predefined criterion, that the event qualifies as a surveillance event;

determining a confidence value associated with the surveillance event based in part on the analyzing the surveillance image captured by the UAV;

generating a surveillance alert for the surveillance event based at least in part on the confidence value being above a threshold; and

adding a visual indication of the surveillance alert to the geo-clipped surveillance image.

7. The system of claim 6, wherein the visual indication of the surveillance alert includes at least one of:

changing a color of a portion of the geo-clipped surveillance image to visually communicate the surveillance alert;

or

adding a descriptive marker to the surveillance image as the visual indication.

8. The system of claim 6, wherein the surveillance image metadata further includes at least one of:

an altitude of the UAV;

an altitude of the camera;

a heading of the UAV;

LIDAR data from the UAV;

or

RADAR data from the UAV.

9. The system of claim 6, wherein the adding the visual indication of the surveillance alert is based in part on the confidence value.

10. The system of claim 6, wherein the acts further comprise causing the camera of the UAV to capture the surveillance image.

11. The system of claim 6, wherein the determining the geo-fence data corresponding to the surveillance location includes identifying a public land that is authorized for image capture, and wherein the surveillance image includes imagery of at least part of the public land.

12. The system of claim 6, wherein the determining the geo-fence data further includes comparing global positioning system (GPS) coordinates to property boundaries to create the geo-fence to define the authorized area that correspond with the property boundaries.

13. The system of claim 6, wherein the surveillance alert is sent to a computing device associated with an owner of the surveillance location.

14. A non-transitory computer-readable medium storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:

determining, based at least in part on a geo-fence corresponding with a boundary surrounding a surveillance location, geo-fence data corresponding to the surveillance location, the geo-fence data distinguishing between an authorized area and an unauthorized area;

receiving a surveillance image captured by a camera of an unmanned aerial vehicle (UAV), the surveillance image depicting the surveillance location and including first image data of the authorized area within the geo-fence and second image data of the unauthorized area outside the geo-fence;

receiving surveillance image metadata associated with the surveillance image, the surveillance image metadata including at least location data of the UAV and a direction that the camera faces during a capture of the surveillance image;

analyzing, by the UAV, the surveillance image using the geo-fence data and the surveillance image metadata to identify the first image data of the authorized area and the second image data of the unauthorized area;

generating a geo-clipped surveillance image by modifying the surveillance image to at least obscure or remove the second image data depicting the unauthorized area while maintaining the first image data of the authorized area;

detecting, based at least in part on the first image data, an event;

determining, based at least in part on a predefined criterion, that the event qualifies as a surveillance event;

determining a confidence value associated with the surveillance event based in part on the analyzing the surveillance image captured by the UAV;

generating a surveillance alert for the surveillance event based at least in part on the confidence value being above a threshold; and

adding a visual indication of the surveillance alert to the geo-clipped surveillance image.

15. The non-transitory computer-readable medium of claim 14, wherein the visual indication of the surveillance alert includes at least one of:

changing a color of a portion of the geo-clipped surveillance image to visually communicate the surveillance alert;

or

adding a descriptive marker to the surveillance image as the visual indication.

16. The non-transitory computer-readable medium of claim 14, wherein determining the geo-fence data corresponding to the surveillance location includes identifying a public land that is authorized for image capture, and wherein the surveillance image includes imagery of at least part of the public land.

17. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise causing the camera of the UAV to capture the surveillance image.

18. The non-transitory computer-readable medium of claim 14, wherein the determining the geo-fence data corresponding to the surveillance location includes identifying a public land that is authorized for image capture, and wherein the surveillance image includes imagery of at least part of the public land.

19. The non-transitory computer-readable medium of claim 14, wherein the determining the geo-fence data further includes comparing global positioning system (GPS) coordinates to property boundaries to create the geo-fence to define the authorized area that correspond with the property boundaries.

20. The non-transitory computer-readable medium of claim 14, wherein the surveillance alert is sent to a computing device associated with an owner of the surveillance location.