Computer vision-based surveillance for services

• US 10,388,092 B1
• Filed: 09/28/2016
• Issued: 08/20/2019
• Est. Priority Date: 09/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Solh, Mashhour, Ding, Siyi, Vijaya Kumar, Rohith Mysore, Wang, Xiaoran
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US15/278,937
• Time in Patent Office: 1,056 Days
• Field of Search: 382118
• CPC Class Codes: G06Q 10/083 Shipping
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G07C 9/00563 using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns
• CPC Class Codes: G07C 9/00896 specially adapted for particular uses
• CPC Class Codes: G07C 9/37 using biometric data, e.g. fingerprints, iris scans or voice recognition

Abstract

This application describes techniques for providing computer vision-based surveillance of in-home services. In some instances, a remote system may authenticate a guest at an environment using data that the remote system receives from a camera apparatus. To authenticate the guest, the remote system can determine that an identification of the guest matches an identification of a worker associated with a scheduled service at the environment. When the scheduled service includes an in-home delivery, the remote system can further determine that a package in possession of the guest matches an item for the scheduled in-home delivery. In some instances, after the guest enters the environment, the remote system can further monitor the environment to ensure that the guest stays within a designated location of the environment. The remote system can further send a message to an electronic device when the guest is outside of the designated location.

Claims

1. A system comprising:

at least one database to store;

scheduling data associated with delivery of a physical package, the scheduling data indicating at least a delivery item and a delivery location; and

profile data associated with a delivery person, the profile data including facial recognition data that can be used to identify the delivery person from image data representing the delivery person;

one or more network interfaces to communicate with a camera apparatus located at the delivery location;

one or more processors; and

one or more computer-readable media storing computer-instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

receiving, from a device associated with the profile data, a first message indicating that the device is at the delivery location;

sending, to the camera apparatus located at an environment associated with the delivery location, a second message requesting the image data captured by the camera apparatus;

receiving, from the camera apparatus, the image data;

determining, using the facial recognition data and a first portion of the image data, that the delivery person is at the delivery location;

analyzing the image data using object recognition;

determining, based at least in part on analyzing the image data, that a second portion of the image data represents an item;

determining, based at least in part on the second portion of the image data representing the item, that the delivery person is in possession of the delivery item;

sending, to a lock control apparatus, a third message that causes a locking apparatus associated with an entrance of the environment to provide access to the delivery person; and

sending, to the device, a fourth message indicating a zone within the environment for placing the delivery item.

2. The system as recited in claim 1, wherein the image data represents a particular person, and the acts further comprising:

determining the first portion of the image data, the first portion of the image data representing a face of the particular person; and

analyzing features of the first portion with respect to the facial recognition data using facial recognition,

and wherein determining that the delivery person is at the delivery location is based at least in part on analyzing the features of the first portion.

3. The system as recited in claim 1, the acts further comprising:

receiving, from an additional camera apparatus located within the environment, additional image data captured by the additional camera apparatus;

determining that the additional camera apparatus is located within an area of the environment corresponding to the zone;

determining that a portion of the additional image data represents the delivery person; and

determining that a position of the delivery person corresponds to the zone based, at least in part, on the portion of the additional image data representing the delivery person.

4. The system as recited in claim 1, the acts further comprising:

receiving, from an additional camera apparatus located within the environment, additional image data captured by the additional camera apparatus;

determining that the additional camera apparatus is located within an area of the environment that is outside of the zone;

determining that a portion of the additional image data represents the delivery person;

determining that a position of the delivery person is outside of the zone based, at least in part, on the portion of the additional image data representing the delivery person; and

sending a fifth message to an electronic device, the fifth message indicating that the delivery person is outside of the zone.

5. The system as recited in claim 1, wherein determining that the image data represents the delivery item comprises:

determining, based at least in part on analyzing the image data, that the second portion of the image data represents a package;

analyzing the second portion in order to determine a size of the package; and

determining that the package is associated with the delivery item based, at least in part, the size of the package corresponding to a predetermined size of the delivery item.

6. A method comprising:

storing, by one or more computing devices, data representing an object that is to be at an environment;

receiving, by the one or more computing devices, information indicating that a guest is at the environment;

sending, by the one or more computing devices, a first message to a camera apparatus, the first message including a request for image data representing a zone of the environment;

receiving, by the one or more computing devices and from the camera apparatus, the image data representing the zone of the environment;

analyzing, by the one or more computing devices, the image data with respect to the data representing the object;

determining, by the one or more computing device, and based at least in part on analyzing the image data, that at least a portion of the image data represents the object;

determining, by the one or more computing device, that the guest is in the environment based, at least in part, on determining that the at least the portion of the image data represents the object;

sending, by the one or more computing devices and to a device located at the environment, a second message that causes the device to grant access to the environment;

determining, by the one or more computing devices, that the guest is leaving the environment; and

sending, by the one or more computing devices, a third message that causes the device to secure the environment.

7. The method as recited in claim 6, wherein the camera apparatus comprises a first camera apparatus and the image data comprises first image data, and wherein the method further comprises:

receiving, from a second camera apparatus, second image data representing an additional area of the environment;

determining that the additional area of the environment corresponds to a designated zone for the guest based, at least in part, on a location of the second camera apparatus within the environment;

determining that a portion of the second image data represents the guest; and

determining that a current position of the guest within the environment corresponds to the designated zone based, at least in part, on the portion of the second image data representing the guest.

8. The method as recited in claim 6, wherein the camera apparatus comprises a first camera apparatus and the image data comprises first image data, and wherein the method further comprises:

receiving, from a second camera apparatus, second image data representing an additional area of the environment;

determining that the additional area of the environment is outside of a designation zone for the guest based, at least in part, on a location of the second camera apparatus within the environment;

determining that a portion of the second image data represents the guest;

determining that a current position of the guest within the environment is outside of the designated zone based, at least in part, on the portion of the second image data representing the guest; and

sending a fourth message to a user device, the fourth message indicating that the guest is outside of the designated zone.

9. The method as recited in claim 6, wherein the guest is to perform a delivery of a physical package, and wherein the method further comprises:

identifying an item being delivered within the environment for the delivery;

determining, based at least in part on analyzing the portion of the image data, a size of the object; and

determining that the object is associated with the item based, at least in part, on the size of the object corresponding to a predetermined size of the item,

and wherein determining that the guest is within the environment is further based, at least in part, on determining that the object is associated with the item.

10. The method as recited in claim 6, wherein the guest is to perform a delivery of a physical package, the camera apparatus comprises a first camera apparatus, and the image data comprises first image data, and wherein the method further comprises:

identifying a type of item being delivered within the environment for the delivery;

determining, based at least in part on the type of item, a designated zone within the environment for performing the delivery;

receiving, from a second camera apparatus, second image data representing an additional area within the environment;

determining that the additional area corresponds to the designated zone based, at least in part, on a location of the second camera apparatus within the environment;

determining that a portion of the second image data represents the guest; and

determining, based at least in part on the portion of the second image data representing the guest, that a current position of the guest is within the designated zone.

11. The method as recited in claim 6, wherein the guest is to perform a delivery of a physical package, the camera apparatus comprises a first camera apparatus, and the image data comprises first image data, and wherein the method further comprises:

identifying a type of item being delivered within the environment for the delivery;

determining, based at least in part on the type of item, a designated zone within the environment for performing the delivery;

receiving, from a second camera apparatus, second image data representing an additional area within the environment;

determining that the additional area is outside of the designated zone based, at least in part, on a location of the second camera apparatus within the environment;

determining that a portion of the second image data represents the guest;

determining, based at least in part on the portion of the second image data representing the guest, that a current position of the guest is outside of the designated zone; and

sending a fifth message to a user device, the fifth message indicating that the guest is outside of the designated zone.

12. The method as recited in claim 6, further comprising:

identifying an item that the guest should be in possession of while within the environment;

analyzing the image data using object recognition;

determining, based at least in part on analyzing the image data, that a portion of the image data represents the item; and

determining that the guest is in possession of the item based, at least in part, on the portion of the image data representing the item,

and wherein determining that the guest is within the environment is further based, at least in part, on determining that the guest is in possession of the item.

13. The method as recited in claim 6, wherein the camera apparatus comprises a first camera apparatus and the image data comprises first image data, and wherein the method further comprises:

receiving, from the first camera apparatus or a second camera apparatus, second image data representing an outer area of the environment;

determining that a portion of the second image data represents the guest; and

determining that a position for the guest is within the outer area of the environment,

and wherein determining that the guest is leaving the environment is based, at least in part, on the position.

20. The method as recited in claim 6, wherein:

the object is the guest;

the data representing the object comprises facial-recognition data representing the guest; and

the analyzing the image data comprises analyzing, by the one or more computing devices, the image data using facial recognition and the facial-recognition data.

14. An electronic device comprising:

one or more processors; and

one or more computer-readable media storing computer-instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

storing first data representing a guest;

storing second data indicating a designated zone for the guest within an environment;

sending a message to a device located at the environment, the message including an instruction to provide access to the environment;

based at least in part on sending the message, receiving, from a camera apparatus, image data representing an area within the environment;

determining that the area is associated with the designated zone for the guest based, at least in part, on a location of the camera apparatus within the environment and the second data;

analyzing the image data with respect to the first data representing the guest;

determining, based at least in part on analyzing the image data, that a portion of the image data represents the guest; and

determining, based at least in part on the portion of the image data representing the guest, that a position of the guest is within the designated zone.

15. The electronic device as recited in claim 14, the acts further comprising:

receiving, from another camera apparatus, additional image data representing an additional area within the environment;

determining that the additional area is outside of the designated zone for the guest based, at least in part, on a location of the additional camera apparatus within the environment;

determining that a portion of the additional image data represents the guest;

determining, based at least in part on the portion of the additional image data representing the guest, that a new position of the guest is outside of the designated zone; and

sending an additional message to an electronic device, the additional message indicating that the guest is outside of the designated zone.

16. The electronic device as recited in claim 14, the acts further comprising receiving an additional message designating:

the designated zone as a first area within the environment in which the guest is authorized; and

an additional designated zone within the environment as a second area within the environment in which the guest is not authorized.

17. The electronic device as recited in claim 14, the acts further comprising determining, based at least in part a type of service being performed by the guest within the environment, the designated zone for the guest.

18. The electronic device as recited in claim 14, wherein the guest is to perform a delivery of a physical package, and wherein the acts further comprise:

identifying an item being delivered at the environment for the delivery; and

determining, based at least in part on the item, the designated zone for the guest.

19. The electronic device as recited in claim 14, the acts further comprising:

receiving, from an additional camera apparatus, additional image data representing an additional area of the environment;

determining, based at least in part on the additional image data, that an object is in the environment; and

determining that the object is the guest.