Systems and methods for person recognition data management

• US 10,599,950 B2
• Filed: 08/14/2017
• Issued: 03/24/2020
• Est. Priority Date: 05/30/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bapat, Akshay R., Heitz, III, George Alban, Chaudhry, Rizwan Ahmed, Dalal, Navneet, Stewart, James Edward, Bush, Jennifer, Venters, Joe Delone, Gates, Kara, Butler, Timothy, Kifle, Yohannes, Bastov, Anton
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/676,868
• Publication Number:
  US 20180349736A1
• Time in Patent Office: 953 Days
• Field of Search: None
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/1966 Wireless systems, other than telephone systems, used to communicate with a camera
• CPC Class Codes: G08B 13/19663 Surveillance related processing done local to the camera
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The various implementations described herein include systems and methods for recognizing persons in video streams. In one aspect, a method includes: (1) obtaining images collected by video cameras in a smart home environment, each image including a detected person; (2) for each image, obtaining personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image; (3) grouping the images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique person; (4) receiving from a user a request to remove a first image from the first group; and (5) in response to the request: (a) removing the first image from the first group; and (b) disassociating the corresponding personally identifiable information from the first group.

Claims

1. A method, comprising:

obtaining a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person;

for each image of the plurality of images, obtaining personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image;

grouping the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons;

causing the first group of images to be presented to a user, the first group corresponding to a first detected person;

receiving from the user a request to remove a first image from the first group;

in response to the request;

removing the first image from the first group; and

generating updated personally identifiable information for the first group by disassociating the corresponding personally identifiable information from the first group;

after receiving the user request, detecting a subsequent event captured by a first video camera of the one or more video cameras by monitoring a video feed from the first video camera;

determining based on the updated personally identifiable information that the subsequent event involves the first detected person; and

generating an event notification to be presented to the user for the subsequent event, the notification including an image of the first detected person selected from the first group of images.

2. The method of claim 1,

wherein the user request to remove the first image corresponds to a user selection of the first image and a subsequent user action.

3. The method of claim 1, wherein detecting the subsequent event comprises:

obtaining a new image including the first detected person;

obtaining new personally identifiable information of the first detected person, the new personally identifiable information generated from analysis of the new image;

determining that the new image includes the first detected person based on a comparison of the new personally identifiable information with personally identifiable information associated with the first group; and

in accordance with the determination that the new image includes the first detected person;

including the new image in the first group; and

associating the new personally identifiable information with the first group.

4. The method of claim 1, further comprising, for each group of the plurality of groups, selecting an image of the group as a profile image for the group.

5. The method of claim 4, wherein the profile image for a respective group is selected based on a determined mood of a respective detected person in each image.

6. The method of claim 4, further comprising displaying a listing of the one or more groups, the listing including display of the profile image for each group.

7. The method of claim 4, further comprising displaying an event timeline corresponding to at least one of the one or more video cameras, including:

displaying on the timeline indication of one or more detected events; and

for at least one of the one or more detected events, displaying one or more profile images corresponding to one or more persons in the detected event.

8. A server system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

obtaining a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person;

for each image of the plurality of images, obtaining personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image;

grouping the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons;

causing the first group of images to be presented to a user, the first group corresponding to a first detected person;

receiving from the user a request to remove a first image from the first group;

in response to the request;

removing the first image from the first group; and

generating updated personally identifiable information for the first group by disassociating the corresponding personally identifiable information from the first group;

after receiving the user request, detecting a subsequent event captured by a first video camera of the one or more video cameras by monitoring a video feed from the first video camera;

determining based on the updated personally identifiable information that the subsequent event involves the first detected person; and

generating an event notification to be presented to the user for the subsequent event, the notification including an image of the first detected person selected from the first group of images.

9. The system of claim 8, wherein the plurality of images are grouped into the first group in response to a determination that the personally identifiable information corresponding to each image in the plurality of images is within a similarity threshold to other personally identifiable information for the first group.

10. The system of claim 8, wherein the one or more programs further comprise instructions for, for a first image of the plurality of images, adjusting the personally identifiable information based on a determined pose of the person in the image.

11. The system of claim 8, wherein the one or more programs further comprise instructions for, for a first image of the plurality of images, adjusting the personally identifiable information based on a camera operating mode.

12. The system of claim 8, wherein:

the request to remove the first image comprises a request to assign the first image to a different group of the plurality of groups;

removing the first image from the first group comprises assigning the first image to the different group; and

disassociating the personally identifiable information of the first image comprises associating the personally identifiable information with the different group.

13. The system of claim 8, wherein the one or more programs further comprise instructions for:

receiving from the user a request to remove a group of the plurality of groups; and

in response to the request;

deleting all images in the group; and

deleting corresponding personally identifiable information.

14. The system of claim 8, wherein the personally identifiable information comprises facial feature information of the detected person.

15. The system of claim 8, wherein the personally identifiable information comprises a feature signature for the detected person, the feature signature generated based on analysis of the image.

16. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

obtain a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person;

for each image of the plurality of images, obtain personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image;

group the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons;

cause the first group of images to be presented to a user, the first group corresponding to a first detected person;

receive from the user a request to remove a first image from the first group;

in response to the request;

remove the first image from the first group; and

generate updated personally identifiable information for the first group by disassociating the corresponding personally identifiable information from the first group

after receiving the request, detect a subsequent event captured by a first video camera of the one or more video cameras by monitoring a video feed from the first video camera;

determine based on the updated personally identifiable information that the subsequent event involves the first detected person; and

generate an event notification to be presented to the user for the subsequent event, the notification including an image of the first detected person selected from the first group of images.

17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to, subsequent to generating the event notification:

receive, from the user, identification information of the first detected person for the first group; and

remove the first group from the plurality of groups based on the received identification information.

18. The non-transitory computer-readable storage medium of claim 17, wherein the identification information indicates that the detected person is a stranger to the user;

and

wherein removing the first group comprises deleting (i) all images in the first group, and (ii) the corresponding personally identifiable information.

19. The non-transitory computer-readable storage medium of claim 17, wherein the identification information indicates an identity of the detected person;

and

wherein removing the first group comprises combining the first group with another group representing the same identity.

20. The non-transitory computer-readable storage medium of claim 16, wherein:

the request to remove the first image comprises a request to delete the first image;

removing the first image from the first group comprises deleting the first image; and

disassociating the personally identifiable information of the first image comprises deleting the personally identifiable information.