Targeted search of backup data using facial recognition

• US 10,642,886 B2
• Filed: 02/14/2018
• Issued: 05/05/2020
• Est. Priority Date: 02/14/2018
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Bedadala, Pavan Kumar Reddy
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US15/896,816
• Publication Number:
  US 20190251191A1
• Time in Patent Office: 811 Days
• Field of Search: 707769
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

This application relates to targeted search of backup data. A data storage system can provide a targeted search of backup data using facial recognition. Upon receiving a search query that identifies a specific user whose profile is stored in a user database, the data storage system can retrieve a profile photograph of the specific user, perform facial recognition on photographs included in the backup data, and output a search result including a set of photographs in the backup data that match the facial features of the specific user.

Claims

1. A computer-implemented method for performing a targeted search on backup data, the method comprising:

receiving a request to create a backup copy of primary data stored on one or more primary storage devices associated with a client computing device in a primary storage subsystem onto one or more secondary storage devices in a secondary storage subsystem, wherein the client computing device is associated with a requesting user associated with one of a plurality of profiles stored in a user database maintained in the secondary storage subsystem;

identifying a plurality of photographs included in the primary data, the plurality of photographs each associated with a respective location and a respective time;

creating the backup copy of the primary data on the one or more secondary storage devices in the secondary storage subsystem such that secondary copies of the plurality of photographs included in the backup copy are tagged as having a photographic data type and associated with the respective location and the respective time;

subsequent to creating the backup copy in the secondary storage subsystem, receiving, from the client computing device, a request to search the backup copy, the request including a search query;

identifying, based on the search query, a target user'"'"'s profile in the plurality of profiles stored in the user database;

obtaining a profile photograph associated with the target user'"'"'s profile from the user database;

detecting an image region in the profile photograph that corresponds to the target user'"'"'s face;

extracting one or more facial features from the detected image region in the profile photograph;

identifying a set of photographs that match the one or more facial features from the secondary copies of the plurality of photographs included in the backup copy; and

transmitting a search result including the identified set of photographs to the client computing device associated with the requesting user.

2. The computer-implemented method of claim 1, further comprising determining, for each respective photograph of the plurality of photographs included in the primary data, the respective location and the respective time, wherein the respective location comprises global positioning system (GPS) coordinates of a location at which the respective photograph was taken, and the respective time comprises a time at which the respective photograph was taken.

3. The computer-implemented method of claim 1, further comprising:

extracting a plurality of blocks of content from the search query; and

identifying (i) a first block of the plurality of blocks that indicates the target user'"'"'s name associated with the target user'"'"'s profile and (ii) a second block of the plurality of blocks that indicates a photographic data type associated with at least some of the secondary data included in the backup copy stored in the one or more secondary storage devices.

4. The computer-implemented method of claim 3, further comprising:

identifying a third block of the plurality of blocks that indicates one of a plurality of events stored in an event database maintained in the secondary storage subsystem;

accessing the event database to identify an event time associated with the identified event and an event location associated with the identified event; and

filtering the identified set of photographs based on the event time and the event location such that the transmitted search result does not include any photograph that is not associated with the event time and the event location.

5. The computer-implemented method of claim 3, further comprising:

identifying a third block of the plurality of blocks that indicates one of a plurality of events stored in an event database maintained in the secondary storage subsystem;

accessing the event database to identify an event time associated with the identified event and an event location associated with the identified event; and

identifying the set of photographs that match the one or more facial features from a subset of the secondary copies of the plurality of photographs that are associated with the event time and the event location such that a determination of whether a photograph matches the one or more facial features is not made with respect to a photograph that is not associated with the event time and the event location.

6. The computer-implemented method of claim 1, wherein the primary data further includes a plurality of videos, the method further comprising:

identifying a set of videos, from secondary copies of the plurality of videos included in the backup copy, that are each associated with a respective thumbnail image that matches the one or more facial features; and

including the set of videos in the search result transmitted to the client computing device.

7. The computer-implemented method of claim 1, wherein the primary data further includes a plurality of videos, the method further comprising:

identifying a set of videos, from secondary copies of the plurality of videos included in the backup copy, that each include at least one frame that matches the one or more facial features; and

including the set of videos in the search result transmitted to the client computing device.

8. The computer-implemented method of claim 1, further comprising:

receiving the request via a graphical user interface presented on the client computing device; and

causing the search result to be displayed on the graphical user interface presented on the client computing device.

9. The computer-implemented method of claim 1, further comprising:

storing the secondary copies of the plurality of photographs in association with the requesting user; and

identifying additional photographs that match the one or more facial features and are stored on the one or more secondary storage devices in association with another user different than the requesting user.

10. The computer-implemented method of claim 1, further comprising:

storing additional photographs in response to another request to create a backup copy of other primary data associated with another client computing device; and

identifying another set of photographs that match the one or more facial features from the additional photographs stored in response to said another request such that the search result transmitted to the client computing device further includes said another set of photographs.

11. A system for performing a targeted search on backup data, the system comprising:

a client computing device comprising computer hardware and associated with one or more primary storage devices storing primary data, wherein the client computing device is in a primary storage subsystem and associated with a requesting user;

one or more secondary storage devices comprising computer hardware and residing in a secondary storage subsystem, wherein the one or more secondary storage devices are configured to store secondary data associated with the client computing device; and

one or more secondary storage computing devices comprising computer hardware and residing in a secondary storage subsystem, wherein the one or more secondary storage computing devices are configured to;

receive a request to create a backup copy of the primary data stored on the one or more primary storage devices onto the one or more secondary storage devices;

identify a plurality of photographs included in the primary data, the plurality of photographs each associated with a respective location and a respective time;

create the backup copy of the primary data on the one or more secondary storage devices in the secondary storage subsystem such that secondary copies of the plurality of photographs included in the backup copy are tagged as having a photographic data type and associated with the respective location and the respective time;

subsequent to the creation of the backup copy in the secondary storage subsystem, receive, from the client computing device, a request to search the backup copy, the request including a search query;

identify, based on the search query, a target user'"'"'s profile in the plurality of profiles stored in the user database;

obtain a profile photograph associated with the target user'"'"'s profile from the user database;

detect an image region in the profile photograph that corresponds to the target user'"'"'s face;

extract one or more facial features from the detected image region in the profile photograph;

identify a set of photographs that match the one or more facial features from the secondary copies of the plurality of photographs included in the backup copy; and

transmit a search result including the identified set of photographs to the client computing device associated with the requesting user.

12. The system of claim 11, wherein the one or more secondary storage computing devices are further configured to determine, for each respective photograph of the plurality of photographs included in the primary data, the respective location and the respective time, wherein the respective location comprises global positioning system (GPS) coordinates of a location at which the respective photograph was taken, and the respective time comprises a time at which the respective photograph was taken.

13. The system of claim 11, wherein the one or more secondary storage computing devices are further configured to:

extract a plurality of blocks of content from the search query; and

identify (i) a first block of the plurality of blocks that indicates the target user'"'"'s name associated with the target user'"'"'s profile and (ii) a second block of the plurality of blocks that indicates a photographic data type associated with at least some of the secondary data included in the backup copy stored in the one or more secondary storage devices.

14. The system of claim 13, wherein the one or more secondary storage computing devices are further configured to:

identify a third block of the plurality of blocks that indicates one of a plurality of events stored in an event database maintained in the secondary storage subsystem;

access the event database to identify an event time associated with the identified event and an event location associated with the identified event; and

filter the identified set of photographs based on the event time and the event location such that the transmitted search result does not include any photograph that is not associated with the event time and the event location.

15. The system of claim 13, wherein the one or more secondary storage computing devices are further configured to:

identify a third block of the plurality of blocks that indicates one of a plurality of events stored in an event database maintained in the secondary storage subsystem;

access the event database to identify an event time associated with the identified event and an event location associated with the identified event; and

identify the set of photographs that match the one or more facial features from a subset of the secondary copies of the plurality of photographs that are associated with the event time and the event location such that a determination of whether a photograph matches the one or more facial features is not made with respect to a photograph that is not associated with the event time and the event location.

16. The system of claim 11, wherein the primary data further includes a plurality of videos, the one or more secondary storage computing devices further configured to:

identify a set of videos, from secondary copies of the plurality of videos included in the backup copy, that are each associated with a respective thumbnail image that matches the one or more facial features; and

include the set of videos in the search result transmitted to the client computing device.

17. The system of claim 11, wherein the primary data further includes a plurality of videos, the one or more secondary storage computing devices further configured to:

identify a set of videos, from secondary copies of the plurality of videos included in the backup copy, that each include at least one frame that matches the one or more facial features; and

include the set of videos in the search result transmitted to the client computing device.

18. The system of claim 11, wherein the one or more secondary storage computing devices are further configured to:

receive the request via a graphical user interface presented on the client computing device; and

cause the search result to be displayed on the graphical user interface presented on the client computing device.

19. The system of claim 11, wherein the one or more secondary storage computing devices are further configured to:

store the secondary copies of the plurality of photographs in association with the requesting user; and

identify additional photographs that match the one or more facial features and are stored on the one or more secondary storage devices in association with another user different than the requesting user.

20. The system of claim 11, wherein the one or more secondary storage computing devices are further configured to:

store additional photographs in response to another request to create a backup copy of other primary data associated with another client computing device; and

identify another set of photographs that match the one or more facial features from the additional photographs stored in response to said another request such that the search result transmitted to the client computing device further includes said another set of photographs.