Apparatus and methods for facial recognition and video analytics to identify individuals in contextual video streams

• US 10,094,655 B2
• Filed: 07/15/2016
• Issued: 10/09/2018
• Est. Priority Date: 07/15/2015
• Status: Active Grant

• Current Assignee: 15 Seconds Of Fame Inc. (Forwardly, Inc.)
• Original Assignee: 15 Seconds Of Fame Inc. (Forwardly, Inc.)
• Inventors: Sabitov, Ruslan, Betser, Alexander, Resnick, Adam, Joshpe, Brett Terry, Huestis, E. Michael
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US15/211,261
• Publication Number:
  US 20170017834A1
• Time in Patent Office: 816 Days
• Field of Search: 382118, 382209, 382218
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01J 3/26 using multiple reflection, e.g. Fabry-Perot interferometer, variable interference filters
• CPC Class Codes: G01J 3/2823 Imaging spectrometer
• CPC Class Codes: G01J 3/4535 Devices with moving mirror G01J3/4532 takes precedence
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

An apparatus includes a memory, a communication interface in communication with the memory and configured to communicate via a network, and a processor in communication with the memory and the communication interface. The processor receives facial image data associated with a user of a client device, registers the facial image data, and stores the facial image data and contextual data associated with the user in a database. The processor also receives video stream data from at least one image capture device in communication with the network, analyzes the video stream data and contextual data associated with the video stream data to define analyzed video data and analyzed contextual data, respectively, and defines a confidence level based on comparing the data associated with the video stream data to the data stored in the database. The processor defines a user-specific contextual video stream when the confidence level satisfies a criterion.

Claims

1. An apparatus, comprising:

a memory;

a communication interface in communication with the memory and configured to communicate via a network; and

a processor in communication with the memory and the communication interface, the processor configured to receive, via the network and the communication interface, facial image data associated with a user of a client device,

in response to receiving the facial image data, the processor configured to (1) register the facial image data associated with the user based on at least one of two-dimensional facial recognition analytics, three-dimensional facial recognition analytics, or convolutional neural nets (CNN) and (2) store the facial image data and contextual data associated with at least one of the user or the client device associated with the user in a database operably coupled to the processor,

the processor configured to receive, via the network and the communication interface, video stream data from at least one image capture device in communication with the network, the video stream data including an indication of at least one facial image and data indicative of a coinciding event occurring at a predetermined venue,

in response to receiving the video stream data, the processor configured to (1) analyze the video stream data based on at least one of the two-dimensional facial recognition analytics, the three-dimensional facial recognition analytics, or the CNN to define an analyzed video stream data set, (2) analyze contextual data associated with the video stream data to define an analyzed contextual data set associated with the video stream data, and (3) define a confidence level based on comparing the analyzed video stream data set to the facial image data stored in the database and comparing the analyzed contextual data set associated with the video stream data to the contextual data associated with at least one of the user or the client device associated with the user and stored in the database,

the processor configured to define user-specific data (1) representing a user-related video stream and (2) including an indication of the coinciding event, when the confidence level satisfies a criterion, the processor further configured to send the user-specific data such that the client device associated with the user can access the user-specific data.

2. The apparatus of claim 1, wherein the processor is configured to send to the client device, via the network and the communication interface, a signal indicative of an instruction to graphically represent the user-specific data on a display of the client device.

3. The apparatus of claim 1, wherein the processor is configured to compare the analyzed contextual data set associated with the video stream data to the contextual data associated with at least one of the user or the client device associated with the user prior to comparing the analyzed video stream data set to the facial image data.

4. The apparatus of claim 1, wherein the confidence level is indicative of (1) a level of matching between the analyzed video stream data set and the facial image data and (2) a level of matching between the analyzed contextual data set and the contextual data associated with at least one of the user or the client device associated with the user.

5. The apparatus of claim 1, wherein the confidence level is indicative of a level of matching between at least the analyzed video stream data set and the facial image data, the criterion is a matching threshold of at least a 70% match between the analyzed video stream data set and the facial image data.

6. The apparatus of claim 1, wherein the user is a user from a plurality of users, the database is configured to store at least facial image data associated with each user from the plurality of users and contextual data associated with at least one of each user from the plurality of users or a client device associated with each user from the plurality of users.

7. The apparatus of claim 1, wherein the user is a user from a plurality of users, the database is configured to store at least facial image data associated with each user from the plurality of users and contextual data associated with at least one of each user from the plurality of users or a client device associated with each user from the plurality of users, and

the processor is configured to compare (1) the analyzed video stream data set to the facial image data of each user from the plurality of users and (2) the analyzed contextual data set to the contextual data associated with at least one of each user from the plurality of users or the client device associated with each user from the plurality of users.

8. The apparatus of claim 1, wherein the user is a first user from a plurality of users, the user-specific data is a first user-specific data set, and the user-related video stream is a first user-related video stream, the database is configured to store at least facial image data associated with each user from the plurality of users and contextual data associated with at least one of each user from the plurality of users or a client device associated with each user from the plurality of users,

the processor is configured to define a second user-specific data set (1) representing a second user-related video stream associated with a second user from the plurality of users and (2) including an indication of the coinciding event when a confidence level associated with the second user from the plurality of users satisfies the criterion,

the processor is configured to send via the network and to a client device associated with the second user from the plurality of users a signal indicative of an instruction to graphically represent the second user-specific data set on a display of the client device associated with the second user from the plurality of users.

9. A method, comprising:

receiving, at a host device and from a client device via a network, a signal indicative of a request to register facial image data associated with a user of the client device;

registering the facial image data via facial recognition and image analysis;

storing the facial image data and data associated with the registering in a database;

receiving, at the host device, contextual video stream data including an indication of at least one facial image and data indicative of a coinciding event;

analyzing the contextual video stream data via video analytics to determine if the contextual video stream data satisfies a criterion associated with identifying, via facial recognition, the facial image data in the contextual video stream data;

defining user-specific data (1) representing a user-related video stream and (2) including an indication of the coinciding event, when the criterion is satisfied; and

sending, from the host device to the client device, a signal indicative of an instruction to present the user-specific data on a display of the client device.

10. The method of claim 9, wherein the client device is one of a tablet, a smartphone, or a wearable electronic device.

11. The method of claim 9, wherein the facial image data associated with the user is image data captured via a camera operably coupled to the client device.

12. The method of claim 9, wherein the criterion is a confidence level associated with identifying the facial image data in the contextual video stream data.

13. The method of claim 9, further comprising:

sending, from the host device to the client device and prior to the analyzing of the contextual video stream data, a request for contextual data associated with at least one of the user or the client device associated with the user.

14. The method of claim 9, further comprising:

sending, from the host device to the client device and prior to the analyzing the contextual video stream data, a request for contextual data associated with at least one of the user or the client device associated with the user, the contextual data associated with at least one of the user or the client device associated with the user is at least one of calendar data, location data, preference data, or purchase data.

15. A system, comprising:

an image capture assembly in communication with a network, the image capture assembly including at least one image capture device configured to capture video stream data, the video stream data being associated with a predetermined venue; and

a host device having at least a processor and a memory, the host device being in communication with the network and a database, the host device configured to (1) receive facial image data of a user from a client device via the network, (2) register the facial image data via facial recognition and image analysis, and (3) store the facial image data in the database,

the host device configured to receive the video stream data from the image capture assembly and contextual data associated with the predetermined venue and to define a confidence level based on comparing the video stream data to the facial image data and comparing the contextual data associated with the predetermined venue to contextual data associated with at least one of the user or the client device associated with the user, the host device configured to define user-specific data (1) representing at least one user-related image and (2) including an indication of a coinciding event occurring at the predetermined venue, the host device configured to send the user-specific data when the confidence level satisfies a criterion such that the client device associated with the user can access the user-specific data.

16. The system of claim 15, wherein the host device is configured to receive the facial image data from the client device in response to a camera of the client device capturing the facial image data of the user.

17. The system of claim 15, wherein the facial image data received by the host device is facial image data automatically captured by a camera of the client device when a graphical representation of the face of the user is in a predetermined alignment on a display of the client device.

18. The system of claim 15, wherein the facial image data received by the host device is facial image data automatically captured by a camera of the client device when the user aligns a graphical representation of the face of the user on a display of the client device such that a distance between the eyes of the user is between about 40 pixels and about 65 pixels.

19. The system of claim 15, wherein the facial image data associated with the user of the client device includes a plurality of images of the face of the user, each image from the plurality of images capturing the face of the user at a different orientation relative to the camera of the client device.

20. The system of claim 15, wherein the at least one image capture device is at least one camera configured to capture image data associated with at least one of a picture or a video recording.

21. The system of claim 15, wherein the image capture assembly includes a camera of the client device.

22. The system of claim 15, wherein the image capture device is a camera of the client device, the host device configured to receive the video stream data and the contextual data associated with the predetermined venue from the client device.