Systems and methods for automated object recognition

• US 10,482,345 B2
• Filed: 06/22/2017
• Issued: 11/19/2019
• Est. Priority Date: 06/23/2016
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Jouhikainen, Hannes Mikael, Jacobs, Drew
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/630,913
• Publication Number:
  US 20170372165A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47815 Electronic shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

A method for recognizing an object in a video stream may include receiving a video stream from a video source, the video stream comprising a plurality of video frames. The method may also include selecting at least one video frame from the video frames according to a frame selection rate. The method may also include partitioning the selected video frame into a first plurality of image blocks. The method may also include recognizing, out of the first plurality of image blocks, a second plurality of image blocks which comprise an image of an object, the recognition being based on an image recognition parameter determined by a machine-learning algorithm. The method may also include determining that at least one of the second plurality of image blocks corresponds to the object based on a likelihood metric, the likelihood metric being determined by the processor based on at least the frame selection rate. The method may further include displaying, on a display, information identifying the object. A system and non-transitory computer-readable medium may also be provided.

Claims

1. A system for recognizing an object in a video stream, comprising:

a memory storing instructions; and

a processor configured to execute the stored instructions to;

receive the video stream from a video source, the video stream comprising a first set of video frames;

select at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames;

partition the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame;

identify, within one or more sets of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter;

calculate a likelihood metric that the region corresponds to the object;

adjust the frame selection rate when the likelihood metric is less than a predetermined threshold; and

display, on a display, information identifying the object.

2. The system of claim 1, wherein the processor is further configured to execute the stored instructions to apply the machine-learning algorithm to a second set of video frames.

3. The system of claim 1, wherein the processor is further configured to execute the stored instructions to adjust the image recognition parameter in response to an input received from a user device.

4. The system of claim 1, wherein determining that the region corresponds to the object comprises:

comparing the likelihood metric to a predetermined threshold; and

when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object.

5. The system of claim 1, wherein the processor is further configured to execute the stored instructions to adjust the frame selection rate in response to an input from a user device.

6. The system of claim 1, wherein the processor is further configured to execute the stored instructions to determine the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames.

7. The system of claim 1, wherein the processor is further configured to execute the stored instructions to determine the likelihood metric based additionally on the information identifying the object.

8. The system of claim 7, wherein the information identifying the object comprises information specifying at least one of a price of the object, availability of the object, or a location of the object.

9. The system of claim 1, wherein the processor is further configured to execute the stored instructions to receive the information identifying the object from at least one data server.

10. The system of claim 1, wherein:

the processor is further configured to execute the stored instructions to display the video stream; and

displaying the information identifying the object comprises displaying a digital shopping cart over the displayed video stream.

11. A computer-implemented method for recognizing an object in a video stream, comprising:

receiving the video stream from a video source, the video stream comprising a first set of video frames;

selecting at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames;

partitioning the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame;

identifying within one or more sets, of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter;

calculating a likelihood metric that the region corresponds to the object;

adjusting the frame selection rate when the likelihood metric is less than a predetermined threshold; and

displaying, on a display, information identifying the object.

12. The method of claim 11, further comprising:

applying the machine-learning algorithm to a second set of video frames.

13. The method of claim 11, wherein determining that the region corresponds to the object comprises:

comparing the likelihood metric to a predetermined threshold; and

when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object.

14. The method of claim 11, further comprising:

determining the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames.

15. The method of claim 11, further comprising:

determining the likelihood metric based additionally on the information identifying the object.

16. A non-transitory computer-readable medium storing instructions which, when executed, cause at least one processor to perform operations for recognizing an object in a video stream, the operations comprising:

receiving the video stream from a video source, the video stream comprising a first set of video frames;

selecting at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames;

partitioning the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame;

identifying within one or more sets, of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter;

calculating a likelihood metric that the region corresponds to the object;

adjusting the frame selection rate when the likelihood metric is less than a predetermined threshold; and

displaying, on a display, information identifying the object.

17. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

applying the machine-learning algorithm to a second set of video frames.

18. The non-transitory computer-readable medium of claim 16, wherein determining that the region corresponds to the object comprises:

comparing the likelihood metric to a predetermined threshold; and

when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object.

19. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

determining the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames.

20. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

determining the likelihood metric based additionally on the information identifying the object.