Multi-cue object detection and analysis

• US 10,037,604 B2
• Filed: 06/08/2016
• Issued: 07/31/2018
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Datta, Ankur, Feris, Rogerio S., Pankanti, Sharathchandra U., Wang, Xiaoyu
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/176,289
• Publication Number:
  US 20160284097A1
• Time in Patent Office: 783 Days
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

Foreground objects of interest are distinguished from a background model by dividing a region of interest of a video data image into a grid array of individual cells. Each of the cells are labeled as foreground if accumulated edge energy within the cell meets an edge energy threshold, or if color intensities for different colors within each cell differ by a color intensity differential threshold, or as a function of combinations of said determinations.

Claims

1. A computer-implemented method for distinguishing foreground objects of interest from a background model, the method comprising executing on a processing unit the steps of:

acquiring frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;

for each of the cells, determining a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;

for each of the cells, determining a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;

labelling each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and

using the frame image data from the cells labeled as foreground to define a foreground object.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable hardware storage device; and

wherein the computer readable program code is embodied on the computer readable hardware storage device and comprises instructions for execution by the processing unit via the computer readable memory that cause the processing unit to perform the steps of acquiring the frame image data for each of the plurality of individual cells, determining the first foreground values and the second foreground values for each of the cells, labelling each of the cells as foreground in response to the true value output of the combination rule, and using the frame image data from the cells labeled as foreground to define the foreground object.

3. The method of claim 1, wherein the step of determining for each of the cells the second foreground value comprises:

generating a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for the each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and

determining the second foreground value as the generated combination edge energy logical output value.

4. The method of claim 1, wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;

wherein the first and the second foreground values are differentially weighted; and

wherein the step of labelling each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.

5. The method of claim 1, further comprising:

applying a Sobel operator to each cell to create an edge patch; and

determining for each cell the second foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0,”

according to;

6. The method of claim 5, wherein the different colors of the determined color intensities are one of red, relative green and blue color channels.

7. The method of claim 6, further comprising:

determining for each cell the first foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;

8. The method of claim 6, further comprising:

determining for each cell a third foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;

9. A system, comprising:

a processing unit;

a computer readable memory in communication with the processing unit; and

a computer-readable hardware storage device in communication with the processing unit;

wherein the processing unit executes program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby;

acquires frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;

for each of the cells, determines a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;

for each of the cells, determines a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;

labels each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and

uses the frame image data from the cells labeled as foreground to define a foreground object.

10. The system of claim 9, wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:

generates a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and

determines the second foreground value as the generated combination edge energy logical output value.

11. The system of claim 9, wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;

wherein the first and the second foreground values are differentially weighted; and

wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby labels each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.

12. The system of claim 9, wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:

applies a Sobel operator to each of the cells to create an edge patch; and

determines for each cell the second foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0,”

according to;

13. The system of claim 12, wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:

determines for each cell the first foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;

14. The system of claim 13, wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:

determines for each cell a third foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;

15. An article of manufacture, comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, wherein the computer readable hardware storage device is not transitory per se, and computer readable program code comprises instructions for execution by a computer processing unit that cause the computer processing unit to;

acquire frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;

for each of the cells, determine a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;

for each of the cells, determine a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;

label each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and

use the frame image data from the cells labeled as foreground to define a foreground object.

16. The article of manufacture of claim 15, wherein the computer readable program code instructions cause the computer processing unit to:

generate a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and

determine the second foreground value as the generated combination edge energy logical output value.

17. The article of manufacture of claim 15, wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;

wherein the first and the second foreground values are differentially weighted; and

wherein the computer readable program code instructions cause the computer processing unit to label each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.

18. The article of manufacture of claim 15, wherein the computer readable program code instructions cause the computer processing unit to:

apply a Sobel operator to each of the cells to create an edge patch; and

determine for each cell the second foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0,”

according to;

19. The article of manufacture of claim 18, wherein the computer readable program code instructions cause the computer processing unit to:

determine for each cell the first foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;

20. The article of manufacture of claim 19, wherein the computer readable program code instructions cause the computer processing unit to:

determine for each cell a third foreground value as true in response to determining a value of “

1”

, and as false in response to determining a value of “

0”

, according to;