MULTI-CUE OBJECT DETECTION AND ANALYSIS

• US 20130336581A1
• Filed: 06/14/2012
• Published: 12/19/2013
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Datta, Ankur, Feris, Rogerio S., Pankanti, Sharathchandra U., Wang, Xiaoyu
• Granted Patent: US 8,917,934 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

Foreground objects of interest are distinguished from a background model by dividing a region of interest of a video data image into a grid array of individual cells that are each smaller than that a foreground object of interest. More particularly, image data of the foreground object of interest spans a contiguous plurality of the cells. Each of the cells are labeled as foreground if accumulated edge energy within the cell meets an edge energy threshold, if color intensities for different colors within each cell differ by a color intensity differential threshold, or as a function of combinations of said determinations in view of one or more combination rules.

Claims

1. A method for distinguishing foreground objects of interest from a background model, the method comprising:

dividing by a programmable device a region of interest of a video data image into a grid array of a plurality of individual cells that are each initialized with a background label, wherein each of the cells has a two-dimensional area dimension that is smaller than a two-dimensional area size of a foreground object of interest so that image data of the foreground object in an image data frame spans a contiguous plurality of the cells;

acquiring by the programmable device frame image data for each of the cells;

detecting and accumulating by the programmable device energy of edges within each of the cells, and relabeling as foreground each of the cells that have an accumulated edge energy that meets an edge energy threshold and are currently labeled as background;

determining by the programmable device color intensities for each of a plurality of different colors within each cell, and relabeling as foreground each of the cells that have one of the determined color intensities greater than another of the determined color intensities for that cell by a color intensity differential threshold, and are currently labeled as background; and

using by the programmable device the frame image data from the cells relabeled as foreground to define a foreground object.

2. The method of claim 1, further comprising:

for each of the cells that are relabeled as foreground by the programmable device in response to having accumulated edge energy that meets the edge energy threshold, the programmable device;

summing the different color intensities determined for the cell to generate a color intensity sum of said relabeled cell;

comparing the generated color intensity sum of said relabeled cell to sums of the color intensities determined for each of a plurality of cells that are adjacent to said relabeled cell; and

revising the label of said relabeled cell to the background label if the generated color intensity sum of said relabeled cell is greater than the compared sums of the color intensities determined for each of plurality of cells that are adjacent to said relabeled cell by a color consistency threshold, wherein the revising the label is performed prior to the step of the programmable device using the frame image data from the cells relabeled as foreground to define the foreground object.

3. The method of claim 2, further comprising the programmable device:

comparing the size of blocks formed by contiguous groups of the relabeled foreground cells to a template filter; and

revising the labels of each of the relabeled foreground cells of the contiguous group of blocks that do not fit the compared template filter, wherein the revising the labels is performed prior to the step of the programmable device using the frame image data from the cells relabeled as foreground to define the foreground object.

4. The method of claim 3, wherein the template has a height and a width, and the programmable device revises the label of each of the relabeled foreground cells of the ones of the contiguous group of blocks if a height of the contiguous group of blocks is less than the template height, or if a width of the contiguous group of blocks is less than the template width.

5. The method of claim 3, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable tangible storage medium, wherein the computer readable program code is embodied on the computer readable tangible storage medium and comprises instructions that, when executed by the processing unit via the computer readable memory, cause the processing unit to perform the steps of;

dividing the region of interest into the grid array;

acquiring the frame image data for each of the cells;

detecting and accumulating the energy of edges within each of the cells, and relabeling as foreground each of the cells that have the accumulated edge energy that meets the edge energy threshold and are currently labeled as background;

determining the device color intensities for each of the plurality of different colors within each cell, and relabeling as foreground each of the cells that have the one color intensities determined greater than another of the determined color intensities for that cell by the color intensity differential threshold, and are currently labeled as background; and

using the frame image data from the cells relabeled as foreground to define the foreground object.

6. The method of claim 3, wherein the step of the programmable device detecting and accumulating the energy of edges within each of the cells, and relabeling as foreground each of the cells that have the accumulated edge energy that meets the edge energy threshold and are currently labeled as background, further comprises:

applying a Sobel operator to each cell to create an edge patch; and

determining the label of the each cell as a binary label according to;

7. The method of claim 3, wherein the different colors of the determined color intensities are one of red, relative green and blue color channels.

8. The method of claim 7, wherein the programmable device determines the color intensities for each of the different colors within each cell, and re-labels as foreground each of the cells that have one of the determined color intensities greater than another of the determined color intensities for that cell by the color intensity differential threshold, as a function of:

9. The method of claim 8, wherein the steps of, for each of the cells that are relabeled as foreground by the programmable device in response to having accumulated edge energy that meets the edge energy threshold, the programmable device, summing the different color intensities determined for the cell to generate the color intensity sums, comparing the generated color intensity sums to the color intensity sums determined for the adjacent cells, and revising the labels of the relabeled cells to the background label if their generated color intensity sum is greater than the compared sums of color intensities determined for the adjacent cells, comprises:

computing context adjacent cell color intensity consistency information according to;

10. A method for distinguishing foreground objects of interest from a background model, the method comprising:

dividing by a programmable device a region of interest of a video data image into a grid array of a plurality of individual cells, wherein each of the cells has a two-dimensional area dimension that is smaller than a two-dimensional area size of a foreground object of interest so that image data of the foreground object in an image data frame spans a contiguous plurality of the cells;

acquiring by the programmable device frame image data for each of the cells;

detecting and accumulating by the programmable device energy of edges within each of the cells, generating an edge energy foreground indication output for each of the cells that indicates foreground if an accumulated edge energy meets an edge energy threshold, or indicates background if the accumulated edge energy does not meet the edge energy threshold;

determining by the programmable device color intensities for each of a plurality of different colors within each of the cells, and generating a color intensity foreground indication output for each of the cells that indicates foreground if one of the determined color intensities is greater than another of the determined color intensities for that cell by a color intensity differential threshold, or indicates background if no one of the determined color intensities is greater than any other one of the determined color intensities for that cell by the color intensity differential threshold;

labeling each of the cells as foreground or background in response to the color intensity foreground indication output and the color intensity foreground indication output for the each cell as a function of a foreground indication output combination rule; and

using by the programmable device the frame image data from the cells labeled as foreground cells to define a foreground object.

11. The method of claim 10, wherein the foreground indication output combination rule requires both of the color intensity foreground indication output and the edge energy foreground indication output to indicate foreground to label the cell as foreground, and wherein the step of labeling by the programmable device each of the cells as foreground or background comprises:

applying a logical AND operation to inputs of the edge energy foreground indication output and the color intensity foreground indication output that generates a true output for each one of the cells if both of the inputs indicate that the cell is foreground; and

labeling each of the cells having generated true outputs as the foreground cells.

12. The method of claim 11, further comprising:

for each of the cells that are indicated as foreground in response to having accumulated edge energy that meets the edge energy threshold, the programmable device;

summing the different determined color intensities to generate a color intensity sum;

comparing the generated color intensity sum to sums of the color intensities determined for each of a plurality of cells that are adjacent to said cell having the generated color intensity sum; and

revising the foreground indication output for said cell having the generated color intensity sum to indicate background if said generated color intensity sum is greater than the compared sums of the color intensities determined for each of the plurality of adjacent cells by a color consistency threshold, wherein the revising the foreground indication output is performed prior to the step of labeling said cells as foreground or background as the function of the foreground indication output combination rule.

13. The method of claim 12, further comprising the programmable device:

comparing the size of blocks formed by contiguous groups of the labeled foreground cells to a template filter; and

revising the labels of each of the cells in the contiguous group of blocks that do not fit the compared template filter to background labels, wherein the revising the labels is performed prior to the step of the programmable device using the frame image data from the cells labeled as foreground to define the foreground object.

14. The method of claim 13, wherein the template has a height and a width, and the programmable device revises the label of each of the labeled foreground cells of the contiguous group of blocks if a height of the contiguous group of blocks is less than the template height, or if a width of the contiguous group of blocks is less than the template width.

15. The method of claim 10, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable tangible storage medium, wherein the computer readable program code is embodied on the computer readable tangible storage medium and comprises instructions that, when executed by the processing unit via the computer readable memory, cause the processing unit to perform the steps of;

dividing the region of interest of a video data image into the grid array of individual cells;

acquiring the frame image data for each of the cells;

detecting and accumulating the energy of edges within each of the cells, and generating the edge energy foreground indication output for each of the cells that indicates foreground if an accumulated edge energy meets an edge energy threshold, or indicates background if the accumulated edge energy does not meet the edge energy threshold;

determining the color intensities for each of the different colors within each of the cells, and generating the color intensity foreground indication output for each of the cells that indicates foreground if the determined one color intensity is greater than another of the determined color intensities for that cell by the color intensity differential threshold, or indicates background if no one of the determined color intensities is greater than any other one of the determined color intensities for that cell by the color intensity differential threshold;

labeling each of the cells as foreground or background in response to the color intensity foreground indication output and the color intensity foreground indication output for the each cell as a function of the foreground indication output combination rule; and

using the frame image data from the cells labeled as foreground cells to define a foreground object.

16. A system, comprising:

a processing unit in communication with a computer readable memory and a tangible computer-readable storage medium;

wherein the processing unit, when executing program instructions stored on the tangible computer-readable storage medium via the computer readable memory;

divides a region of interest of a video data image into a grid array of a plurality of individual cells that are each initialized with a background label, wherein each of the cells has a two-dimensional area dimension that is smaller than a two-dimensional area size of a foreground object of interest so that image data of the foreground object in an image data frame spans a contiguous plurality of the cells;

acquires device frame image data for each of the cells;

detects and accumulates energy of edges within each of the cells, and re-labels as foreground each of the cells that have an accumulated edge energy that meets an edge energy threshold and are currently labeled as background;

determines color intensities for each of a plurality of different colors within each cell, and re-labels as foreground each of the cells that have one of the determined color intensities greater than another of the determined color intensities for that cell by a color intensity differential threshold, and are currently labeled as background; and

uses the frame image data from the cells relabeled as foreground to define a foreground object.

17. The system of claim 16, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, for each of the cells that are relabeled as foreground in response to having accumulated edge energy that meets the edge energy threshold, further:

sums the different color intensities determined for the cell to generate a color intensity sum of said relabeled cell;

compares the generated color intensity sum of said relabeled cell to sums of the color intensities determined for each of a plurality of cells that are adjacent to said relabeled cell; and

revises the label of said relabeled cell to the background label if the generated color intensity sum of said relabeled cell is greater than the compared sums of the color intensities determined for each of a plurality of cells that are adjacent to said relabeled cell by a color consistency threshold, wherein the label is revised prior to using the frame image data from the cells relabeled as foreground to define the foreground object.

18. The system of claim 17, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further:

compares the size of blocks formed by contiguous groups of the relabeled foreground cells to a template filter; and

revises the labels of each of the relabeled foreground cells of the contiguous group of blocks that do not fit the compared template filter, wherein the labels are revised prior to using the frame image data from the cells relabeled as foreground to define the foreground object.

19. The system of claim 18, wherein the template has a height and a width, and wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further revises the label of each of the relabeled foreground cells of the ones of the contiguous group of blocks if a height of the contiguous group of blocks is less than the template height, or if a width of the contiguous group of blocks is less than the template width.

20. The system of claim 19, wherein the template height is at least the height of two of the cells, and the template width is at least the width of two of the cells.

21. An article of manufacture, comprising:

a computer readable tangible storage medium having computer readable program code embodied therewith, the computer readable program code comprising instructions that, when executed by a computer processing unit, cause the computer processing unit to;

divide a region of interest of a video data image into a grid array of a plurality of individual cells, wherein each of the cells has a two-dimensional area dimension that is smaller than a two-dimensional area size of a foreground object of interest so that image data of the foreground object in an image data frame spans a contiguous plurality of the cells;

acquire frame image data for each of the cells;

detect and accumulate energy of edges within each of the cells, and generate an edge energy foreground indication output for each of the cells that indicates foreground if an accumulated edge energy meets an edge energy threshold, or indicates background if the accumulated edge energy does not meet the edge energy threshold;

determine color intensities for each of a plurality of different colors within each of the cells, and generate a color intensity foreground indication output for each of the cells that indicates foreground if one of the determined color intensities is greater than another of the determined color intensities for that cell by a color intensity differential threshold, or indicates background if no one of the determined color intensities is greater than any other one of the determined color intensities for that cell by the color intensity differential threshold;

label each of the cells as foreground or background in response to the color intensity foreground indication output and the color intensity foreground indication output for the each cell as a function of a foreground indication output combination rule; and

use the frame image data from the cells labeled as foreground cells to define a foreground object.

22. The article of manufacture of claim 21, wherein the foreground indication output combination rule requires both of the color intensity foreground indication output and the edge energy foreground indication output to indicate foreground to label the cell as foreground, and wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to label each of the cells as foreground or background by:

applying a logical AND operation to inputs of the edge energy foreground indication output and the color intensity foreground indication output that generates a true output for each one of the cells if both of the inputs indicate that the cell is foreground; and

labeling each of the cells having generated true outputs as the foreground cells.

23. The article of manufacture of claim 22, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to, for each of the cells that are indicated as foreground in response to having accumulated edge energy that meets the edge energy threshold:

sum the different determined color intensities to generate a color intensity sum;

compare the generated color intensity sum to sums of the color intensities determined for each of a plurality of cells that are adjacent to said cell having the generated color intensity sum; and

revise the foreground indication output for said cell having the generated color intensity sum to indicate background if said generated color intensity sum is greater than the compared sums of the color intensities determined for each of the plurality of adjacent cells by a color consistency threshold, wherein the foreground indication output is revised prior to labeling of said cells as foreground or background as the function of the foreground indication output combination rule.

24. The article of manufacture of claim 23, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to:

compare the size of blocks formed by contiguous groups of the labeled foreground cells to a template filter; and

revise the labels of each of the cells in the contiguous group of blocks that do not fit the compared template filter to background labels, wherein the labels are revised prior to using the frame image data from the cells labeled as foreground to define the foreground object.

25. The article of manufacture of claim 24, wherein the template has a height and a width, and wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to revise the label of each of the labeled foreground cells of the contiguous group of blocks if a height of the contiguous group of blocks is less than the template height, or if a width of the contiguous group of blocks is less than the template width.