Apparatus and methods for saliency detection based on color occurrence analysis

• US 9,870,617 B2
• Filed: 03/03/2015
• Issued: 01/16/2018
• Est. Priority Date: 09/19/2014
• Status: Expired due to Fees

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Piekniewski, Filip, Richert, Micah, Fisher, Dimitry
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): RIVERA-MARTINEZ, GUILLERMO M
• Application Number:
  US14/637,191
• Publication Number:
  US 20160086052A1
• Time in Patent Office: 1,050 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 9/7973 by dividing the luminance or colour component signal samples or frequency bands among a plurality of recording channels H04N9/804, H04N9/825 take precedence
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Apparatus and methods for detecting and utilizing saliency in digital images. In one implementation, salient objects may be detected based on analysis of pixel characteristics. Least frequently occurring pixel values may be deemed as salient. Pixel values in an image may be compared to a reference. Color distance may be determined based on a difference between reference color and pixel color. Individual image channels may be scaled when determining saliency in a multi-channel image. Areas of high saliency may be analyzed to determine object position, shape, and/or color. Multiple saliency maps may be additively or multiplicative combined in order to improve detection performance (e.g., reduce number of false positives). Methodologies described herein may enable robust tracking of objects utilizing fewer determination resources. Efficient implementation of the methods described below may allow them to be used for example on board a robot (or autonomous vehicle) or a mobile determining platform.

Claims

1. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions being executable by a processing apparatus to configure a process enabled to track an object in a plurality of images each comprised of pixels, wherein the instructions, when executed by the processing apparatus cause the process to:

evaluate a pixel chromaticity distribution for pixels of a given image of the plurality of images, the evaluation of the pixel chromaticity distribution comprising a reconstruction of the given image and an assignment of a saliency value within the reconstructed image;

based at least on the evaluation, identify a portion of pixels of the given image, the identification of the portion of pixels being based on the assigned saliency value within the reconstructed image;

provide positions corresponding respectively to individual pixels of the portion to the process; and

based on the provided positions, navigate a robotic apparatus in communication with the processing apparatus;

wherein;

the portion of pixels corresponds to a representation of the object in the given image;

the provided positions are configured to enable the process to determine a location of the object in a subsequent image of the plurality of images;

the evaluation of the pixel chromaticity distribution comprises a determination of a histogram of occurrence of pixel chromaticity within the given image.

2. The non-transitory computer-readable storage medium of claim 1, wherein:

the pixel chromaticity distribution is configured to characterize a frequency of occurrence of pixel chromaticity within the given image; and

the identification of the portion of pixels is further based at least in part on the frequency of occurrence of pixel chromaticity breaching a threshold.

3. The non-transitory computer-readable storage medium of claim 2, wherein identification of the portion of pixels of the given image comprises:

(i) a determination of a spatial distribution of a difference measure between chromaticity of pixels of the given image and a reference chromaticity; and

(ii) a comparison of the difference measure at a pixel location to a level.

4. The non-transitory computer-readable storage medium of claim 3, wherein the reference chromaticity is determined based at least on evaluation of the frequency of occurrence of pixel chromaticity.

5. The non-transitory computer-readable storage medium of claim 4, wherein:

the reference chromaticity is determined further based at least on a prevailing chromaticity of pixels satisfying a condition, and

the condition comprises the frequency of occurrence of pixel chromaticity being below the threshold.

6. The non-transitory computer-readable storage medium of claim 2, wherein the configuration to enable the process further includes producing a back projected map comprising a plurality of entries, a value of a given entry at a given location being assigned a value corresponding to a chromaticity occurrence of a corresponding pixel at the given location within the given image.

7. The non-transitory computer-readable storage medium of claim 2, wherein the portion of pixels corresponds to a pixel chromaticity occurrence being in the range comprising 0.1% to 5%, inclusive.

8. The non-transitory computer-readable storage medium of claim 2, wherein:

the pixel chromaticity is characterized by first and second chromaticity channels; and

the histogram comprises a first plurality of bins characterizing the first chromaticity channel, and a second plurality of bins characterizing the second chromaticity channel.

9. The non-transitory computer-readable storage medium of claim 2, wherein:

a number of bins of a first plurality of the pixel chromaticity is characterized by first and second chromaticity channels; and

the histogram comprises a first plurality of bins characterizing the first chromaticity channel, and a second plurality of bins characterizing the second chromaticity channel.

10. The non-transitory computer-readable storage medium of claim 9, wherein:

values of the first and the second chromaticity channels are characterized by an 8-bit value; and

a number of at least one of the first plurality of bins and/or the second plurality of bins is selected from a range comprising 2 to 8 bins, inclusive.

11. The non-transitory computer-readable storage medium of claim 1, wherein the evaluation of the pixel chromaticity distribution comprises a determination of a pixel chromaticity distribution model characterized by at least two parameters.

12. The non-transitory computer-readable storage medium of claim 1, wherein the evaluation of the pixel chromaticity distribution comprises a determination of a pixel chromaticity distribution model, the determination of the model based at least on analysis of a portion of a plurality of images preceding the given image.

13. The non-transitory computer-readable storage medium of claim 1, wherein the provision of the positions is configured to reduce a number of occurrences of false positives associated with detection of the object by the process in the another image, as compared to detection of the object by the process in the another image in the absence of the provision of the positions.

14. The non-transitory computer-readable storage medium of claim 1, wherein:

the identification of the portion of pixels is further configured based at least on a determination of a low-pass filtered version of the difference measure; and

the configuration of the process further includes a determination of a shape parameter associated with a portion of the low-pass filtered version.

15. The non-transitory computer-readable storage medium of claim 14, wherein the portion of the low-pass filtered version is identified based at least on a greatest contiguous number of pixels.

16. The non-transitory computer-readable storage medium of claim 14, wherein the portion of the low-pass filtered version is identified based at least on a greatest cumulative saliency value, the greatest cumulative saliency value being determined by at least accumulation of a plurality of saliency values.

17. The non-transitory computer-readable storage medium of claim 14, wherein the shape parameter comprises a circularity determined based at least on:

a determination of one or more contours associated with one or more contiguous pixel groups within the low-pass filtered version;

for individual ones of the one or more contours, a determination of a smallest enclosing circle; and

a ratio between an area of the individual ones of the one or more contours and an area of the smallest enclosing circle.

18. The non-transitory computer-readable storage medium of claim 1, wherein:

the object comprises a target for the robotic apparatus; and

the tracking of the object is configured to cause a provision of a control command to the robotic apparatus to cause the robotic apparatus to approach the object.

19. The non-transitory computer-readable storage medium of claim 1, wherein:

the object comprises a target object of a given color; and

the tracking of the object is configured to enable the robotic device to pick up the target object from a plurality of objects characterized by a plurality of colors, individual ones of the plurality of objects being characterized by a color of the plurality of colors.

20. A method of tracking an object in a sequence of images, the method comprising:

evaluating a pixel chromaticity distribution for pixels of a given image of the sequence of images, the evaluating of the pixel chromaticity distribution comprising reconstructing the given image and assigning a saliency value within the reconstructed image, the evaluating of the pixel chromaticity distribution comprising determining a histogram of occurrence of pixel chromaticity within the given image;

based at least on the evaluating, identifying a portion of pixels of the given image, the identifying of the portion of pixels being based on the assigned saliency value within the reconstructed image;

providing, to a computerized process enabled to track the object, positions corresponding respectively to individual pixels of the portion of pixels, the provided positions enabling the process to determine a location of the object in a subsequent image of the sequence of images; and

based on the provided positions, navigating a robotic apparatus that is in communication with the computerized process;

wherein the portion of pixels corresponds to a representation of the object in the given image.

21. The method of claim 20, wherein the evaluating of the pixel chromaticity distribution further comprises determining a pixel chromaticity distribution model, the determining of the model being based at least on analyzing a portion of the sequence of images preceding the given image.

22. The method of claim 20, wherein:

the reconstructing of the given image comprises reconstructing the given image using the pixel chromaticity distribution for pixels of the given image; and

the assigning of the saliency value within the reconstructed image comprises assigning the saliency value at a pixel location corresponding to a pixel location within the given image.

23. The method of claim 20, wherein:

the evaluating of the pixel chromaticity distribution further comprises determining a frequency of occurrence of a pixel value of a given pixel within the reconstructed image; and

the identifying of the portion of pixels is further based on the frequency of occurrence of the pixel value of the given pixel within the reconstructed image, the pixel value being inversely related to the saliency value, the saliency value being associated with the given pixel.