Apparatus and methods for tracking salient features

• US 10,032,280 B2
• Filed: 03/03/2015
• Issued: 07/24/2018
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Piekniewski, Filip, Richert, Micah, Fisher, Dimitry
• Primary Examiner(s): CESE, KENNY A
• Application Number:
  US14/637,164
• Publication Number:
  US 20160086051A1
• Time in Patent Office: 1,239 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 9/7973 by dividing the luminance or colour component signal samples or frequency bands among a plurality of recording channels H04N9/804, H04N9/825 take precedence
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Apparatus and methods for detecting and utilizing saliency in digital images. In one implementation, salient objects may be detected based on analysis of pixel characteristics. Least frequently occurring pixel values may be deemed as salient. Pixel values in an image may be compared to a reference. Color distance may be determined based on a difference between reference color and pixel color. Individual image channels may be scaled when determining saliency in a multi-channel image. Areas of high saliency may be analyzed to determine object position, shape, and/or color. Multiple saliency maps may be additively or multiplicative combined in order to improve detection performance (e.g., reduce number of false positives). Methodologies described herein may enable robust tracking of objects utilizing fewer determination resources. Efficient implementation of the methods described below may allow them to be used for example on board a robot (or autonomous vehicle) or a mobile determining platform.

Claims

1. A robotic apparatus configured to track a target object comprising:

a camera; and

a non-transitory computer-readable medium having computer-readable instructions stored thereon, that when executed by at least one processor apparatus, configure the at least one processor apparatus to,

obtain a plurality of images via the camera while navigating a trajectory associated with a physical task;

identify a salient area in the plurality of images, the identification of the salient area being based at least on a contrast between (i) a relative motion associated with the target object and (ii) a relative motion associated with another object within a field of view of the camera, the target object being within the field of view, where the relative motion associated with the target object is determined from a combination of a motion component associated with a movement of the object and a motion component associated with the navigation of the trajectory of the robotic apparatus;

initialize an object tracking process based at least on the identification of the salient area, the object tracking process comprising a detection of the object from the salient area; and

adjust the trajectory associated with the physical task based on the detection of the object from the salient area.

2. A non-transitory computer-readable medium having computer readable instructions stored thereon, that when executed by at least one processor, configure the at least one processor to,

determine a relative motion of an object in a sequence of one or more images when a robotic apparatus is in motion along a trajectory, the relative motion of the object being determined based on (i) determining a first motion component, (ii) determining a second motion component, and (iii) combining the first motion component and the second motion component, the first motion component being associated with the motion along the trajectory of the robotic apparatus, the second motion component being associated with a movement or an absence of the movement of the object, the object being located within a visual scene;

identify a salient area in the sequence of one or more images based on the determined relative motion of the object, the salient area identified being based on determining a contrast between (i) the determined relative motion of the object and (ii) a relative motion of at least one other object within the visual scene, the relative motion of the at least one other object being based on the first motion component and a third motion component, the third motion component being associated with a movement or an absence of the movement of the at least one other object; and

initialize an object tracking process based at least on the identifying of the salient area.

3. The non-transitory computer readable medium of claim 2, wherein the at least one processor is further configured to execute the computer readable instructions to reduce a number of occurrences of false positives associated with detecting the object in subsequent one or more images of the plurality of images when the object tracking process is initialized, as compared to detecting the object in the subsequent one or more images in absence of the initializing.

4. The non-transitory computer readable medium of claim 3, wherein,

the one or more images include individual pixels, the individual pixels characterized by pixel characteristics, and

the at least one processor is further configured to execute the computer readable instructions to determine a saliency distribution of pixels of the one or more images of the sequence of images based on at least one of the one or more pixel characteristics during identifying of the salient area.

5. The non-transitory computer readable medium of claim 4, wherein,

the one or more pixel characteristics comprises a pixel chromaticity parameter;

and

the at least one processor is further configured to execute the computer readable instructions to determine a spatial distribution of pixel color distance in the one or more images when the saliency distribution is determined, the pixel color distance being determined based at least on a distance measure between the chromaticity parameter of individual pixels of the one or more images and a reference chromaticity.

6. The non-transitory computer readable medium of claim 5, wherein the reference chromaticity is determined based at least on determining a pixel chromaticity occurrence statistics parameter.

7. The non-transitory computer readable medium of claim 5, wherein the identification of the salient area in the one or more images comprises causing the at least one processor to execute the computer readable instructions to,

determine that one or more values of the spatial distribution of pixel color distance breach a threshold;

and

determine a thresholded pixel distance map based at least on determining a location of one or more pixels within the one or more images, the location corresponding to the one or more values of the spatial distribution of pixel color distance.

8. The non-transitory computer readable medium of claim 7, wherein the identification of the salient area in an image further comprises, the at least one processor is further configured to execute the computer readable instructions to,

determine a low-pass filtered version of the thresholded pixel distance map;

and

determine a shape parameter associated with at least a portion of the low-pass filtered version.

9. The non-transitory computer readable medium of claim 8, wherein the identification of the salient area in the image further comprises causing the at least one processor to execute the computer readable instructions to,

determine one or more contours associated with one or more contiguous pixel groups within the low pass filtered version;

and

for a given contour of the one or more individual contours, the at least one processor is further configured to execute the computer readable instructions to,

determine a smallest enclosing circle; and

determine a circularity based at least on a ratio between an area of the given contour and an area of the smallest enclosing circle.

10. The non-transitory computer readable medium of claim 9, wherein the at least portion of the low-pass filtered version is selected based at least on an evaluation of a greatest contiguous number of pixels.

11. The non-transitory computer readable medium of claim 8, wherein the at least portion of the low-pass filtered version is selected based at least on a greatest cumulative saliency value, the value determined by accumulating a plurality of saliency values.

12. The non-transitory computer readable medium of claim 3, wherein the identification of the salient area in the one or more images comprises causing the at least one processor to execute the computer readable instructions to,

partition the image into a plurality of partitions, individual ones of the plurality of partitions comprising a plurality of adjacent pixels in a contiguous area;

convolve individual ones of the plurality of partitions with one or more spatial filter masks to produce a plurality of convolution magnitudes, individual ones of the one or more spatial filters characterized by filter orientation; and

determine the salient area corresponding to a least frequently occurring filter convolution magnitude above a threshold.

13. The non-transitory computer readable medium of claim 3, wherein the salient area is identified based at least on analysis of displacement of pixels of the one or more images relative to pixels of a preceding image of the sequence of images.

14. The non-transitory computer readable medium of claim 13, wherein:

the analysis of displacement comprises causing the at least one processor to execute the computer readable instructions to,

determine a motion map of pixels of the one or more images, the motion map at least characterizing magnitude and direction of motion of pixels in the one or more images associated with representation of the object; and

determine the saliency map based at least on an additional information related to motion of the object.

15. The non-transitory computer readable medium of claim 14, wherein the additional information is obtained during a tracker priming operation, the operation comprising the at least one processor is further configured to execute the computer readable instructions to analyze a plurality of images characterizing motion of the object, the priming operation comprising motion of the object in a given pattern and in absence of other sources motion in individual ones of the plurality of images.

16. The non-transitory computer readable medium of claim 3, wherein:

the one or more images of the sequence of images are obtained by a sensor configured to sample energy reflected from an environment;

individual pixels of the one or more images are characterized by a range channel corresponding to a range value between the sensor and a location within the environment; and

the salient area is determined based at least on information related to a given range to the object and a comparison of range channel value associated with pixels in the salient area to the range to the object.

17. The non-transitory computer readable medium of claim 4, wherein:

the one or more images are characterized by a color space comprising at least one dimension; and

the pixel characteristics are configured to convey information related to the at least one dimension.

18. The non-transitory computer readable medium of claim 17, wherein:

the salient area is characterized by a first pixel occurrence;

the saliency distribution comprises a background portion characterized by a second pixel occurrence, the second occurrence being greater than the first occurrence; and

the first pixel occurrence is configured to provide a reference within the color space.

19. The non-transitory computer readable medium of claim 18, wherein:

the at least one dimension comprises a chromaticity dimension; and

the reference comprises a reference chromaticity.

20. The non-transitory computer readable medium of claim 19, wherein the at least one processor is further configured to execute the computer readable instructions to determine the reference chromaticity based on a distribution of pixel chromaticity occurrence within the one or more images.

21. The non-transitory computer readable medium of claim 20, wherein the at least one processor is further configured to execute the computer readable instructions to determine a difference measure between chromaticity of individual pixels within the one or more images and the reference chromaticity;

and the determination of the reference chromaticity based on the pixel chromaticity occurrence is configured to increase a discriminability between a distance measure associated with pixels corresponding to the salient area of the one or more images and a distance measure corresponding to the background portion of the one or more images.

22. The non-transitory computer readable medium of claim 21, wherein the increased discriminability is configured based on a greater difference between (i) the distance measure associated with pixels corresponding to the salient area of the one or more images, and (ii) a distance measure corresponding to the background portion of the one or more images when compared to respective distance measures determined based on another reference chromaticity, the another reference chromacity determined based at least on pixels disposed outside the salient area of the one or more images.