Methods and apparatus for tracking objects using saliency

• US 10,268,919 B1
• Filed: 09/21/2015
• Issued: 04/23/2019
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Piekniewski, Filip, Richert, Micah
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US14/860,579
• Time in Patent Office: 1,310 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 9/7973 by dividing the luminance or colour component signal samples or frequency bands among a plurality of recording channels H04N9/804, H04N9/825 take precedence
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Methods and apparatus for tracking and discerning objects using their saliency. In one embodiment of the present disclosure, the tracking of objects is based on a combination of object saliency and additional sources of signal about object identity. Under certain simplifying assumptions, the present disclosure allows for robust tracking of simple objects with limited processing resources. In one or more variants, efficient implementation of the methods described allow sensors (e.g., cameras) to be used on board a robot (or autonomous vehicle) on a mobile determining platform, such as to capture images to determine the presence and/or identity of salient objects. Such determination of salient objects allow for e.g., adjustments to vehicle or other moving object trajectory.

Claims

1. A method of operating a robotic device, the method comprising:

obtaining a plurality of input images of a visual scene during navigation of a trajectory by the robotic device;

determining a map of a back projected color histogram based upon a plurality of bins, a respective bin of the plurality of bins grouping a range of values corresponding to a difference between a first pixel and a second pixel in a respective input image of the plurality of input images, the second pixel corresponding to an average value of a plurality of pixels of the respective input image, the respective input image corresponding to a surrounding region of the visual scene;

determining a salient feature of a respective pixel of the respective input image based on evaluation of the map, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective input image; and

adjusting the trajectory of the robotic device in accordance with a parameter associated with the salient feature.

2. The method of claim 1, further comprising:

transforming the plurality of input images, the transforming includes applying a low-pass imaging filter to the plurality of input images.

3. The method of claim 2, wherein the transforming of the plurality of input images comprises applying a Gaussian kernel to the plurality of input images.

4. The method of claim 1, further comprising:

determining a shape characteristic of the determined salient feature.

5. The method of claim 4, wherein the determining of the shape characteristic comprises determining a circularity of the salient feature.

6. A controller configured for use with a robotic apparatus, the controller comprising:

a sensor processor configured to execute computer readable instructions to analyze a sensor input, the sensor input being analyzed based at least on;

initialization of a tracking process using color saliency, the tracking process comprising at least one of (i) generation of a heat map corresponding to the sensor input from the color saliency, (ii) determination of a peak value in the generated heat map based on a threshold, and (iii) application of a plurality of thresholds below the peak value to produce a threshold heat map;

identification of a salient feature using the threshold heat map, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective input image; and

communication of tracking information to the controller, the controller configured to cause an adjustment of a trajectory of the robotic apparatus in accordance with a characteristic of the salient feature.

7. The apparatus of claim 6, wherein

the identification of the salient feature is based at least in part on the color saliency;

the tracking information is automatically provided to the tracking process; and

the salient feature is associated with an object and the trajectory adjustment is configured to reduce a spatial separation between the object and the robotic apparatus.

8. The apparatus of claim 7, further comprising:

a manipulator, wherein the tracking information to the controller is further configured to cause the manipulator to pick up the object.

9. The apparatus of claim 6, wherein the identification of the salient feature comprises evaluation of a transformed version of the sensor input comprising at least one of:

(i) a back projected image based on a histogram, and/or (ii) a low-passed image.

10. A non-transitory computer-readable apparatus having a plurality of computer readable instructions stored thereon, that when executed by at least one processor, configure the at least one processor to:

obtain a plurality of images during navigation of a robotic device along a trajectory;

transform the plurality of images via a low-pass operation;

determine a map based upon a difference between (i) a first pixel of a respective image of the plurality of images, and (ii) a second pixel, the second pixel corresponding to an average value of a plurality of pixels of a corresponding image of the transformed plurality of images;

determine change in position of a salient feature of an object from the map between one or more subsequent pairs of the plurality of images, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the corresponding image;

determine a salient object based on an evaluation of the map; and

adjust the trajectory of the robotic device in accordance with the determined salient object.

11. The non-transitory computer-readable apparatus of claim 10, wherein the at least one processor is further configured to execute the computer readable instructions to cause the robotic device to determine a shape characteristic of the determined salient feature.

12. The non-transitory computer-readable apparatus of claim 11, wherein the shape characteristic is characterized by a circularity of the salient feature.

13. The non-transitory computer-readable apparatus of claim 10, wherein the determined salient feature comprises a gesture or a presence of a user.

14. The non-transitory computer-readable apparatus of claim 10, wherein:

the robotic device comprises two or more cameras; and

the two or more cameras are configured to obtain the plurality of images during the navigation of the trajectory.

15. The non-transitory computer-readable apparatus of claim 14, wherein the at least one processor is further configured to execute the computer readable instructions to,

cause the robotic device to determine a distance between the robotic device, and

determined salient feature, the determination of the distance comprising an interleaving of images from the two or more cameras.

16. The non-transitory computer-readable apparatus of claim 15, wherein the at least one processor is further configured to execute the computer readable instructions to,

cause a robotic device to determine a distance between the determined salient feature and a second salient feature, the second salient feature being determined based on an evaluation of a saliency map.

17. A method of operating a robotic platform, the method comprising:

receiving a sensory input including a plurality of images, a respective image including a plurality of colors;

determining a color saliency of the sensory input based on discriminability of a respective color of the plurality of colors in the sensory input from a reference color prototype;

initializing a tracking process using the determined color saliency to produce a heat map, the heat map comprising one or more potential salient features;

identifying a salient feature using shape information associated with the one or more potential salient features in the heat map, the shape information being correlated to each of the one or more potential salient features and a respective auxiliary shape associated with each of the one or more potential salient features, the salient feature corresponding to a number of pixels representing the same exact value in remaining of the respective image; and

adjusting trajectory of the robotic platform in accordance with a characteristic of the salient feature upon the tracking information being communicated to the controller.

18. The method of claim 17, further comprising:

determining a saliency map, the saliency map comprising the color saliency; and

identifying the salient feature within the saliency map.

19. The method of claim 18, wherein the identifying of the salient feature comprises:

applying a filtering operation to the saliency map;

determining the color saliency of an object comprising the salient feature; and

implementing a variable threshold for detecting saliency.

20. The method of claim 17, wherein the causing of the adjustment of the trajectory of the robotic platform comprises:

reducing a spatial separation between the robotic platform and an object comprising the salient feature; and

picking up the object.