Methods and Apparatus for Autonomous Robotic Control

• US 20170024877A1
• Filed: 09/12/2016
• Published: 01/26/2017
• Est. Priority Date: 03/19/2014
• Status: Active Grant

• Current Assignee: Neurala, LLC
• Original Assignee: Neurala, LLC
• Inventors: GORSHECHNIKOV, Anatoly, Versace, Massimiliano
• Granted Patent: US 10,083,523 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: Y04S 10/50 Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
• CPC Class Codes: Y10S 901/44 inspection

Abstract

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot'"'"'s environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements.

Claims

1. A system for automatically locating and identifying an object in an environment, the system comprising:

at least one sensor to acquire sensor data representing of at least a portion of the object;

a spatial attention module, operably coupled to the at least one sensor, to produce a foveated representation of the object based at least in part on the sensor data, to track a position of the object within the environment based at least in part on the foveated representation, and to select another portion of the environment to be sensed by the at least one sensor based at least in part on the foveated representation of the object; and

a semantics module, operably coupled to the spatial attention module, to determine an identity of the object based at least in part on the foveated representation of the object.

2. The system of claim 1, wherein the at least one sensor comprises an image sensor to acquire at least one image of the at least a portion of the environment.

3. The system of claim 2, further comprising:

at least one actuator, operably coupled to the image sensor, to provide sensor position data representative of an orientation and/or a position of the image sensor, and

wherein the spatial attention module is configured to select the other portion of the environment based at least in part on the orientation and/or a position of the image sensor.

4. The system of claim 1, wherein the spatial attention module configured to select the other portion of the environment such that the object appears at or near a center of the other portion of the environment.

5. The system of claim 1, wherein the spatial attention module comprises:

a memory to store the sensor position data representing an orientation and/or a position of the sensor; and

an inhibition of return module, operably coupled to the memory, to inhibit repeated selections of a given object based at least in part on the sensor position data stored in the memory.

6. The system of claim 1, wherein the spatial attention module comprises:

at least one log-polar transformation module to transform the sensor data into a log-polar representation of the environment so as to provide invariance to translation and/or rotation of the at least one sensor with respect to the object and/or so as to reduce processing time.

7. The system of claim 1, wherein the spatial attention module comprises:

a segmentation module to generate at least one contour representation of the object based at least in part on the sensor data.

8. The system of claim 7, wherein the spatial attention module further comprises:

a figure/ground segregation module, operably coupled to the segmentation module, to determine at least one spatial shroud fitting a form of the object based at least in part on the at least one contour representation of the object.

9. The system of claim 8, wherein the sensor data comprises a plurality of images and the semantics module comprises:

a view layer to group views of the object in the plurality of images based at least in part on the at least one spatial shroud;

an object layer, operably coupled to the view layer, to map the views of the object to an object node associated with the object; and

a name layer, operably coupled to the object layer, to classify the object based at least in part on the object node.

10. The system of claim 9, wherein the view layer is operably coupled to the name layer and configured to group the views of the object based at least in part on classification of the object by the name layer.

11. The system of claim 1, wherein the semantics module is configured to learn the identity of the object based at least in part on the location of the object.

12. The system of claim 1, further comprising:

a teaching module, operably coupled to the at least one sensor and to the semantic system, to provide a label for the object, and

wherein the semantics module is configured to assign the label to the object.

13. A method of automatically locating and identifying an object in an environment, the method comprising:

(A) estimating a position and/or an orientation of at least one sensor with respect to the environment;

(B) acquiring, with the at least one sensor, sensor data representing at least a portion of the object;

(C) producing a foveated representation of the object based at least in part on the sensor data acquired in (B);

(D) determining an identity of the object based at least in part on the foveated representation of the object produced in (C); and

(E) selecting another portion of the environment to be sensed by the at least one sensor based at least in part on the foveated representation of the object produced in (C) and the position and/or the orientation estimated in (A);

(F) acquiring additional sensor data, with the at least one sensor, in response to selection of the other portion of the environment in (D).

14. The method of claim 13, wherein (D) comprises selecting the other portion of the environment such that the object appears at or near a center of the other portion of the environment.

15. The method of claim 13, wherein (D) comprises inhibiting repeated selections of a given portion of the environment based at least in part on the position estimated in (A).

16. The method of claim 13, wherein (D) comprises transforming the sensor data into a log-polar representation of the environment so as to provide invariance to translation and/or rotation and/or so as to reduce processing time.

17. The method of claim 13, wherein:

(A) comprises acquiring a plurality of images; and

(D) comprises generating at least one contour representation of the object based at least in part on the at least one image.

18. The method of claim 17, wherein (D) further comprises determining at least one spatial shroud fitting a form of the object based at least in part on the at least one contour representation of the object.

19. The method of claim 18, wherein (E) comprises:

(E1) grouping views of the object in the plurality of images based at least in part on the at least one spatial shroud;

(E2) mapping the views of the object to an object node associated with the object; and

(E3) classifying the object based at least in part on the object node.

20. The method of claim 19, wherein (El) comprises grouping the views of the object based at least in part on classification of the object in (E3).

21. A system for automatically locating and identifying an object in an environment, the system comprising:

a plurality of sensors to acquire multi-sensory data representative of the environment;

a plurality of first spatial attention modules, each first spatial attention module in the plurality of first spatial attention modules operably coupled to a respective sensor in the plurality of sensors and configured to produce a respective representation of the object based on a respective portion of the multi-sensory data;

a plurality of first semantics modules, each first semantics module in the plurality of first spatial attention modules operably coupled to a respective first spatial attention module in the plurality of first spatial attention modules and configured to produce respective identifications of the object based on an output from the respective first spatial attention module;

a second spatial attention module, operably coupled to the plurality of first spatial attention modules, to estimate a location of the object based at least in part on the multi-sensory representation of the environment; and

a second semantics module, operably coupled to the plurality of first semantics modules and the second spatial attention module, to identify the at least one object based in part of the respective identifications produced by the plurality of first semantics modules and the location of the object estimated by the second spatial attention module.

22. The system of claim 21, wherein the second spatial attention module is configured to dynamically allocate the plurality of sensors so as to enhance detection of the at least one object and/or to suppress noise in the multi-sensory representation of the scene.

23. The system of claim 21, wherein the second semantics module is configured to generate scene metadata representative of an identity of the at least one object and/or the at least one change to the object based at least in part on the multi-sensory representation of the environment.