Methods and apparatus for autonomous robotic control

• US 10,083,523 B2
• Filed: 09/12/2016
• Issued: 09/25/2018
• Est. Priority Date: 03/19/2014
• Status: Active Grant

• Current Assignee: Neurala, LLC
• Original Assignee: Neurala, LLC
• Inventors: Versace, Massimiliano, Gorshechnikov, Anatoly
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/262,637
• Publication Number:
  US 20170024877A1
• Time in Patent Office: 743 Days
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: Y04S 10/50 Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
• CPC Class Codes: Y10S 901/44 inspection

Abstract

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot'"'"'s environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements.

Claims

1. A system for automatically locating and identifying an object in an environment, the system comprising:

at least one sensor to acquire sensor data representing at least a portion of the environment;

at least one processor operably coupled to the at least one sensor; and

at least one memory operably coupled to the at least one processor, the at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to implement;

a spatial attention module to produce a foveated representation of the object based at least in part on the sensor data, to track a position of the object within the environment based at least in part on the foveated representation, and to select another portion of the environment to be sensed by the at least one sensor based at least in part on the foveated representation of the object; and

a semantics module to determine an identity of the object based at least in part on the foveated representation of the object,

wherein the spatial attention module comprises a segmentation module to generate at least one contour representation of the object based at least in part on the sensor data,

wherein the spatial attention module further comprises a figure/ground segregation module to determine at least one spatial shroud fitting a form of the object based at least in part on the at least one contour representation of the object, and

wherein the sensor data comprises a plurality of images and the semantics module comprises;

a view layer to group views of the object in the plurality of images based at least in part on the at least one spatial shroud;

an object layer to map the views of the object to an object node associated with the object; and

a name layer to classify the object based at least in part on the object node.

2. The system of claim 1, wherein the at least one sensor comprises an image sensor to acquire at least one image of the at least a portion of the environment.

3. The system of claim 2, further comprising:

at least one actuator, operably coupled to the image sensor, to provide sensor position data representative of an orientation and/or a position of the image sensor, and

wherein the spatial attention module is configured to select the other portion of the environment based at least in part on the orientation and/or a position of the image sensor.

4. The system of claim 3, wherein the

at least one memory is configured to store the sensor position data representing an orientation and/or a position of the image sensor and to store instructions that, when executed by the at least one processor, cause the processor to implement:

an inhibition of return module to inhibit repeated selections of the object based at least in part on the sensor position data stored in the memory.

5. The system of claim 1, wherein the spatial attention module configured to select the other portion of the environment such that the object appears at or near a center of the other portion of the environment.

6. The system of claim 1, wherein the spatial attention module comprises:

at least one log-polar transformation module to transform the sensor data into a log-polar representation of the environment so as to provide invariance to translation and/or rotation of the at least one sensor with respect to the object and/or so as to reduce processing time.

7. The system of claim 1, wherein the view layer is configured to group the views of the object based at least in part on classification of the object by the name layer.

8. The system of claim 1, wherein the semantics module is configured to learn the identity of the object based at least in part on the location of the object.

9. The system of claim 1, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to implement:

a teaching module to provide a label for the object, and

wherein the semantics module is configured to assign the label to the object.

10. A method of automatically locating and identifying an object in an environment, the method comprising:

(A) estimating a position and/or an orientation of at least one sensor with respect to the environment;

(B) acquiring, with the at least one sensor, sensor data representing at least a portion of the environment;

(C) producing a foveated representation of the object based at least in part on the sensor data acquired in (B);

(D) determining an identity of the object based at least in part on the foveated representation of the object produced in (C); and

(E) selecting another portion of the environment to be sensed by the at least one sensor based at least in part on the foveated representation of the object produced in (C) and the position and/or the orientation estimated in (A);

(F) acquiring additional sensor data, with the at least one sensor, in response to selection of the other portion of the environment in (D), wherein;

(A) comprises acquiring a plurality of images,

(D) comprises generating at least one contour representation of the object based at least in part on at least one image and determining at least one spatial shroud fitting a form of the object based at least in part on the at least one contour representation of the object, and

(E) comprises;

(E1) grouping views of the object in the plurality of images based at least in part on the at least one spatial shroud;

(E2) mapping the views of the object to an object node associated with the object; and

(E3) classifying the object based at least in part on the object node.

11. The method of claim 10, wherein (D) comprises selecting the other portion of the environment such that the object appears at or near a center of the other portion of the environment.

12. The method of claim 10, wherein (D) comprises inhibiting repeated selections of a given portion of the environment based at least in part on the position estimated in (A).

13. The method of claim 10, wherein (D) comprises transforming the sensor data into a log-polar representation of the environment so as to provide invariance to translation and/or rotation and/or so as to reduce processing time.

14. The method of claim 10, wherein (E1) comprises grouping the views of the object based at least in part on classification of the object in (E3).