Methods and apparatus for autonomous robotic control

• US 10,503,976 B2
• Filed: 03/20/2017
• Issued: 12/10/2019
• Est. Priority Date: 03/19/2014
• Status: Active Grant

• Current Assignee: Neurala, LLC, Trustees of Boston University
• Original Assignee: Neurala, LLC
• Inventors: Versace, Massimiliano, Gorshechnikov, Anatoly
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US15/463,299
• Publication Number:
  US 20170193298A1
• Time in Patent Office: 995 Days
• Field of Search: None
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39082 Collision, real time collision avoidance
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot'"'"'s environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements.

Claims

1. A method comprising:

acquiring a plurality of images of at least a portion of an environment with an image sensor;

constructing a spatial shroud fitting a form of an object in a first image of the plurality of images from the first image;

identifying the object in the first image based at least in part on the spatial shroud;

translating and/or transforming the spatial shroud based at least in part on a change in a position and/or an orientation of the image sensor;

determining if the spatial shroud fits the form of the object in a second image in the plurality of images; and

in response to determining that the spatial shroud fits the form of the object, identifying a position of the object in the environment based at least in part on the spatial shroud.

2. The method of claim 1, wherein constructing the spatial shroud comprises segmenting the first image.

3. The method of claim 1, wherein identifying the object includes creating a shroud-gated representation of a view of the object.

4. The method of claim 1, further comprising:

extracting a feature from the first image; and

identifying the object in the first image based at least in part on the feature.

5. The method of claim 4, wherein extracting the feature comprises applying a non-linear transformation to the first image.

6. The method of claim 4, further comprising:

identifying similarities between the feature and a plurality of features identified in other images in the plurality of images.

7. The method of claim 6, further comprising:

clustering the images in the plurality of images based at least in part on the similarities.

15. The method of claim 1, further comprising:

positioning and/or orienting the image sensor with respect to the object based at least in part on the object.

16. The method of claim 1, further comprising:

adjusting a field of view of the image sensor based at least in part on the change in the position and/or the orientation of the image sensor.

19. The method of claim 1, wherein the object is a previously unknown new object.

20. The method of claim 1, further comprising:

determining that the spatial shroud does not fit the form of the object in a third image in the plurality of images; and

in response to determining that the spatial shroud does not fit the form of the object in the third image, identifying a second object in the third image different from the object; and

transmitting instructions to the image sensor to capture images of the second object.

21. The method of claim 1, wherein constructing the spatial shroud occurs without supervision from a user.

22. The method of claim 1, wherein constructing the spatial shroud occurs without manual labeling of the object.

23. The method of claim 1, wherein constructing the spatial shroud occurs without corrupting or forgetting previously learned knowledge.

8. A system comprising:

an image sensor to acquire a plurality of images of at least a portion of an environment;

a processor communicably coupled to the image sensor, the processor to;

construct a spatial shroud fitting a form of an object in a first image of the plurality of images from the first image;

identify the object in the first image based at least in part on the spatial shroud;

translate and/or transform the spatial shroud based at least in part on a change in a position and/or an orientation of the image sensor;

determine if the spatial shroud fits the form of the object in a second image in the plurality of images; and

in response to determining that the spatial shroud fits the form of the object in the second image, identify a position of the object in the environment based at least in part on the spatial shroud.

9. The system of claim 8, wherein the processor is configured to construct the spatial shroud by segmenting the first image.

10. The system of claim 8, wherein the processor is configured to identify the object by creating a shroud-gated representation of a view of the object.

11. The system of claim 8, wherein the processor is further configured to:

extract a feature from the first image; and

identify the object in the first image based at least in part on the feature.

12. The system of claim 11, wherein the processor is configured to extract the feature by applying a non-linear transformation to the first image.

13. The system of claim 11, wherein the processor is further configured to:

identify similarities between the feature and a plurality of features identified in other images in the plurality of images.

14. The system of claim 13, wherein the processor is further configured to:

cluster the images in the plurality of images based at least in part on the similarities.

17. The system of claim 8, further comprising:

an actuator, operably coupled to the processor and to the image sensor, to adjust a field of view of the image sensor based at least in part on the object.

18. The system of claim 17, wherein the actuator is further configured to adjust the field of view of the image sensor based at least in part on the change in the position and/or the orientation of the image sensor.