Methods and apparatus for early sensory integration and robust acquisition of real world knowledge

• US 10,300,603 B2
• Filed: 11/20/2015
• Issued: 05/28/2019
• Est. Priority Date: 05/22/2013
• Status: Active Grant

• Current Assignee: Neurala, LLC, Trustees of Boston University
• Original Assignee: Neurala, LLC
• Inventors: Gorshechnikov, Anatoly, Versace, Massimiliano, Barnes, Tim
• Primary Examiner(s): Diallo, Mamadou L
• Application Number:
  US14/947,516
• Publication Number:
  US 20160082597A1
• Time in Patent Office: 1,285 Days
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: Y04S 10/50 Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/46 Sensing device
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The systems and methods disclosed herein include a path integration system that calculates optic flow, infers angular velocity from the flow field, and incorporates this velocity estimate into heading calculations. The resulting system fuses heading estimates from accelerometers, gyroscopes, engine torques, and optic flow to determine self-localization. The system also includes a motivational system that implements a reward drive, both positive and negative, into the system. In some implementations, the drives can include: a) a curiosity drive that encourages exploration of new areas, b) a resource drive that attracts the agent towards the recharging base when the battery is low, and c) a mineral reward drive that attracts the agent towards previously explored scientific targets.

Claims

1. A method for forming at least one neural network representation of a robot with respect to an environment, the method comprising:

(A) receiving, by a processor, a plurality of sets of data readings from at least one sensor, the plurality of sets of data readings representing independent measurements by the at least one sensor of the robot with respect to the environment;

(B) mapping the plurality of sets of data readings received in (A) to respective first sets of data cells, each first set of data cells forming a corresponding first neural representation of the robot with respect to the environment;

(C) applying respective weights to values in the respective first sets of data cells mapped in (B), the respective weights representing respective precisions of the independent measurements represented by the corresponding sets of data readings;

(D) combining the values in the first sets of data cells weighted in (C) to form a plurality of aggregated data readings; and

(E) mapping the plurality of aggregated data readings formed in (D) to a second set of data cells stored in the memory, the second set of data cells forming a second neural representation of the robot with respect to the environment that is more precise than the first neural representations of the robot with respect to the environment.

2. The method of claim 1, wherein (A) comprises receiving data readings representative of an orientation, motion, and/or position of the robot with respect to the environment.

3. The method of claim 1, wherein (A) comprises processing data from a single sensor to form the first plurality of data readings and the second plurality of data readings by extracting different modalities of data reading from the data from the single sensor.

4. The method of claim 1, wherein (A) comprises receiving the first plurality of data readings from a first sensor and the second plurality of data readings from a second sensor.

5. The method of claim 1, wherein (A) comprises receiving the first plurality of data readings from at least one of a motor, a motor controller, an accelerometer, a gyroscope, an inertial measurement unit, an imaging sensor, a proximity sensor, a photosensor, a sonar, a lidar, a radar, a rangefinder, and a microphone.

6. The method of claim 1, wherein the first plurality of data readings represents at least one of a motor outflow, an optic flow, an inertial measurement unit measurement, a color measurement, a proximity measurement, a luminance measurement, a velocity measurement, an orientation measurement, an acceleration measurement, and a distance measurement.

7. The method of claim 1, wherein (A) comprises:

receiving a first set of data readings representing measurements by an inertial measurement unit (IMU) of the robot;

receiving a second set of data readings representing commands issued to a motor of the robot; and

receiving a third set of data readings representing an optic flow detected by an imaging sensor of the robot.

8. The method of claim 7, wherein (C) comprises:

applying a first weight to the first set of data readings, a second weight to the second set of data readings, and a third weight to the third set of data readings;

increasing the first weight relative to the second weight and the third weight for data reading indicative of an increasing speed of the robot;

increasing the third weight relative to the first weight and the second weight for data reading indicative of a decreasing speed of the robot; and

varying the second weight from a value of zero for data readings indicative of no motion of the robot to a value between the first weight and the third weight for data readings indicative of motion of the robot.

9. The method of claim 8, further comprising:

(E) estimating the position and the orientation of the robot with respect to the environment based on the second neural representation.

10. The method of claim 1, wherein (A) comprises:

receiving sensor data from an imaging sensor associated with the robot;

deriving a first set of data readings from the sensor data, the first set of data readings representing at least one edge sensed by the imaging sensor;

deriving a second set of data readings from the sensor data, the second set of data readings representing an estimate of distance between the robot and at least one object in the environment; and

deriving a third set of data readings from the sensor data, the third set of data readings representing motion of the robot.

11. The method of claim 10, wherein (C) comprises:

applying a first weight to the first set of data readings, a second weight to the second set of data readings, and a third weight to the third set of data readings;

decreasing the first weight relative to the second weight and the third weight for data readings indicative of low contrast imagery sensed by the imaging sensor;

decreasing the second weight relative to the first weight and the third weight for data readings indicative of changes in depth below a predetermined threshold sensed by the imaging sensor; and

decreasing the third weight relative to the first weight and the second weight for data readings indicative of motion of the robot.

12. The method of claim 1, wherein (B) comprises:

generating at least one first set of data cells representing respective data points in a measurement range of the at least one sensor, the respective data points representing a continuous portion of the environment;

defining a resolution of the first neural representation corresponding to the at least one first set of data cells based at least in part on a range of a corresponding set of data readings and on a number of data cells in the at least one first set of data cells;

determining a scale of the first neural representation corresponding to the at least one first set of data cells based at least in part on the range of the corresponding set of data readings; and

generating, with the processor, a set of graph connections among the at least one first set of data cells, the set of graph connections providing continuity between representations of the respective data points.

13. The method of claim 1, further comprising:

adjusting data readings received from the at least one sensor based at least in part on the plurality of aggregated data readings.

14. A system for estimating at least one position of a robot moving through an environment, the system comprising:

an interface to receive first sensor data from a first sensor and second sensor data from a second sensor, the first sensor data and the second sensor data representing measurements of the environment with respect to the robot;

a memory, operably coupled to the interface, to store the first sensor data in a first neural representation of possible readings from the first sensor and to store the second sensor data in a second neural representation of possible readings from the second sensor; and

a path integration module, operably coupled to the memory, to;

(i) apply a first weight to the first sensor data in the first neural representation, the first weight representing a first measurement precision of the first sensor;

(ii) apply a second weight to the second sensor data in the second neural representation, the second weight representing a first measurement precision of the first sensor;

(iii) combine the first sensor data in the first neural representation with the second sensor data in the second neural representation so as to produce aggregated sensor data in an aggregated neural representation of the environment; and

(iv) estimate a position of the robot with respect to the environment based at least in part on the aggregated neural representation of the environment.

15. The system of claim 14, further comprising:

a map module, operably coupled to the path integration module, to generate a plurality of maps of the environment based on the position of the robot estimated by the path integration module.

16. The system of claim 15, further comprising:

a motivation and goal selection module, operably coupled to the path integration module, to implement a reward drive for the moving the robot and to generate a goal location for the robot based at least in part on the reward drive and the position of the robot estimated by the path integration module.

17. The system of claim 16, further comprising:

a path planning module, operably coupled to the path integration module, the map module, and the motivation and goal selection module, to generate a motor command for the robot based at least in part on the position of the robot estimated by the path integration module, the plurality of maps of the environment generated by the map module, and the goal location generated by the motivation and goal selection module.

18. The system of claim 17, wherein the robot is configured to perform an action based on the motor command generated by path planning module.

19. The system of claim 16, wherein the reward drive comprises at least one of a curiosity drive, a consumable resource drive, and an external reward drive.

20. The system of claim 19, wherein the curiosity drive is configured to set the goal location for the robot in an unfamiliar area of the environment.

21. The system of claim 19, wherein the resource drive is configured to monitor a consumable resource of the robot and to instruct the robot to seek replenishment for the consumable resource when the consumable resource falls below a predetermined level.

22. The system of claim 19, wherein the external reward drive is configured to set the goal location in an area of the environment area where the robot had previously found an external reward.

23. The system of claim 19, wherein the goal location is based at least in part on a location of a reward object selected by the external reward dive, a location of a replenishment object selected by the resource drive, and a location selected by curiosity drive.

24. A method of providing an estimate of at least one of a position of a robot and an orientation of the robot, the method comprising:

(A) receiving a vestibular signal representative of the at least one of the orientation of the robot and the position of the robot;

(B) receiving a motor outflow signal representative of commanded motion of the robot;

(C) receiving an optical flow signal representative of the at least of the orientation of the robot and the position of the robot with respect to an environment of the robot;

(D) weighting the vestibular signal received in (A), the motor outflow signal received in (B), and the optical flow signal received in (C) based on a type of the motion of the robot;

(E) combining the vestibular signal, motor outflow signal, and optical flow signal weighted in (D) to produce a localization signal; and

(F) generating, via at least one processor, the estimate of the at least one of the position of the robot and the orientation of the robot based on the localization signal produced in (E).

25. The method of claim 24, wherein (A) further comprises receiving the vestibular signal from at least one of a gyroscope and an accelerometer.

26. The method of claim 24, wherein (B) further comprises receiving the motor outflow signal from a controller operably coupled to a motor of the robot.

27. The method of claim 24, wherein (C) further comprises:

(C1) receiving data from at least one of a camera, an infrared sensor, an ultrasonic sensor, a laser rangefinder, and a radio-frequency antenna; and

(C2) deriving the optical flow signal from the data received in (C1).

28. The method of claim 24, wherein (C) further comprises spatiotemporally filtering the optical flow signal.

29. The method of claim 24, wherein (D) further comprises at least one of:

(D1) determining rotation along at least one axis; and

(D2) determining translation along at least one axis.

30. The method of claim 24, wherein (D) further comprises selecting at least one weights based at least in part on at least one of a speed of the robot, a velocity of the robot, an acceleration of the robot, and a trajectory of the robot.

31. The method of claim 24, wherein the robot comprises an autonomous vehicle.