Data collection from living subjects and controlling an autonomous robot using the data

• US 9,676,098 B2
• Filed: 07/31/2015
• Issued: 06/13/2017
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: Heinz Hemken
• Original Assignee: Heinz Hemken
• Inventors: Hemken, Heinz
• Primary Examiner(s): Caputo, Lisa
• Assistant Examiner(s): COTEY, PHILIP L
• Application Number:
  US14/815,624
• Publication Number:
  US 20170028551A1
• Time in Patent Office: 683 Days
• Field of Search: 73488, 73491
• CPC Class Codes: B25J 11/0085 Cleaning
• CPC Class Codes: B25J 11/009 Nursing, e.g. carrying sick persons, pushing wheelchairs, distributing drugs
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: G05B 2219/39254 Behaviour controller, robot have feelings, learns behaviour
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

Using various embodiments, methods, systems, and apparatuses are disclosed for capturing the behavior of living subjects, where the data can be processed by a machine-learning algorithm to control an autonomous robot. In one embodiment, using the data captured by a user performing everyday operations, the autonomous robot can behave in a substantially similar manner as the living subject did in the environment where the behavior capture sessions of the living subject were carried out. Further, once the autonomous robot is provided with the processed data, the actions of the robot can iteratively be stored and processed by the machine learning algorithm to further refine the robot'"'"'s movements performing the everyday operations. In another embodiment, behavior recorded from the living subject can be processed such that the autonomous robot is able to speak and understand spoken speech similar to that which occurred most frequently in the behavior capture sessions.

Claims

1. A sensor apparatus for periodically gathering data from a living subject for use by an autonomous robot, the sensor apparatus comprising:

a covering material configured to cover at least a portion of the living subject, wherein the covering material includes a garment covering the entire body of the living subject, headgear, footwear, and gloves;

one or more first sensors coupled to the covering material, wherein the one or more first sensors are at least one of an accelerometer, bend sensor, stretch sensor, pressure sensor, or proximity sensor, wherein the one or more first sensors can measure movements, bending, rotations, or angular displacement of the entire body of the living subject including arms, legs, joints, and extremities of the living subject in a controlled environment; and

a data computing device connected to the one or more sensors, wherein the data computing device can periodically capture and save the data generated from the one or more first sensors in a data file on a storage medium, wherein the data can be retrieved from the storage medium and processed by another computing device for use by an autonomous robot in a manner such that the autonomous robot exhibits substantially the same movement or action, as exhibited by the living subject when the data was being captured, and wherein the another computing device can generate robot control signals for smooth operation performed by the autonomous robot, the another computing device configured to;

process a first set of n consecutive time intervals of sensor data to compute a first time interval of robot control signals, wherein the first set of n consecutive time intervals of sensor data includes sensor data of i to n consecutive time intervals, and

process a second set of n consecutive time intervals of sensor data to compute a second time interval of robot control signals, wherein the second set of n consecutive time intervals of sensor data includes sensor data of i+1 to n+1 consecutive time intervals,

wherein i and n represent first and last consecutive time intervals, respectively, of the first set of n consecutive time intervals, and wherein i+1 and n+1 represent first and last consecutive time intervals, respectively, of the second set of n consecutive time intervals.

2. The sensor apparatus of claim 1, wherein the data generated by the first sensors include measurements of motions, movements, change in posture, change in position, of the living subject or the immediate environment of the living subject, in the controlled environment.

3. The sensor apparatus of claim 1, further comprising:

one or more second sensors coupled to the covering material, wherein the one or more second sensors are video cameras that can provide a plurality of views of the surroundings around the living subject in the controlled environment.

4. The sensor apparatus of claim 3, wherein the data generated by the one or more second sensors includes data related to gestures, grasping, gait, or other bodily dynamics and multi-media recordings of the living subject and the immediate environment of the living subject.

5. The sensor apparatus of claim 1, further comprising:

one or more third sensors coupled to the covering material, wherein the one or more third sensors can capture auditory sounds signals transmitted from or around the living subject in the controlled environment, and wherein the data generated by the one or more third sensors includes data related to vocalizations of the living subject and the immediate environment of the living subject.

6. The sensor apparatus of claim 1, further comprising:

one or more fourth sensors coupled to the covering material, wherein the one or more fourth sensors are at least one of a temperature sensor, humidity sensor, barometric pressure sensor, radioactivity detector, magnetic field detector, electromagnetic radiation detector, or airborne chemical detectors from or around the living subject in the controlled environment, and wherein the data generated by the one or more fourth sensors includes data related to the environment around the living subject.

7. The sensor apparatus of claim 1, wherein, the data computing device, at each periodic capture of data, uniquely tags the data generated from the one or more sensors with an identification code, each tag identifying the sensor from which the data was generated.

8. The sensor apparatus of claim 7, wherein the data generated at each periodic capture is saved in a track or channel in the data file.

9. A method for periodically gathering data from a living subject using a sensor apparatus to be used by an autonomous robot, the method comprising:

detecting, by a data computing device of the sensor apparatus, a signal transmitted by one or more first sensors of the sensor apparatus, wherein the one or more first sensors are coupled to a covering material of the sensor apparatus, the covering material configured to cover at least a portion of the living subject, wherein the covering material includes a garment covering the entire body of the living subject, headgear, footwear, and gloves;

based on the detection, receiving data from the one or more first sensors, wherein the one or more first sensors are at least one of an accelerometer, bend sensor, stretch sensor, pressure sensor, or proximity sensor, and wherein the one or more first sensors can measure movements, bending, rotations, or angular displacement of the entire body of the living subject including arms, legs, joints, and extremities of the living subject in a controlled environment; and

storing the data generated by the one or more first sensors, wherein the data computing device can periodically capture and save the data generated from the one or more first sensors in a data file on a storage medium, wherein the data can be retrieved from the storage medium and processed by another computing device for use by an autonomous robot in a manner such that the autonomous robot exhibits substantially the same movement or action exhibited by the living subject when the data was being captured, and wherein the another computing device can generate robot control signals for generating smooth operations performed by the autonomous robot by;

processing a first set of n consecutive time intervals of sensor data to compute a first time interval of robot control signals, wherein the first set of n consecutive time intervals of sensor data includes sensor data of i to n consecutive time intervals, and

processing a second set of n consecutive time intervals of sensor data to compute a second time interval of robot control signals, wherein the second set of n consecutive time intervals of sensor data includes sensor data of i+1 to n+1 consecutive time intervals,

wherein i and n represent first and last consecutive time intervals, respectively, of the first set of n consecutive time intervals, and wherein i+1 and n+1 represent first and last consecutive time intervals, respectively, of the second set of n consecutive time intervals.

10. The method of claim 9, wherein the data generated by the one or more first sensors include measurements of motions, movements, change in posture, change in position, of the living subject or the immediate environment of the living subject, in the controlled environment.

11. The method of claim 9, further comprising:

receiving data from the one or more second sensors, wherein the one or more second sensors are video cameras that can provide a plurality of views of the surroundings around the living subject in the controlled environment, and wherein the data generated by the one or more second sensors includes data related to gestures, grasping, gait, or other bodily dynamics and multi-media recordings of the living subject and the immediate environment of the living subject.

12. The method of claim 9, further comprising:

receiving data from one or more third sensors coupled to the covering material, wherein the one or more third sensors can capture auditory sounds signals transmitted from or around the living subject in the controlled environment, and wherein the data generated by the one or more third sensors includes data related to vocalizations of the living subject and the immediate environment of the living subject.

13. The method of claim 9, further comprising:

receiving data from one or more fourth sensors coupled to the covering material, wherein the one or more fourth sensors are at least one of a temperature sensor, humidity sensor, barometric pressure sensor, radioactivity detector, magnetic field detector, electromagnetic radiation detector, or airborne chemical detectors from or around the living subject in the controlled environment, and wherein the data generated by the one or more fourth sensors includes data related to the environment around the living subject.

14. The method of claim 9, wherein, the data computing device, at each periodic capture of data, uniquely tags the data generated from the one or more first sensors with an identification code, each tag identifying the sensor from which the data was generated, and wherein the data generated at each periodic capture is saved in a track or channel in the data file.

15. A non-transitory computer readable medium comprising instructions, which when executed by a processing system including one or more processors, executes a method for periodically gathering data from a living subject using a sensor apparatus to be used by an autonomous robot, the method comprising:

detecting a signal transmitted by one or more first sensors of the sensor apparatus, wherein the one or more first sensors are coupled to a covering material of the sensor apparatus, the covering material configured to cover at least a portion of the living subject, wherein the covering material includes a garment covering the entire body of the living subject, headgear, footwear, and gloves;

based on the detection, receiving data from the one or more first sensors, wherein the one or more first sensors are at least one of an accelerometer, bend sensor, stretch sensor, pressure sensor, or proximity sensor, and wherein the one or more first sensors can measure movements, bending, rotations, or angular displacement of the entire body of the living subject arms, legs, joints, and extremities of the living subject in a controlled environment; and

storing the data generated by the one or more first sensors, wherein the data computing device can periodically capture and save the data generated from the one or more first sensors in a data file on a storage medium, wherein the data can be retrieved from the storage medium and processed by another computing device for use by an autonomous robot in a manner such that the autonomous robot exhibits substantially the same movement or action exhibited by the living subject when the data was being captured, and wherein the another computing device can generate robot control signals for generating smooth operations performed by the autonomous robot by;

processing a first set of n consecutive time intervals of sensor data to compute a first time interval of robot control signals, wherein the first set of n consecutive time intervals of sensor data includes sensor data of i to n consecutive time intervals, and

processing a second set of n consecutive time intervals of sensor data to compute a second time interval of robot control signals, wherein the second set of n consecutive time intervals of sensor data includes sensor data of i+1 to n+1 consecutive time intervals,

wherein i and n represent first and last consecutive time intervals, respectively, of the first set of n consecutive time intervals, and wherein i+1 and n+1 represent first and last consecutive time intervals, respectively, of the second set of n consecutive time intervals.

16. The non-transitory computer readable medium of claim 15, wherein the data generated by the first sensors include measurements of motions, movements, change in posture, change in position, of the living subject or the immediate environment of the living subject, in the controlled environment.

17. The non-transitory computer readable medium of claim 15, further comprising:

receiving data from one or more second sensors coupled to the covering material, wherein the one or more second sensors are video cameras that can provide a plurality of views of the surroundings around the living subject in the controlled environment, and wherein the data generated by the one or more second sensors includes data related to gestures, grasping, gait, or other bodily dynamics and multi-media recordings of the living subject and the immediate environment of the living subject.

18. The non-transitory computer readable medium of claim 15, further comprising:

receiving data from one or more third sensors coupled to the covering material, wherein the one or more third sensors can capture auditory sounds signals transmitted from or around the living subject in the controlled environment, and wherein the data generated by the one or more third sensors includes data related to vocalizations of the living subject and the immediate environment of the living subject.

19. The non-transitory computer readable medium of claim 15, further comprising:

receiving data from one or more fourth sensors coupled to the covering material, wherein the one or more fourth sensors are at least one of a temperature sensor, humidity sensor, barometric pressure sensor, radioactivity detector, magnetic field detector, electromagnetic radiation detector, or airborne chemical detectors from or around the living subject in the controlled environment, and wherein the data generated by the one or more fourth sensors includes data related to the environment around the living subject.

20. The non-transitory computer readable medium of claim 15, wherein, the data computing device, at each periodic capture of data, uniquely tags the data generated from the one or more sensors with an identification code, each tag identifying the sensor from which the data was generated, and wherein the data generated at each periodic capture is saved in a track or channel in the data file.