Dynamic multi-sensor and multi-robot interface system

• US 10,179,407 B2
• Filed: 11/16/2015
• Issued: 01/15/2019
• Est. Priority Date: 11/16/2014
• Status: Active Grant

• Current Assignee: Robologics Limited
• Original Assignee: Robologics Limited
• Inventors: Benaim, Carlos
• Primary Examiner(s): Ilan, Ruth
• Application Number:
  US14/941,879
• Publication Number:
  US 20170136621A1
• Time in Patent Office: 1,156 Days
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/35503 Eye tracking associated with head mounted display to detect eye position
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/16 Cartesian, three degrees of freedom

Abstract

An adaptive learning interface system for end-users for controlling one or more machines or robots to perform a given task, combining identification of gaze patterns, EEG channel'"'"'s signal patterns, voice commands and/or touch commands. The output streams of these sensors are analyzed by the processing unit in order to detect one or more patterns that are translated into one or more commands to the robot, to the processing unit or to other devices. A pattern learning mechanism is implemented by keeping immediate history of outputs collected from those sensors, analyzing their individual behavior and analyzing time correlation between patterns recognized from each of the sensors. Prediction of patterns or combination of patterns is enabled by analyzing partial history of sensors'"'"' outputs. A method for defining a common coordinate system between robots and sensors in a given environment, and therefore dynamically calibrating these sensors and devices, is used to share characteristics and positions of each object detected on the scene.

Claims

1. A method for generating a common coordinate system between robotic devices and sensors in a given environment comprising:

providing at least one processing unit, a first sensor and a second sensor, and at least one robotic device;

collecting a sequence of images from said first sensor showing said second sensor and said robotic device;

analyzing said sequence of images to uniquely identify said second sensor and said robot device and their relative location and pose;

generating a set of conversion parameters for permuting said relative location to location data relative to at least one of said second sensor and said robotic device, wherein said second sensor is a gaze tracker.

2. The method according to claim 1, wherein said first sensor is a digital camera.

3. The method of claim 1, wherein said first sensor is a depth sensor.

4. The method according to claim 3, wherein collecting a sequence of images comprises collecting a depth mapping input stream, and wherein identifying said relative location and pose is performed dynamically on said depth map input stream.

5. The method according to claim 1, wherein determining the relative location and pose further comprises determining coordinates of additional objects detected in said sequence of images.

6. The method according to claim 1, wherein determining the relative location comprises determining coordinates of objects relative to said robotic device.

7. The method of claim 1, wherein said robotic device includes a gripper mounted on a robot end point.

8. The method of claim 7, wherein determining the relative location and pose of said robotic device comprises determining a relative location and pose of said gripper with respect to at least one of said first and said second sensor.