Method and apparatus for controlling a robotic device via wearable sensors

• US 9,221,170 B2
• Filed: 06/13/2013
• Issued: 12/29/2015
• Est. Priority Date: 06/13/2013
• Status: Expired due to Fees

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Barajas, Leandro G., Lee, Jinhan
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): MOYER, DALE S
• Application Number:
  US13/916,741
• Publication Number:
  US 20140371906A1
• Time in Patent Office: 929 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 13/02 Hand grip control means handles or pedals for crane control B66C13/56; for measuring the force applied to control members G01L5/22; hand-held casings for switching devices, e.g. joy-sticks H01H9/0214
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: Y10S 901/02 Arm motion controller

Abstract

A method of training a robotic device to perform predetermined movements utilizing wearable sensors worn by a user. A position and movement of the user is sensed in response to signals from the wearable sensors. The wearable sensors include at least one six-degree of freedom accelerometer. Gestures are recognized by a gesture recognizer device in response to the signals. A position, orientation, and velocity of the user are estimated by a position and orientation processing unit. The sensed gesture inputs and the estimated position, orientation, and velocity inputs of the wearable sensors are received within a service requester device. The gestures, orientations, positions, and velocities are converted into predefined actions. Controls signals are output from the service requester device to a robot controller for executing the predefined actions.

Claims

1. A method of training a robotic device to perform predetermined movements utilizing wearable sensors worn by a user, the method comprising the steps of:

sensing a position and movement of the user in response to signals from the wearable sensors, the wearable sensors including at least one six-degree of freedom accelerometer;

recognizing gestures by a gesture recognizer device in response to the signals;

estimating a position, orientation, and velocity of the user by a position and orientation processing unit;

receiving the recognized gestures and the estimated position, orientation, and velocity of the wearable sensors within a service requester device;

converting the gestures, orientations, positions, and velocities into predefined actions; and

outputting controls signals from the service requester device to a robot controller for executing the predefined actions;

wherein a position, velocity and orientation of the wearable sensor is updated utilizing delayed integration, wherein transitioning the robotic device from a first position to a second position is delayed until the wearable sensors reach the second position.

2. The method of claim 1 wherein transition states between the first position and the second position are identified when utilizing delayed integration, the transition states include a null state, static state, and a motion state as detected by a control state machine.

3. The method of claim 2 wherein two static states are determined prior to a sensed movement and after the sensed movement of the wearable sensor.

4. The method of claim 3 wherein movement of the wearable sensors is detected based on a magnitude, a differential, and a variance of the wearable sensors, wherein the magnitude represents a distance that the wearable sensors moves, wherein the differential represents a speed of change in the movement, and wherein the variance indicates whether the wearable sensors is moving.

5. The method of claim 4 wherein the delayed integration utilizes gravity interpolation of the wearable sensors when transitioning the robotic device from the first position to the second position.

6. The method of claim 5 wherein the position and orientation of the wearable sensors after each integrated delay movement is corrected to compensate for errors.

7. The method of claim 5 wherein the gravity acceleration of the wearable sensors during the movement is interpolated utilizing quaternion averaging.

8. The method of claim 1 wherein the estimated position is identified by three linear coordinates and the orientation is represented by three angular parameters.

9. The method of claim 1 wherein the predetermined movements include a start position, an interim position, and a final position.

10. A human machine interface system for training a robotic device by a human operator, the system comprising:

a robot controller for controlling movement of a robotic device;

a wearable sensing device including at least one six-degree of freedom accelerometer that senses position and orientation of the wearable sensing device, a movement of the wearable sensing device being used to control a movement of the robotic device via the robotic controller;

a service provider device in communication with the wearable sensing device for receiving position and orientation data from the wearable sensing device, the service provider device including a gesture recognition device and a position and orientation processing device, the gesture recognition device identifying gestures generated by the wearable sensors, the position and orientation processing device estimating position, orientation, and velocity of the wearable sensors;

a service requester device including a control signal generator and a control state machine, the control signal generator integrating data received from the gesture recognition device and from the position and orientation processing device for generating control signals and providing the control signals to the robotic device, the control state machine identifying transitions states that include null states, static states, and motion states;

wherein the robotic device is trained to perform a respective operation by the identifying a sequence of gestures, orientations, positions, and velocities as sensed by the wearable sensing device, wherein the sequence of gestures, orientations, positions, and velocities are converted to predefined actions and executed via the robot controller.

11. The system of claim 10 wherein the gesture recognizer is a state space vector classifier.

12. The system of claim 11 wherein the gesture recognizer recognizes dynamic gestures.

13. The system of claim 12 wherein the gesture recognizer recognizes static gestures.

14. The system of claim 13 wherein the wearable sensing device includes an accelerometer glove, wherein a six-degree of freedom accelerometer disposed on each finger of the glove and the six-degree of freedom accelerometer is disposed on a palm of the glove.

15. A method of training a robotic device to perform predetermined movements utilizing wearable sensors worn by a user, the method comprising the steps of:

sensing a position and movement of the user in response to signals from the wearable sensors, the wearable sensors including at least one six-degree of freedom accelerometer;

recognizing gestures by a gesture recognizer device in response to the signals, the gesture recognizer being a state space vector classifier;

estimating a position, orientation, and velocity of the user by a position and orientation processing unit;

receiving the recognized gestures and the estimated position, orientation, and velocity of the wearable sensors within a service requester device;

converting the gestures, orientations, positions, and velocities into predefined actions; and

outputting controls signals from the service requester device to a robot controller for executing the predefined actions.

16. The method of claim 15 wherein the gesture recognizer recognizes dynamic gestures based on an active movement of the wearable sensors between a first location and second location.

17. The method of claim 16 wherein the gesture recognizer recognizes static gestures of a static position and orientation of the wearable sensors.

18. A method of training a robotic device to perform predetermined movements utilizing wearable sensors worn by a user, the method comprising the steps of:

sensing a position and movement of the user in response to signals from the wearable sensors, the wearable sensors including at least one six-degree of freedom accelerometer;

recognizing gestures by a gesture recognizer device in response to the signals;

estimating a position, orientation, and velocity of the user by a position and orientation processing unit;

receiving the recognized gestures and the estimated position, orientation, and velocity of the wearable sensors within a service requester device;

converting the gestures, orientations, positions, and velocities into predefined actions; and

outputting controls signals from the service requester device to a robot controller for executing the predefined actions;

wherein a position, velocity, and orientation of the wearable sensors is updated utilizing instantaneous integration, wherein transitioning the robotic device from a first training position to a second training position is instantaneously moved along a path of transition as the wearable sensors transitions from the first position to the second position.

19. The method of claim 18 wherein a sensor calibration for instantaneous integration estimates three gain factors and three biases for each six-degree of freedom accelerometer.