MACHINE BASED SIGN LANGUAGE INTERPRETER

• US 20110301934A1
• Filed: 06/04/2010
• Published: 12/08/2011
• Est. Priority Date: 06/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tardif, John
• Granted Patent: US 8,751,215 B2
• Time in Patent Office: Days
• US Class Current: 704/1
• CPC Class Codes: E05F 15/00 Power-operated mechanisms for wings motor-operated accessories in locks for completing closing or initiating opening of a wing E05B17/00
• CPC Class Codes: E05Y 2400/852 Sensors
• CPC Class Codes: E05Y 2400/86 by hand
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

A computer implemented method for performing sign language translation based on movements of a user is provided. A capture device detects motions defining gestures and detected gestures are matched to signs. Successive signs are detected and compared to a grammar library to determine whether the signs assigned to gestures make sense relative to each other and to a grammar context. Each sign may be compared to previous and successive signs to determine whether the signs make sense relative to each other. The signs may further be compared to user demographic information and a contextual database to verify the accuracy of the translation. An output of the match between the movements and the sign is provided.

Claims

1. A computer implemented method for interpreting sign language, comprising:

capturing a scene using a capture device, the scene including a human target;

tracking movements of the human target in the scene;

detecting one or more gestures of the human target in the scene;

comparing the one or more gestures to a library of sign language signs;

determining a match between the one or more gestures and one or more sign; and

generating an output indicating a visual translation of the one or more signs.

2. The computer implemented method of claim 1 wherein the step of detecting includes detecting gestures using an RGB camera and a depth camera.

3. The computer implemented method of claim 1 wherein the step of detecting includes detecting at least a first and a second adjacent gestures, and the determining includes determining a match of the first and second adjacent gestures with a first sign and a second sign, respectively.

4. The computer implemented method of claim 3 further including comparing the first and second signs relative to each other to determine the accuracy of the match between the first sign and the first gesture.

5. The computer implemented method of claim 4 wherein the step of detecting further includes detecting a third sign after the second sign and further including comparing the first sign and the third sign to the second sign to determine the accuracy of the second sign to the second gesture.

6. The computer implemented method of claim 5 wherein each step of comparing includes determining whether each sign makes sense in a linguistic context.

7. The computer implemented method of claim 5 further including the step of acquiring user demographic information and further including comparing each match with the user demographic information to verify the accuracy of the match.

8. The computer implemented method of claim 1 wherein the step of determining a match includes assigning a probability weight indicating the strength of the match between the gesture and the sign.

9. The system of claim 8 wherein the step of comparing includes adjusting the probability weight of each detected sign based on acquired user information and grammatical information, and comparing each probability weight against other signs likely to be assigned to a detected gesture.

10. A computer readable medium including instructions for programming a processing device to perform a series of steps, comprising:

capturing a scene using a capture device, the scene including a human target;

tracking movements of the human target in the scene;

detecting a plurality of gestures of the human target in the scene;

assigning a first sign to a detected gesture;

assigning a second sign to an adjacent detected gesture;

comparing the first and the second sign to verify the accuracy of the second sign; and

generating an output indicating a visual translation of the first and second sign.

11. The computer readable medium of claim 10 wherein the step of detecting further includes detecting a third sign after the second sign and further including comparing the first sign and the third sign to the second sign to determine the accuracy of the second sign to the adjacent detected gesture.

12. The computer implemented method of claim 11 wherein the step of comparing the first sign and the third sign to the second sign includes determining whether each sign makes sense in a linguistic context.

13. The computer implemented method of claim 12 further including instructions for performing a step of acquiring user demographic information and comparing each match with the user demographic information to verify the accuracy of the match.

14. The computer implemented method of claim 10 each step of assigning includes determining a match between a detected gesture and a sign, and further includes assigning a probability weight to each sign indicating the strength of the match between the gesture and the sign.

15. The system of claim 14 further including the step of acquiring user information and wherein the step of comparing the first and third sign includes comparing each match with the user demographic information to verify the accuracy of the match.

16. The system of claim 14 wherein the step of comparing includes adjusting the probability weight of the second sign based on the user information and the grammer database and comparing the probability weight against other signs likely to be assigned to a detected gesture.

17. An image capture system including:

a capture device including an RGB sensor and a depth sensor;

a host device, the host device providing a video output, the host device including a processor and instructions for programming the processor to perform a method comprising;

tracking movements of the human target in a scene acquired by the capture device,

detecting one or more gestures of the human target in the scene;

comparing the one or more gestures to a library of sign language signs and assigning a first sign to a first detected gesture, assigning a second sign to an a second detected gesture, and assigning a third sign after the second sign to a third detected gesture,

comparing the first sign and the third sign to the second sign to determine the accuracy of the second sign to the first and third signs; and

generating an output reflecting a meaning of the first, second and third signs.

18. The system of claim 17 wherein the step of comparing the first sign and the third sign to the second sign includes comparing each sign against a grammatical database to determine a probability that the signs appear adjacent to each other.

19. The system of claim 18 further including the step of acquiring user information and wherein the step of comparing the first and third sign includes comparing each match with the user demographic information to verify the accuracy of the match.

20. The system of claim 19 wherein the step of comparing includes assigning a probability weight indicating a strength of the match between the gesture and the sign.