Stroke extraction in free space

• US 10,169,670 B2
• Filed: 11/30/2015
• Issued: 01/01/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dow, Eli M., Fitzsimmons, Thomas D., Garrett, Tynan J., Metruck, Emily M.
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US14/953,713
• Publication Number:
  US 20170154230A1
• Time in Patent Office: 1,128 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 40/126 Character encoding
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/2276 with probabilistic networks, e.g. hidden Markov models
• CPC Class Codes: G06V 30/36 Matching; Classification

Abstract

An approach for stroke extraction in free space utilizing a paired ring device is provided. The approach receives one or more images transmitted from the paired ring device, wherein the one or more images are transcribed sequentially from data related to one or more movements recorded by the paired ring device, and wherein the one or more images include one or more of a plurality of vector points, a plurality of coordinates, and a plurality of dots interconnected by a plurality of lines. The approach inputs the one or more images into a character training model. The approach maps the one or more images into one or more characters. The approach transcribes the one or more characters into a digital document.

Claims

1. A method for stroke extraction in free space utilizing a paired ring device and a paired display device including a video recording module connected to the paired display device, the method comprising:

receiving, by one or more computer processors, a video from the video recording module of a free space environment;

determining, by one or more computer processors, one or more bounded surface areas within the video of the free space environment, with each bounded surface area including at least two dimensions of space;

displaying, by one or more computer processors, on the paired display device, a modified free space environment including the determined one or more bounded surface areas;

receiving, by one or more computer processors, one or more images transmitted from the paired ring device, wherein the one or more images are transcribed sequentially from one or more language movements recorded by the paired ring device utilizing a magnitude value and a direction value of proper acceleration of the one or more language movements, and wherein the one or more images include one or more of a plurality of vector points, a plurality of coordinates, and a plurality of dots interconnected by a plurality of lines;

determining, by one or more processors, at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas;

responsive to determining that at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas, providing, by the one or more processors, haptic feedback to the paired ring device;

inputting, by one or more computer processors, the one or more images into a character training model;

mapping, by one or more computer processors, the one or more images into one or more characters in the character training model, wherein mapping includes determining whether the one or more images indicate at least one of the one or more characters in the character training model utilizing a pre-determined range of motion of the paired ring device during a time period of movement in free space, where the pre-determined range of motion creates a specific image; and

responsive to a determination that the one or more images indicate at least one of the one or more characters in the character training model, transcribing, by one or more computer processors, the at least one of the one or more characters into a digital document, wherein the digital document is a visual augmentation for displaying in a heads up display of the paired display device.

2. The method of claim 1, further comprises:

wherein the paired ring device includes an accelerometer and a gyroscope for monitoring the one or more language movements; and

wherein the one or more language movements include a change in a magnitude value and a direction value of proper acceleration of the paired ring device, an orientation of the paired ring device, a position of the paired ring device, a location of the paired ring device, a change in the orientation of the paired ring device, a change in the position of the paired ring device, and a change in the location of the paired ring device, in free space.

3. The method of claim 1, wherein inputting the one or more images into a character training model, further comprises:

wherein the character training model is a character recognition model specifically trained utilizing a neural network to recognize which of the one or more language movements map to which of the at least one of the one or more characters.

4. The method of claim 3, wherein the character training model is a character recognition model initially trained utilizing handwritten character recognition (ICR).

5. The method of claim 1, wherein determining whether the one or more images indicate at least one of the one or more characters in the character training model, further utilizing one or more changes in data recorded on the paired ring device, further comprises:

responsive to determining that the one or more images indicate at least one of the one or more characters in the character training model, accumulating, by one or more computer processors, each of the at least one of the one or more characters into one or more character streams, wherein the one or more character streams include at least one of one or more sentences, the at least one of the one or more characters, one or more paragraphs, and one or more keywords; and

associating, by one or more computer processors, the one or more character streams, based, at least in part, on character recognition results from the character training model, with a time stamp and identifiers to indicate a textual representation of at least one of the one or more characters in the character training model.

6. The method of claim 1, wherein transcribing the at least one of the one or more characters into a digital document, further comprises:

determining, by one or more computer processors, a representation of the at least one of the one or more characters as a written language or a gesture output derived from a series of strokes in free space; and

encoding, by one or more computer processors, the at least one of the one or more characters into a visual augmentation.

7. The method of claim 1, further comprises:

displaying, by one or more computer processors, the digital document on one or more paired devices utilizing an interactive user interface, wherein the interactive user interface provides the capability to view the at least one of the one or more characters in real time.

8. The method of claim 1 wherein the paired ring device is a wearable ring device for a human thumb and/or finger.

9. A computer program product for stroke extraction in free space utilizing a paired ring device and a paired display device including a video recording module connected to the paired display device, the computer program product comprising:

one or more computer readable storage mediums and program instructions stored on the one or more computer readable storage mediums, the program instructions comprising;

program instructions to receive a video from the video recording module of a free space environment,

program instructions to determine one or more bounded surface areas within the video of the free space environment, with each bounded surface area including at least two dimensions of space,

program instructions to display on the paired display device, a modified free space environment including the determined one or more bounded surface areas,

program instructions to receive one or more images transmitted from the paired ring device, wherein the one or more images are transcribed sequentially from one or more language movements recorded by the paired ring device utilizing a magnitude value and a direction value of proper acceleration of the one or more language movements, and wherein the one or more images include one or more of a plurality of vector points, a plurality of coordinates, and a plurality of dots interconnected by a plurality of lines,

program instructions to determine, by one or more processors, at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas,

responsive to determining that at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas, program instructions to provide, by the one or more processors, haptic feedback to the paired ring device,

program instructions to input the one or more images into a character training model,

program instructions to map the one or more images into one or more characters in the character training model, wherein mapping includes determining whether the one or more images indicate at least one of the one or more characters in the character training model utilizing a pre-determined range of motion of the paired ring device during a time period of movement in free space, where the pre-determined range of motion creates a specific image, and

responsive to a determination that the one or more images indicate at least one of the one or more characters in the character training model, program instructions to transcribe the at least one of the one or more characters into a digital document, wherein the digital document is a visual augmentation for displaying in a heads up display of the paired display device.

10. The computer program product of claim 9, further comprises:

wherein the paired ring device includes an accelerometer and a gyroscope for monitoring the one or more language movements; and

wherein the one or more language movements includes one or more of a change in a magnitude value and a direction value of proper acceleration of the paired ring device, an orientation of the paired ring device, a position of the paired ring device, a location of the paired ring device, a change in the orientation of the paired ring device, a change in the position of the paired ring device, and a change in the location of the paired ring device, in free space.

11. The computer program product of claim 9, wherein program instructions to input the one or more images into a character training model, further comprises:

wherein the character training model is a character recognition model specifically trained utilizing a neural network to recognize which of the one or more language movements map to which of the at least one of the one or more characters.

12. The computer program product of claim 11, wherein the character training model is a character recognition model initially trained utilizing handwritten character recognition (ICR).

13. The computer program product of claim 9, wherein program instructions to determine whether the one or more images indicate the at least one of the one or more characters in the character training model, further utilizing one or more changes in data recorded on the paired ring device, further comprises:

responsive to determining that the one or more images indicate the at least one of the one or more characters in the character training model, program instructions to accumulate each of the at least one of the one or more characters into one or more character streams, wherein the one or more character streams include at least one of one or more sentences, the at least one of the one or more characters, one or more paragraphs, and one or more keywords; and

program instructions to associate the one or more character streams, based, at least in part, on character recognition results from the character training model, with a time stamp and identifiers to indicate a textual representation of the at least one of the one or more characters in the character training model.

14. The computer program product of claim 9, wherein program instructions to transcribe the at least one of the one or more characters into a digital document, further comprises:

program instructions to determine a representation of the at least one of the one or more characters as a written language or a gesture output derived from a series of strokes in free space; and

program instructions to encode the at least one of the one or more characters into a visual augmentation.

15. The computer program product of claim 9, further comprises:

program instructions to display the digital document on one or more paired devices utilizing an interactive user interface, wherein the interactive user interface provides the capability to view the at least one of the one or more characters in real time.

16. The computer program product of claim 9 wherein the paired ring device is a wearable ring device for a human thumb and/or finger.

17. A computer system for stroke extraction in free space utilizing a paired ring device and a paired display device including a video recording module connected to the paired display device, the computer system comprising:

one or more computer processors;

one or more computer readable storage mediums;

program instructions stored on at least one of the one or more computer readable storage mediums for extraction by at least one of the one or more computer processors, the program instructions comprising;

program instructions to receive a video from the video recording module of a free space environment,

program instructions to determine one or more bounded surface areas within the video of the free space environment, with each bounded surface area including at least two dimensions of space,

program instructions to display on the paired display device, a modified free space environment including the determined one or more bounded surface areas,

program instructions to receive one or more images transmitted from the paired ring device, wherein the one or more images are transcribed sequentially from one or more language movements recorded by the paired ring device utilizing a magnitude value and a direction value of proper acceleration of the one or more language movements, and wherein the one or more images include one or more of a plurality of vector points, a plurality of coordinates, and a plurality of dots interconnected by a plurality of lines,

program instructions to determine, by one or more processors, at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas,

responsive to determining that at least one of the one or more language movements crossed at least one side of at least one bounded surface area of the one or more bounded surface areas, program instructions to provide, by the one or more processors, haptic feedback to the paired ring device,

program instructions to input the one or more images into a character training model,

program instructions to map the one or more images into one or more characters in the character training model, wherein mapping includes determining whether the one or more images indicate at least one of the one or more characters in the character training model utilizing a pre-determined range of motion of the paired ring device during a time period of movement in free space, where the pre-determined range of motion creates a specific image, and

responsive to a determination that the one or more images indicate at least one of the one or more characters in the character training model, program instructions to transcribe the at least one of the one or more characters into a digital document, wherein the digital document is a visual augmentation for displaying in a heads up display of the paired display device.

18. The computer system of claim 17, further comprises:

wherein the paired ring device includes an accelerometer and a gyroscope for monitoring the one or more language movements; and

wherein the one or more language movements includes one or more of a change in a magnitude value and a direction value of proper acceleration of the paired ring device, an orientation of the paired ring device, a position of the paired ring device, a location of the paired ring device, a change in the orientation of the paired ring device, a change in the position of the paired ring device, and a change in the location of the paired ring device, in free space.

19. The computer system of claim 17, wherein program instructions to input the one or more images into a character training model, further comprises:

wherein the character training model is a character recognition model specifically trained utilizing a neural network to recognize which of the one or more language movements map to which of the at least one of the one or more characters.

20. The computer system of claim 19, wherein the character training model is a character recognition model initially trained utilizing handwritten character recognition (ICR).