Data entry system with drawing recognition

• US 10,705,720 B2
• Filed: 01/28/2019
• Issued: 07/07/2020
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tran, To
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US16/260,036
• Publication Number:
  US 20190155480A1
• Time in Patent Office: 526 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

An electronic device is described which has a memory storing an image for display to a user. The device has a shape recognizer which receives drawing input as a result of the user drawing at a location of the displayed image; and a semantic segmentation component which carries out image analysis of the image to compute one or more object classes of objects depicted at the location. The shape recognizer is configured to recognize the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the image. The processor is configured to offer at least one graphical item of the computed graphical item class for input to the electronic device.

Claims

1. An electronic device comprising:

a memory storing content for display to a user;

a shape recognition component which receives drawing input as a result of the user drawing at a location of the displayed content;

a semantic segmentation component configured to carry out analysis of the content to compute one or more object classes of objects depicted at the location in the content; and

wherein

the shape recognition component is configured to recognize the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the content; and

a processor configured to offer at least one graphical item of the computed graphical item class for input to the electronic device.

2. The electronic device of claim 1 wherein the location in the content comprises a bounding box computed to bound the drawing input.

3. The electronic device of claim 1 wherein the location in the content comprises coordinates of the drawing input in the content.

4. The electronic device of claim 1 wherein the processor is configured such that, when user input is received selecting the offered graphical item, the offered graphical item is inserted into the content at the location.

5. The electronic device of claim 1 wherein the processor is configured such that, when user input is received selecting the offered graphical item, the offered graphical item is inserted into the content at the location so as to replace the drawing input.

6. The electronic device of claim 1 wherein the processor is configured to retrieve the at least one graphical item of the computed graphical item class by sending an information retrieval request to a graphical item store.

7. The electronic device of claim 1 wherein the semantic segmentation component is a trained machine learning system selected from one or more of:

a random decision forest, a neural network, a support vector machine.

8. The electronic device of claim 1 wherein the shape recognition component is a trained machine learning system selected from one or more of:

a random decision forest, a neural network, a support vector machine.

9. The electronic device of claim 8 wherein the shape recognition component has been trained using training data comprising pairs, each pair comprising:

semantic segmentation data, location data, drawing input and a shape of a known class from a plurality of specified classes of shape.

10. The electronic device of claim 1 wherein the content comprises an image that is a digital photograph or video.

11. The electronic device of claim 1 being a wearable computer.

12. The electronic device of claim 1 being a hand held computing device.

13. A computer-implemented method of data entry comprising:

storing at a memory content for display to a user;

receiving drawing input as a result of the user drawing at a location of the displayed content;

computing analysis of the content to compute one or more object classes of objects depicted at the location in the content;

recognizing the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the content; and

offering at least one graphical item of the computed graphical item class for input to the electronic device.

14. The method of claim 13 comprising computing a bounding box of the drawing input as the location in the content.

15. The method of claim 13 comprising obtaining coordinates of the drawing input in the content as the location.

16. The method of claim 13 comprising replacing the drawing input on the content by the offered at least one graphical item in response to user input selecting the at least one graphical item.

17. The method of claim 16 comprising adjusting a color balance of the graphical item to match a color balance of the content before the replacing of the drawing input.

18. The method of claim 13 comprising retrieve the at least one graphical item of the computed graphical item class by sending an information retrieval request to a graphical item store.

19. The method of claim 13 comprising recognizing the drawing input using a shape recognizer comprising a machine learning component, and training the shape recognizer using pairs of data, each pair comprising semantic segmentation data, location data, drawing input and a shape of a known class from a plurality of specified classes of shape.

20. One or more computer storage media with device-executable instructions that, when executed by a computing system, direct the computing system to perform operations comprising:

storing at a memory content for display to a user;

receiving drawing input as a result of the user drawing at a location of the displayed content;

computing analysis of the content to compute one or more object classes of objects depicted at the location in the content;

recognizing the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the content; and

offering at least one graphical item of the computed graphical item class for input to the electronic device.