Data entry system with drawing recognition

• US 10,216,992 B2
• Filed: 11/21/2016
• Issued: 02/26/2019
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tran, To
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/358,051
• Publication Number:
  US 20180144189A1
• Time in Patent Office: 827 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

An electronic device is described which has a memory storing an image for display to a user. The device has a shape recognizer which receives drawing input as a result of the user drawing at a location of the displayed image; and a semantic segmentation component which carries out image analysis of the image to compute one or more object classes of objects depicted at the location. The shape recognizer is configured to recognize the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the image. The processor is configured to offer at least one graphical item of the computed graphical item class for input to the electronic device.

Claims

1. An electronic device comprising:

a memory storing an image for display to a user;

a shape recognition component which receives drawing input as a result of the user drawing at a location of the displayed image;

a semantic segmentation component configured to carry out image analysis of the image to compute one or more object classes of objects depicted at the location in the image; and

wherein

the shape recognition component is configured to recognize the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the image; and

a processor configured to offer at least one graphical item of the computed graphical item class for input to the electronic device.

2. The electronic device of claim 1 wherein the location in the image comprises a bounding box computed to bound the drawing input.

3. The electronic device of claim 1 wherein the location in the image comprises coordinates of the drawing input in the image.

4. The electronic device of claim 1 wherein the processor is configured such that, when user input is received selecting the offered graphical item, the offered graphical item is inserted into the image at the location.

5. The electronic device of claim 1 wherein the processor is configured such that, when user input is received selecting the offered graphical item, the offered graphical item is inserted into the image at the location so as to replace the drawing input.

6. The electronic device of claim 1 wherein the processor is configured to retrieve the at least one graphical item of the computed graphical item class by sending an information retrieval request to a graphical item store.

7. The electronic device of claim 1 wherein the semantic segmentation component is a trained machine learning system selected from one or more of:

a random decision forest, a neural network, a support vector machine.

8. The electronic device of claim 1 wherein the shape recognition component is a trained machine learning system selected from one or more of:

a random decision forest, a neural network, a support vector machine.

9. The electronic device of claim 8 wherein the shape recognition component has been trained using training data comprising pairs, each pair comprising:

semantic segmentation data, location data, drawing input and a shape of a known class from a plurality of specified classes of shape.

10. The electronic device of claim 1 wherein the image is a digital photograph or video.

11. The electronic device of claim 1 being a wearable computer.

12. The electronic device of claim 1 being a hand held computing device.

13. A computer-implemented method of data entry comprising:

storing at a memory an image for display to a user;

receiving drawing input as a result of the user drawing at a location of the displayed image;

computing image analysis of the image to compute one or more object classes of objects depicted at the location in the image;

recognizing the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the image; and

offering at least one graphical item of the computed graphical item class for input to the electronic device.

14. The method of claim 13 comprising computing a bounding box of the drawing input as the location in the image.

15. The method of claim 13 comprising obtaining coordinates of the drawing input in the image as the location.

16. The method of claim 13 comprising replacing the drawing input on the image by the offered at least one graphical item in response to user input selecting the at least one graphical item.

17. The method of claim 16 comprising adjusting a color balance of the graphical item to match a color balance of the image before the replacing of the drawing input.

18. The method of claim 13 comprising retrieve the at least one graphical item of the computed graphical item class by sending an information retrieval request to a graphical item store.

19. The method of claim 13 comprising recognizing the drawing input using a shape recognizer comprising a machine learning component, and training the shape recognizer using pairs of data, each pair comprising semantic segmentation data, location data, drawing input and a shape of a known class from a plurality of specified classes of shape.

20. One or more computer storage media with device-executable instructions that, when executed by a computing system, direct the computing system to perform operations comprising:

storing at a memory a video for display to a user;

receiving drawing input as a result of the user drawing at a location of the displayed video;

computing image analysis of the video to compute one or more object classes of objects depicted at the location in the video;

recognizing the drawing input as being in a computed graphical item class of a plurality of pre-specified graphical item classes by analyzing a shape of the drawing input and by using information about the computed object classes of the one or more objects depicted at the drawing input location in the video; and

offering at least one graphical item of the computed graphical item class for input to the electronic device.