Pictures from sketches

• US 9,528,847 B2
• Filed: 10/15/2012
• Issued: 12/27/2016
• Est. Priority Date: 10/15/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Albrecht, Brian
• Primary Examiner(s): AMIN, JWALANT B
• Application Number:
  US13/651,470
• Publication Number:
  US 20140108016A1
• Time in Patent Office: 1,534 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

A graphical sketch can be received, the sketch including one or more representations of text. A query can be automatically generated from the sketch. The generation of the query can include automatically recognizing the text and automatically representing the text in the query. The query can be run to identify a picture in response to the query, with the text describing one or more non-textual features of the picture. The picture can be returned, such as in response to the receipt of the graphical sketch.

Claims

1. A computer system comprising:

at least one processor; and

memory comprising instructions stored thereon that when executed by at least one processor cause at least one processor to perform acts comprising;

receiving a digital graphical sketch;

automatically generating a computer-readable query from the sketch, the automatic generating of the computer-readable query from the sketch comprising automatically performing text recognition on one or more sketched representations of text in the sketch;

identifying a graphical ambiguity in the sketch, with the graphical ambiguity being an ambiguity as to what type of item is depicted by one or more non-textual graphical elements of the sketch, and with identifying the graphical ambiguity comprising deriving a plurality of alternative meanings from the one or more non-textual graphical elements of the sketch;

identifying a meaning for the sketch from among the plurality of alternative meanings;

running the query to identify one or more digital pictures in response to the query; and

returning results of the query, the acts comprising limiting the returned results of the query to those corresponding to the identified meaning for the sketch, the returned results of the query comprising a picture that is representative of the graphical sketch and that corresponds to the identified meaning for the sketch.

2. The system of claim 1, wherein the picture does not include the text.

3. The system of claim 1, wherein the sketch comprises one or more graphical representations of one or more graphical features of the picture.

4. The system of claim 1, wherein automatically generating the query comprises automatically identifying a type of item represented by one or more graphical elements of the sketch and including one or more textual representations of the type of item in the query, wherein automatically identifying the type of item comprises performing pattern matching on the one or more graphical elements.

5. The system of claim 1, wherein automatically generating the query comprises automatically performing pattern matching on one or more graphical elements of the sketch.

6. The system of claim 1, wherein the picture is a photograph.

7. The system of claim 1, wherein the sketch comprises a color and the query comprises a textual representation of the color.

8. The system of claim 1, wherein the running of the query comprises identifying a plurality of pictures, wherein the one or more sketched representations of text in the sketch describe one or more non-textual features of each of the plurality of pictures, and wherein the acts further comprise returning the plurality of pictures.

9. The system of claim 8, wherein the acts further comprise:

receiving user input selecting one or more of the plurality of pictures; and

in response to the user input selecting the one or more of the plurality of pictures, inserting the one or more of the plurality of pictures into a document.

10. The system of claim 1, wherein the acts further comprise displaying a digital document, wherein receiving the sketch comprises receiving user input defining the sketch at the displayed document, and wherein the acts further comprise entering the picture from the returned results into the document.

11. The computer system of claim 1, wherein the one or more sketched representations of text in the sketch comprise a handwritten representation of text in the sketch and the performance of the text recognition comprises performing pattern matching on the handwritten representation of text in the sketch.

12. A computer-implemented method comprising:

receiving a digital graphical sketch;

identifying a graphical ambiguity in the sketch, with the graphical ambiguity being an ambiguity as to what type of item is depicted by one or more non-textual graphical elements of the sketch, and with identifying the ambiguity comprising deriving a plurality of alternative meanings from the one or more non-textual graphical elements of the sketch;

requesting user input to choose a meaning for the sketch, the request for user input identifying the plurality of alternative meanings;

receiving user input in response to the request for user input, the received user input choosing a meaning for the sketch;

sending a computer-readable request to a computer picture service, the request requesting that the picture service return at least one digital picture represented by the sketch;

receiving one or more digital pictures from the picture service in response to the computer-readable request; and

displaying the one or more pictures on a computer display, the method comprising limiting the displayed one or more pictures to those corresponding to the chosen meaning for the sketch, the one or more pictures that are displayed comprising a picture that is representative of the graphical sketch and that corresponds to the chosen meaning for the sketch.

13. The method of claim 12, wherein receiving the sketch comprises receiving user input defining the sketch at a displayed digital document, and wherein displaying the one or more pictures comprises displaying the picture in the document.

14. The method of claim 12, wherein the request to the picture service comprises an image of the sketch.

15. The method of claim 12, wherein the request to the picture service comprises a query generated from the sketch, the request to the picture service requesting that the picture service run the query.

16. The method of claim 12, wherein the sketch comprises one or more representations of text, and wherein the picture does not include the text.

17. The method of claim 12, wherein the sketch comprises one or more graphical representations of one or more graphical features of the picture.

18. The method of claim 12, wherein the service is a remote service.

19. The method of claim 12, wherein the method further comprises automatically performing text recognition on one or more sketched representations of text in the sketch, wherein the request comprises one or more textual representations of the one or more sketched representations of text, and wherein the one or more representations of the one or more sketched representations of text are derived from the text recognition.

20. One or more computer-readable storage media having computer-executable instructions embodied thereon that, when executed by at least one processor, cause at least one processor to perform acts comprising:

receiving user input defining a freehand digital graphical sketch on a computer display;

automatically generating a computer-readable query from the sketch, the automatic generating of the computer-readable query from the sketch comprising automatically performing text recognition on one or more sketched representations of text in the sketch;

identifying a graphical ambiguity in the sketch, with the graphical ambiguity being an ambiguity as to what type of item is depicted by one or more non-textual graphical elements of the sketch, and with identifying the ambiguity comprising deriving a plurality of alternative meanings from the one or more non-textual graphical elements of the sketch;

requesting user input to choose a meaning for the sketch, the request for user input identifying the plurality of alternative meanings;

receiving user input in response to the request for user input, the received user input choosing a meaning for the sketch;

running the query to identify one or more digital pictures in response to the query;

returning results in response to the query; and

displaying one or more pictures from the results on the computer display in response to the results being returned, the acts comprising limiting the displayed one or more pictures to those corresponding to the chosen meaning for the sketch, the one or more displayed pictures comprising a picture that is representative of the graphical sketch and that corresponds to the chosen meaning for the sketch.