Surface computer system and method for integrating display of user interface with physical objects

• US 8,584,029 B1
• Filed: 05/23/2008
• Issued: 11/12/2013
• Est. Priority Date: 05/23/2008
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Gerde, Matthew E., Wolfe, Jeffrey M., Rainwater, Michael J.
• Primary Examiner(s): Sax, Steven
• Application Number:
  US12/126,382
• Time in Patent Office: 1,999 Days
• Field of Search: 715/771, 715/765, 715/702, 715/831, 359/17, 358/401, 358/408, 358/474, 358/486, 358/488, 358496-498
• US Class Current: 715/771
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Systems and methods for assisting user interaction with physical objects. A system may include an interactive display surface configured to display visual output and receive a physical object, where the object possesses encoded data including data other than data directly identifying the object. The system may include a computer-readable storage medium storing program instructions executable by the surface computer to implement software applications. In response to the display surface receiving the object, the surface computer may be configured to read the encoded data passively with respect to the object, without using an active electronic interface between the surface computer and the object, and to display a user interface for software applications dependent upon the passively read encoded data, such that one or more elements of the user interface are visually associated on the interactive display surface with the physical object.

Claims

1. A surface computer, comprising:

an interactive display surface configured to;

display visual output,

detect placement of a physical document on the interactive display surface; and

optically scan the physical document to obtain encoded data,

wherein the physical document possesses the encoded data, and

wherein the encoded data includes data other than data directly identifying the physical document; and

a computer-readable storage medium storing program instructions executable to implement one or more software applications;

wherein in response to the interactive display surface optically scanning the physical document, the surface computer is configured to;

determine, based on the encoded data, a document type from a plurality of document types; and

based on the encoded data and the document type, identify, from a plurality of software applications, a software application capable of functioning in concert with the encoded data, wherein the software application comprises a plurality of application data items; and

wherein, in response to the identifying, the interactive display surface is further configured to;

display a user interface for the software application;

receive user input via a portion of the physical document;

visibly distinguish, based on the user input and software application, the portion of the physical document at a position physically matching the portion of the physical document that received the user input;

select a first application data item of the plurality of application data items based on a relationship between the first application data item and the portion of the physical document; and

display the first application data item.

2. The surface computer as recited in claim 1, wherein optically scanning the physical document to obtain the encoded data comprises scanning optically data from one or more surfaces of the physical document.

3. The surface computer as recited in claim 1, wherein the physical document is a tax form, and wherein the encoded data comprises one selected from a group consisting of a geometric pattern of the tax form and a print pattern of the tax form.

4. The surface computer as recited in claim 1, wherein the physical document further possesses identifying data distinct from the encoded data.

5. The surface computer as recited in claim 1, wherein in response to the user input, the interactive display surface is further configured to display the user interface in a changed state, wherein one or more elements of the user interface in the changed state are dynamically generated in response to the user input.

6. The surface computer as recited in claim 5, wherein the user input comprises tactile input received via the interactive display surface.

7. The surface computer as recited in claim 1, wherein the physical document comprises a surface including a plurality of document data items.

8. The surface computer as recited in claim 7, wherein at least some of the document data items are human-readable.

9. The surface computer as recited in claim 7, wherein the user interface is configured to display a plurality of selectable application data items corresponding to the software application, and wherein, in response to a user'"'"'s selection of an application data item via the user interface, the interactive display surface is configured to visibly indicate a location on the surface of the physical document that includes a document data item corresponding to the application data item selected via the user interface.

10. The surface computer as recited in claim 7, wherein in response to a user selecting a particular area of the surface of the physical document, the interactive display surface is further configured to indicate to the user one or more particular application data items corresponding to the particular area of the surface of the physical document.

11. The surface computer as recited in claim 1, wherein the user interface is further displayed such that one or more elements of the user interface are visually associated with a portion of the encoded data located on the physical document.

12. The surface computer as recited in claim 1, wherein the interactive display surface is further configured to receive a plurality of physical documents including said physical document, and wherein at least one element of the user interface displayed by the interactive display surface is generated dependent upon a context, when the context includes one or more of a currently active application or one or more additional physical documents received by the interactive display surface.

13. A method, comprising:

detecting, by an interactive display surface, placement of a physical document on the interactive display surface;

optically scanning, by the interactive display surface, the physical document to obtain encoded data,

wherein the physical document possesses encoded data, and

wherein the encoded data includes data other than data directly identifying the physical document;

in response to the interactive display surface optically scanning the physical document;

determining, based on the encoded data, a document type from a plurality of document types; and

based on the encoded data and the document type, identifying, from a plurality of software applications, a software application capable of functioning in concert with the encoded data, wherein the software application comprises a plurality of application data items;

in response to the identifying;

displaying, by the interactive display surface, a user interface for the software application;

receiving user input via a portion of the physical document;

visibly distinguishing, based on the user input and software application, the portion of the physical document at a position physically matching the portion of the physical document that received the user input; and

selecting a first application data item of the plurality of application data items based on a relationship between the first application data item and the portion of the physical document; and

displaying the first application data item.

14. The method as recited in claim 13, wherein optically scanning the physical document to obtain the encoded data comprises scanning optically data from one or more surfaces of the physical document.

15. The method as recited in claim 13, wherein the physical document is a tax form, and wherein the encoded data comprises one selected from a group consisting of a geometric pattern of the tax form and a print pattern of the tax form.

16. The method as recited in claim 13, wherein the physical document further possesses identifying data distinct from the encoded data.

17. The method as recited in claim 13, further comprising:

in response to the user input, the interactive display surface displays the user interface in a changed state, wherein one or more elements of the user interface in the changed state are dynamically generated in response to the user input.

18. The method as recited in claim 17, wherein the user input comprises tactile input received via the interactive display surface.

19. The method as recited in claim 13, wherein the physical document comprises a surface including a plurality of document data items.

20. The method as recited in claim 19, wherein at least some of the document data items are human-readable.

21. The method as recited in claim 19, further comprising:

the interactive display surface displaying, via the user interface, a plurality of selectable application data items corresponding to the software application; and

in response to a user'"'"'s selection of an application data item via the user interface, the interactive display surface visibly indicates a location on the surface of the physical document that includes a document data item corresponding to the application data item selected via the user interface.

22. The method as recited in claim 19, further comprising:

in response to a user selecting a particular area of the surface of the physical document, the interactive display surface indicating to the user one or more particular application data items corresponding to the particular area of the surface of the physical document.

23. The method as recited in claim 13, further comprising the interactive display surface displaying the user interface such that one or more elements of the user interface are visually associated with a portion of the encoded data located on the physical document.

24. The method as recited in claim 13, further comprising:

generating an element of the user interface displayed by the interactive display surface based on a context, when the context includes one or more of a currently active application or one or more additional physical documents received by the interactive display surface.

25. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to:

detect, by an interactive display surface, placement of a physical document on the interactive display surface;

optically scan, by the interactive display surface, the physical document to obtain encoded data,

wherein the physical document possesses encoded data, and

wherein the encoded data includes data other than data directly identifying the physical document;

in response to detecting the interactive display surface optically scanning the physical document;

determine, based on the encoded data, a document type from a plurality of document types; and

based on the encoded data and the document type, identify, from a plurality of software applications, a software application capable of functioning in concert with the encoded data, wherein the software application comprises a plurality of application data items; and

in response to the identifying;

display, by the interactive display surface, a user interface for the software application;

receive user input via a portion of the physical document;

visibly distinguish, based on the user input and software application, the portion of the physical document at a position physically matching the portion of the physical document that received the user input; and

select a first application data item of the plurality of application data items based on a relationship between the first application data item and the portion of the physical document; and

display the first application data item.

26. The non-transitory computer-readable storage medium as recited in claim 25, wherein optically scanning the physical document to obtain the encoded data, the program instructions are further computer-executable to optically scan data from one or more surfaces of the physical document.

27. The non-transitory computer-readable storage medium as recited in claim 25, wherein the physical document is a tax form, and wherein the encoded data comprises one selected from a group consisting of a geometric pattern of the tax form and a print pattern of the tax form.

28. The non-transitory computer-readable storage medium as recited in claim 25, wherein the physical document further possesses identifying data distinct from the encoded data.

29. The non-transitory computer-readable storage medium as recited in claim 25, wherein the program instructions are further computer-executable to:

in response to detecting the user input, dynamically generate one or more elements corresponding to a changed state of the user interface, and display the user interface in the changed state via the interactive display surface.

30. The non-transitory computer-readable storage medium as recited in claim 29, wherein the user input comprises tactile input received via the interactive display surface.

31. The non-transitory computer-readable storage medium as recited in claim 25, wherein the physical document comprises a surface including a plurality of document data items.

32. The non-transitory computer-readable storage medium as recited in claim 31, wherein at least some of the document data items are human-readable.

33. The non-transitory computer-readable storage medium as recited in claim 31, wherein the program instructions are further computer-executable to:

display, via the user interface and the interactive display surface, a plurality of selectable application data items corresponding to the software application; and

in response to detecting a user'"'"'s selection of an application data item via the user interface, visibly indicate, via the interactive display surface, a location on the surface of the physical document that includes a document data item corresponding to the application data item selected via the user interface.

34. The non-transitory computer-readable storage medium as recited in claim 31, wherein the program instructions are further computer-executable to:

in response to detecting a user selecting a particular area of the surface of the physical document, indicate to the user via the interactive display surface one or more particular application data items corresponding to the particular area of the surface of the physical document.

35. The non-transitory computer-readable storage medium as recited in claim 25, wherein the instructions are further computer-executable to display the user interface such that the one or more elements of the user interface are visually associated with a portion of the encoded data located on the physical document.

36. The non-transitory computer-readable storage medium as recited in claim 25, wherein the instructions are further computer-executable to:

generate an element of the user interface displayed by the interactive display surface dependent upon a context, when the context includes one or more of a currently active application or one or more additional physical documents received by the interactive display surface.