Methods, apparatus and data structures for providing a user interface, which exploits spatial memory, to objects

• US 6,188,405 B1
• Filed: 09/14/1998
• Issued: 02/13/2001
• Est. Priority Date: 09/14/1998
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dumais, Susan T., Czerwinski, Mary, Thiel, David D., Dziadosz, Susan E., Van Dantzich, Maarten, Robbins, Daniel C., Robertson, George G.
• Primary Examiner(s): Bayerl, Raymond J.
• Application Number:
  US09/152,525
• Time in Patent Office: 883 Days
• Field of Search: 345/339, 345/348, 345/349, 345/351, 345/145, 345/968, 345/335, 345/334, 345/333, 345/428, 707/104, 707/6, 709/302, 709/328
• US Class Current: 715/764
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: Y10S 715/968 interface for database querying and retrieval

Abstract

A graphical user interface in which object thumbnails, as scaled images, are rendered on a two-dimensional wall or screen, and which exploits spatial memory. The objects may be moved, continuously, on the surface with a two-dimensional input device. Pop-up title bars may be rendered over active objects. Intelligent help may be provided to the user, as visual indicators, based on proximity clustering or based on matching algorithms.

Claims

1. A man-machine interface method for permitting a user to act on thumbnails, each thumbnail associated with an object containing information and being a scaled image of the associated object, for use with a machine having a video display device and a user input device, and being able to read from and write to a persistent storage device and a non-persistent storage device, the man-machine interface method comprising steps of:

a) generating a two-dimensional wall to be rendered on the video display device;

b) determining a location, having two coordinates defining a location on the two-dimensional wall, of each of the thumbnails on the two-dimensional wall;

c) generating the thumbnails on the two-dimensional wall, at the determined locations, to be rendered on the video display device;

d) storing, for each of the thumbnails, associated location and state information to the non-persistent storage device; and

e) for each of the thumbnails not being moved, storing the determined location of the thumbnail to the persistent storage device.

2. The man-machine interface method of claim 1 further comprising steps of:

e) accepting inputs from the user input device;

f) determining a cursor location based on the accepted inputs;

g) generating a cursor at the determined cursor location, to be rendered on the video display device; and

h) if the cursor is located on one of the thumbnails, defining that thumbnail as an active thumbnail.

3. The man-machine interface method of claim 2 further comprising a step of:

i) generating a pop-up textual information bar located over the active thumbnail, to be rendered on the video display device.

4. The man-machine interface method of claim 2 further comprising a step of:

i) if the user input provides a selection input and if an active thumbnail exists, then generating a higher resolution visual representation of the object represented by and associated with the active thumbnail, at a preferred viewing location on the two-dimensional wall, to be rendered on the video display device.

5. The man-machine interface method of claim 4 wherein the selection input is a mouse click.

6. The man-machine interface method of claim 4 wherein the step of generating the higher resolution visual representation of the object represented by and associated with the active thumbnail includes a sub-step of:

generating an animation which moves the higher resolution visual representation of the object represented by and associated with the active thumbnail from the location of the active thumbnail to the preferred viewing location, to be rendered on the video display device.

7. The man-machine interface method of claim 6 further comprising a step of:

j) if the user input provides a deselection input and if a selected thumbnail exists, then generating a video output for moving the high resolution visual representation of the object represented by and associated with the selected thumbnail to the location of the active thumbnail, to be rendered on the video display device.

8. The man-machine interface method of claim 7 wherein the deselection input is a mouse click.

9. The man-machine interface method of claim 2 further comprising a step of:

i) if the user input provides a selection input and if an active thumbnail exists, then

i) invoking an application related to the object represented by and associated with the active thumbnail,

ii) loading the object represented by and associated with the active thumbnail into the application, and

iii) generating a video output of the application with the loaded object represented by and associated with the active thumbnail at a preferred viewing location, to be rendered on the video display device.

10. The man-machine interface method of claim 9 wherein the selection input is a mouse click.

11. The man-machine interface method of claim 2 further comprising a step of:

i) if the user input provides a move input and if an active thumbnail exists, then

i) updating a location of the active thumbnail based on the move input.

12. The man-machine interface method of claim 11 wherein the move input is a pointer drag.

13. The man-machine interface method of claim 11 wherein as a thumbnail is moved,

its location is continuously updated, and

outputs of the thumbnail at the updated locations to be rendered on the video display device are generated, such that the position of the object is continuously fed back to a user via the video display device.

14. The man-machine interface method of claim 2 further comprising the steps of:

i) determining a degree to which each one of the other thumbnails is related to the active thumbnail;

j) for each one of the other thumbnails, comparing the degree of match to a predetermined threshold to determine whether said each other one of the thumbnails matches the active thumbnail; and

k) generating visual cues, to be rendered on the video display device, for associating the active thumbnail and the other ones of the thumbnails matching the active thumbnail, wherein each of the visual cues depicts more than two degrees of relatedness.

15. The man-machine interface method of claim 14 further comprising a step of:

1) providing a tool, to be rendered on the video display device, for permitting a user to change the predetermined threshold.

16. The man-machine interface method of claim 14 wherein the generated visual cue, to be rendered on the video display device, is selected from a group of visual cues consisting of (a) a horizontal bar meter, indicating a degree of match and positioned over the visual representation of each one of the other thumbnails matching the active thumbnail, (b) a horizontal sloped bar meter, indicating a degree of match and positioned over the visual representation of each one of the other thumbnails matching the active thumbnail, (c) a pie meter, indicating a degree of match and positioned over the visual representation of each one of the other thumbnails matching the active thumbnail, (d) a roof meter, the height of which indicates a degree of match and positioned over the visual representation of each one of the other thumbnails matching the active thumbnail, (e) a vertical bar meter, indicating a degree of match and positioned close to the visual representation of each of the other thumbnails matching the active thumbnails, and (f) a dial meter, indicating a degree of match and positioned close to the visual representation of each of the other thumbnails matching the active thumbnail.

17. The man-machine interface method of claim 2 further comprising the steps of:

i) determining a degree to which each of the other thumbnails is related to the active thumbnail; and

j) for each one of the other thumbnails, generating a visual degree of match indicator, to be rendered on the video display device adjacent to the visual representation of said each one of the other thumbnails,

wherein each of the visual degree of match indicators depicts more than two degrees of match.

18. The man-machine interface method of claim 17 wherein the visual degree of match indicator is selected from a group of visual indicators consisting of (a) a horizontal bar meter, (b) a horizontal sloped meter, (c) a pie meter, (d) a roof meter, (e) a vertical bar meter, and (f) a dial meter.

19. The man-machine interface method of claim 1 wherein the thumbnails are low resolution bit maps.

20. The man-machine interface method of claim 19 wherein the low resolution bit maps are 64 pixels by 64 pixels and have 24 bit color.

21. The man-machine interface method of claim 1 wherein the objects are selected from a group consisting of (a) an HTML document, (b) a database record, (c) a word processor document, or (d) a contact.

22. The man-machine interface method in claim 1 further comprising the steps of:

e) determining clusters of visual representations of said thumbnails located in close proximity to one another; and

f) generating a visual indicator, to be rendered on the video display device, of any of the clusters so determined, wherein the visual indicator is separate and distinct from the visual representations of said thumbnails.

23. The man-machine interface method of claim 22 wherein the step of determining clusters is performed based on at least one factor selected from a group of factors consisting of:

(a) a distance between a location of one of the thumbnails and a location of another one of the thumbnails, and (b) distances between the location of said one thumbnail and locations of all other ones of the thumbnails.

24. The man-machine interface method in claim 1 further comprising the steps of:

e) determining clusters of visual representations of said thumbnails located in close proximity to one another; and

f) generating a visual indicator, to be rendered on the video display device, of any of the clusters so determined, wherein the visual indicator leaves positions of said thumbnails unchanged.

25. A system which permits a user to interact with thumbnails associated with objects containing information and being a scaled image of the associated object, the system comprising:

a) an input facility for accepting user inputs;

b) a non-persistent storage facility containing

i) corresponding location and state information for each of the thumbnails, and

ii) a cursor location;

c) a persistent storage facility containing

i) a two-dimensional wall, and

ii) a location, having two coordinates defining a location on the two-dimensional wall, of each of the thumbnails not being moved;

d) a processing unit which

i) accepts user inputs from the input facility,

ii) updates (a) the corresponding location and state information for each of the thumbnails contained in the non-persistent storage facility, and (b) the cursor location contained in the non-persistent storage facility, based on the accepted user inputs,

iii) when one of the thumbnails transitions from moving to not moving, updates the corresponding location of each of the thumbnails contained in the persistent storage facility, and

iv) generates video outputs based on

A) the location and state information for each of the thumbnails,

B) the cursor location, and

C) the two-dimensional wall, stored in the non-persistent storage facility, such that, when the video outputs are applied to a video display unit, each of the thumbnails will be displayed at the corresponding location on the wall; and

e) a video display unit for rendering the video outputs generated by the processing unit.

26. The system of claim 25 wherein the state information for each of the thumbnails contained in the non-persistent storage facility includes an indication of whether or not the thumbnail is active, and

wherein the processing unit determines that a thumbnail is active if a cursor is on a thumbnail based on the cursor location and the location of the thumbnail.

27. The system of claim 26 wherein the persistent storage facility further contains descriptive textual information for each of the thumbnails, and

wherein, if a thumbnail is active,

i) the processing unit generates a pop-up bar, based on descriptive textual information, for the active thumbnail, and

ii) the video display unit renders the pop-up bar over the rendered thumbnail.

28. The system of claim 26 wherein, if a thumbnail is active and the input facility accepts a selection input, then

i) the processing unit updates the state of the thumbnail,

ii) the processing unit gets a second, higher resolution, visual representation of the object represented by and associated with the thumbnail,

iii) the processing unit generates a video output based on the higher resolution, visual representation of the object represented by and associated with the thumbnail at a preferred viewing location, and

iv) the video display device renders the video output generated by the processing unit.

29. The system of claim 28 wherein each thumbnail is a 64 pixel by 64 pixel bit map having 24 bit color and wherein each higher resolution, visual representation of the objects is a 512 pixel by 512 pixel bit map having 24 bit color.

30. The system of claim 28 wherein the processing unit further effects a video output based on an animation of the higher resolution, visual representation of the object represented by and associated with the thumbnail, moving from the location of the thumbnail to a location at a center of the two-dimensional wall.

31. The system of claim 28 wherein the input facility includes a mouse and wherein the selection input is a mouse click.

32. The system of claim 28 wherein, if a thumbnail is active and the input facility accepts a move input, then

i) the processing unit updates the state and location of the thumbnail in the non-persistent storage facility,

ii) the processing unit generates a video output based on the updated location of the thumbnail, and

iii) the video display device renders the video output generated by the processing unit.

33. The system of claim 32 wherein the input facility is a pointer and wherein the move input is a pointer drag.

34. The system of claim 32 wherein as a thumbnail is moved,

the processing unit continuously updates the location of the thumbnail stored in the non-persistent storage facility,

the processing unit generates video outputs of the thumbnail at the updated locations, and

the video display device renders the video outputs generated by the processing unit, such that the position of the object is continuously fed back to a user via the video display device.

35. The system of claim 26 wherein, if a thumbnail is active and the input facility accepts a selection input, then

i) the processing unit updates the state of the thumbnail,

ii) the processing unit opens an application with which the object, associated with and represented by the thumbnail, is associated,

iii) the processing unit loads the object into the application,

iv) the processing unit generates a video output based on the object loaded onto the opened application and a preferred viewing location, and

v) the video display device renders the video output generated by the processing unit.

36. The system of claim 35 wherein the input facility includes a mouse and wherein the selection input is a mouse click.

37. The system in claim 26 wherein, if the thumbnail is active;

i) the processing unit determines whether any other ones of the thumbnails match, to a predetermined degree, the active thumbnail; and

ii) the video display unit renders visual indicators for visually associating any of said other thumbnail matching, to a predetermined degree, the active thumbnail, wherein each of the visual indicators depicts more than two degrees of match.

38. The system in claim 26, wherein if a thumbnail is active:

i) the processing unit determines, for each of the thumbnails other than the active thumbnail, the degree to which said each of the other thumbnails matches the active thumbnail; and

ii) the processing unit generates, for each one of the thumbnails other than the active thumbnail, a visual degree of match indicator, to be rendered on the video display device adjacent to the visual representation of said one thumbnail, wherein each of the degree of match indicators depicts more than two degrees of match.

39. The system of claim 38 wherein the visual degree of match indicator is selected from a group of visual indicators consisting of (a) a horizontal bar meter, (b) a horizontal sloped meter, (c) a pie meter, (d) a roof meter, (e) a vertical bar meter, and (f) a dial meter.

40. The system in claim 25 wherein the processing unit further:

groups ones of the thumbnails based on their proximity to one another so as to form at least one group of thumbnails;

generates a visual effect for indicating said one group of thumbnails, the visual effect being separate and distinct from said thumbnails; and

generates a video output based on the visual effect,

wherein the video display unit renders the video output generated by the processing unit.

41. The system in claim 25 wherein the processing unit further:

groups ones of the thumbnails based on their proximity to one another so as to form at least one group of thumbnails;

generates a visual effect for indicating said one group of thumbnails, wherein the visual effect leaves positions of said thumbnails unchanged; and

generates a video output based on the visual effect,

wherein the video display unit renders the video output generated by the processing unit.

42. A machine readable medium containing data and machine executable instructions which, when executed by a machine that can read from and write to a persistent storage device and a non-persistent storage device, perform steps comprising:

a) generating a two-dimensional wall to be rendered on the video display device;

b) determining a location of each of the thumbnails on the two-dimensional wall;

c) generating the thumbnails on the two-dimensional wall, at the determined locations, to be rendered on the video display device;

d) storing, for each of the thumbnails, associated location and state information to the non-persistent storage device; and

e) for each of the thumbnails not being moved, storing the determined location of the thumbnail to the persistent storage device,

wherein, each thumbnail is associated with an object containing information and is a scaled image of the associated object.

43. The machine readable medium of claim 42, including further instructions which, when executed by a machine, further perform steps of:

e) accepting inputs from the user input device;

f) determining a cursor location based on the accepted inputs;

g) generating a cursor at the determined cursor location, to be rendered on the video display device; and

h) if the cursor is located on one of the thumbnails, defining that thumbnail as an active thumbnail.

44. The machine readable medium of claim 43, including further instructions which, when executed by a machine, further perform a step of:

i) if the user input provides a selection input and if an active thumbnail exists, then generating a higher resolution visual representation of the object represented by and associated with the active thumbnail, at a preferred viewing location on the two-dimensional wall, to be rendered on the video display device.

45. The machine readable medium of claim 44, wherein the selection input is a mouse click.

46. The machine readable medium of claim 45, wherein the step of generating the higher resolution visual representation of the object represented by and associated with the active thumbnail includes a sub-step of:

generating an animation which moves the higher resolution visual representation of the object represented by and associated with the active thumbnail from the location of the active thumbnail to the preferred viewing location, to be rendered on the video display device.

47. The machine readable medium of claim 46, including further instructions which, when executed by a machine, further perform a step of:

j) if the user input provides a deselection input and if a selected thumbnail exists, then generating a video output for moving the high resolution visual representation of the object represented by and associated with the selected thumbnail to the location of the active thumbnail, to be rendered on the video display device.

48. The machine readable medium of claim 47 wherein the deselection input is a mouse click.

49. The machine readable medium of claim 43, including further instructions which, when executed by a machine, further perform steps of:

i) if the user input provides a selection input and if an active thumbnail exists, then

i) invoking an application related to the object represented by and associated with the active thumbnail,

ii) loading the object represented by and associated with the active thumbnail into the application, and

iii) generating a video output of the application with the loaded object represented by and associated with the active thumbnail at a preferred viewing location, to be rendered on the video display device.

50. The machine readable medium of claim 49 wherein the selection input is a mouse click.

51. The machine readable medium of claim 43, including further instructions which, when executed by a machine, further perform steps of:

i) if the user input provides a move input and if an active thumbnail exists, then

i) updating a location of the active thumbnail based on the move input.

52. The machine readable medium of claim 51 wherein the move input is a pointer drag.

53. The machine readable medium of claim 52 wherein as a thumbnail is moved,

its location is continuously updated, and

outputs of the thumbnail at the updated locations to be rendered on the video display device are generated, such that the position of the object is continuously fed back to a user via the video display device.

54. The machine readable medium of claim 43, including further instructions which, when executed by a machine, perform the further steps of:

i) determining a degree to which each of the other thumbnails is related to the active thumbnail;

j) for said each of the other thumbnails, comparing the degree of match to a predetermined threshold to determine whether said each other thumbnail matches the active thumbnail; and

k) generating visual cues, to be rendered on the video display device, for associating the active thumbnail and other ones of the thumbnails matching the active thumbnail, wherein each of the visual cues depicts more than two degrees of relatedness.

55. The machine readable medium of claim 43, including further instructions which, when executed by a machine, perform the further steps of:

i) determining a degree to which each of the other thumbnails is related to the active thumbnail; and

j) for each one of the thumbnails other than the active thumbnail, generating a visual degree of match indicator, to be rendered on the video display device adjacent to the visual representation of said each other thumbnail,

wherein each of the visual degree of match indicators depicts more than two degrees of match.

56. The machine readable medium of claim 42, including further instructions which, when executed by a machine, further perform a step of:

i) generating a pop-up textual information bar located over the active thumbnail, to be rendered on the video display device.

57. The machine readable medium of claim 42 wherein the thumbnails are low resolution bit maps.

58. The machine readable medium of claim 57 wherein the low resolution bit maps are 64 pixels by 64 pixels and have 24 bit color.

59. The machine readable medium of claim 42, wherein the objects are selected from a group consisting of (a) an HTML document, (b) a database record, (c) a word processor document, or (d) a contact.

60. The machine readable medium of claim 42 further comprising machine readable instructions for performing the further steps of:

e) determining clusters of visual representations of ones of the thumbnails located in close proximity to one another; and

f) generating a visual indicator of any of the clusters so determined, wherein the visual indicator is separate and distinct from the visual representations of the ones of the thumbnails.

61. The machine readable medium of claim 60 wherein the step of determining clusters is performed based on at least one factor selected from a group of factors consisting of:

(a) a distance between a location of one of the thumbnails and a location of another one of the thumbnails, and (b) distances between the location of said one thumbnail and locations of all other ones of the thumbnails.

62. The machine readable medium of claim 42, including further instructions which, when executed by a machine, perform the further steps of, for a given one thumbnail:

i) determining a degree to which each of the other thumbnails is related to the given one thumbnail; and

j) for each of the other thumbnails, if the degree to which it is related to the given thumbnail exceeds a predetermined threshold, generating a visual degree of match indicator, to be rendered on the video display device adjacent to the visual representation of said each other thumbnail, wherein each of the visual degree of match indicators depicts more than two degrees of match.

63. The machine readable medium of claim 42 further comprising machine readable instructions for performing the further steps of:

e) determining clusters of visual representations of ones of the thumbnails located in close proximity to one another; and

f) generating a visual indicator of any of the clusters so determined, wherein the visual indicator leaves positions of ones of the thumbnails unchanged.