Methods, apparatus and data structures for providing a user interface, which exploits spatial memory in three-dimensions, to objects

• US 6,166,738 A
• Filed: 09/14/1998
• Issued: 12/26/2000
• Est. Priority Date: 09/14/1998
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Robbins, Daniel C., Robertson, George G., Van Dantzich, Maarten Roderik
• Primary Examiner(s): Bayerl, Raymond J.
• Application Number:
  US09/152,491
• Time in Patent Office: 834 Days
• Field of Search: 345/355, 345/339, 345/349, 345/348, 345/427, 345/419, 345/421, 345/440, 345/968, 345/976, 345/977, 345/978, 345/357, 345/356, 345/139, 345/8
• US Class Current: 715/839
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A graphical user interface in which object thumbnails are rendered on a simulated three-dimensional surface which (i) exploits spatial memory and (ii) allows more objects to be rendered on a given screen. The objects may be moved, continuously, on the surface with a two-dimensional input device.

Claims

1. A man-machine interface method for permitting a user to act on thumbnails, each thumbnail being a scaled image of an associated object containing information, for use with a computer-implemented system having a video display device and a user input device, the method comprising the steps of:

a) depicting a virtual three-dimensional environment, having a three-dimensional surface, on the video display device;

b) determining a virtual location of each of the thumbnails in the environment so as to define a plurality of determined locations;

c) rendering each of the thumbnails, within the environment, at a corresponding one of the determined locations so as to define a plurality of rendered thumbnails;

d) if a cursor, as rendered on the display device, is positioned, by the user, on one of the rendered thumbnails, designating said one thumbnail as being an active thumbnail; and

e) if the user instructs the system, through a move input provided through the user input device, to move the active thumbnail in the environment, in response to the move input;

e1) updating a corresponding one of the plurality of determined locations associated with the active thumbnail so as to define an updated location;

e2) rendering the active thumbnail at the updated location; and

e3) if the updated location of the active thumbnail lies within a predetermined distance of a corresponding one of the plurality of determined locations associated with another one of the thumbnails;

e3a) defining the other thumbnail as a pushed thumbnail;

e3b) changing the corresponding one of the determined locations associated with the pushed thumbnail so as to define a changed location, wherein the changed location maintains a predetermined distance between the other thumbnail and the active thumbnail so as to prevent the visual representations of both the pushed and active thumbnails from being totally occluded by each other; and

e3c) rendering the pushed thumbnail at the changed location in the environment.

2. The man-machine interface method of claim 1 wherein the three-dimensional surface comprises an inclined plane.

3. The man-machine interface method of claim 2 wherein the inclined plane includes landmarks.

4. The man-machine interface method of claim 3 wherein the landmarks are depicted as colored circles on the inclined plane.

5. The man-machine interface method of claim 1 further comprising the steps of:

f) accepting inputs from the user input device so as to define accepted inputs;

g) determining the cursor location based on the accepted inputs so as to define a determined cursor location;

h) generating the cursor at the determined cursor location, to be rendered on the video display device.

6. The man-machine interface method of claim 5 wherein the step of determining the cursor location includes the sub-steps of:

f1) determining a cursor position in a two-dimensional plane; and

f2) mapping the cursor position in the two-dimensional plane to the cursor location in the environment.

7. The man-machine interface method of claim 5 further comprising a step of:

i) generating a pop-up information bar located over the active thumbnail, to be rendered on the video display device.

8. The man-machine interface method of claim 5 further comprising a step of:

i) if the user input provides a selection input and if the active thumbnail exists, then generating a higher resolution visual representation of the object represented by and associated with the active thumbnail, at a preferred viewing location at a foreground of the three dimensional environment, to be rendered on the video display device.

9. The man-machine interface method of claim 8 wherein the selection input comprises a mouse click.

10. The man-machine interface method of claim 8 wherein the step of generating the higher resolution visual representation of the object represented by and associated with the active thumbnail includes a sub-step of:

generating an animation which moves the higher resolution visual representation of the object represented by and associated with the active thumbnail from the location of the active thumbnail to the preferred viewing location at the foreground of the three dimensional environment, to be rendered on the video display device.

11. The man-machine interface method of claim 10 further comprising a step of:

i) if the user input provides a deselection input and if a selected thumbnail exists, then generating a video output for moving the high resolution visual representation of the object represented by and associated with the active thumbnail to the location of the selected thumbnail, to be rendered on the video display device.

12. The man-machine interface method of claim 11 wherein the deselection input comprises a mouse click.

13. The man-machine interface method of claim 5 further comprising a step of:

i) if the user input provides a selection input and if the active thumbnail exists, then

i) invoking an application related to the object represented by and associated with the active thumbnail,

ii) loading the object represented by and associated with the active thumbnail into the application, and

iii) generating a video output of the application with the loaded object represented by and associated with the active thumbnail at a preferred viewing location, to be rendered on the video display device.

14. The man-machine interface method of claim 13 wherein the selection input is a mouse click.

15. The man-machine interface method of claim 5 wherein the move input comprises a left button mouse drag.

16. The man-machine interface method of claim 1 wherein the three-dimensional surface defines a foreground and a background, and

wherein the step of generating thumbnails, within the three-dimensional environment, at the determined locations, to be rendered on the video display device, includes a sub-step of:

i) using perspective views so that any thumbnails in the foreground defined by the three-dimensional surface appear larger than any thumbnails in the background defined by the three-dimensional surface.

17. The man-machine interface method of claim 16 wherein a non-active one of the thumbnails partially occludes any other non-active ones of the thumbnails located behind it, based on a viewing point.

18. The man-machine interface method of claim 1 further comprising steps of:

f) accepting inputs from the user input device;

g) determining a head location of the user relative to the video display device based on the accepted inputs;

h) determining a virtual viewing point, in the three-dimensional environment, based on the head location determined in step (g); and

i) generating the three-dimensional environment and the thumbnails with simulated parallax based on the virtual viewing point determined in step (h), to be rendered on the video display device.

19. The man-machine interface method of claim 1 wherein the thumbnails comprise low resolution bit maps.

20. The man-machine interface method of claim 19 wherein the low resolution bit maps comprise 64 pixels by 64 pixels and have 24 bit color.

21. The man-machine interface method of claim 1 wherein the objects are selected from a group consisting of (a) an HTML document, (b) a database record, (c) a word processor document, or (d) a contact.

22. A computer readable medium, having computer executable instructions stored therein, for performing the steps of claim 1.

23. A computer-implemented system which permits a user to act on thumbnails, each thumbnail being a scaled image of an associated object containing information, wherein the system comprises:

a) a processor;

b) a memory, connected to the processor and storing computer executable instructions therein;

c) a user input device which provides user input for use by the processor; and

d) a video display device which displays information provided by the processor;

e) wherein the processor, in response to the executable instructions;

e1) depicts a virtual three-dimensional environment, having a three-dimensional surface, on the video display device;

e2) determines a virtual location of each of the thumbnails in the environment so as to define a plurality of determined locations;

e3) renders each of the thumbnails, within the environment, at a corresponding one of the determined locations so as to define a plurality of rendered thumbnails;

e4) if a cursor, as rendered on the display device, is positioned, by the user, on one of the rendered thumbnails, designates said one thumbnail as being an active thumbnail; and

e5) if the user instructs the system, through a move input provided through the user input device, to move the active thumbnail in the environment, in response to the move input;

e5a) updates a corresponding one of the plurality of determined locations associated with the active thumbnail so as to define an updated location;

e5b) renders the active thumbnail at the updated location; and

e5c) if the updated location of the active thumbnail lies within a predetermined distance of a corresponding one of the plurality of determined locations associated with another one of the thumbnails;

e5c(i)) defines the other thumbnail as a pushed thumbnail;

e5c(ii)) changes the corresponding one of the determined locations associated with the pushed thumbnail so as to define a changed location, wherein the changed location maintains a predetermined distance between the other thumbnail and the active thumbnail so as to prevent the visual representations of both the pushed and active thumbnails from being totally occluded by each other; and

e5c(iii)) renders the pushed thumbnail at the changed location in the environment.

24. The system of claim 23 wherein the memory stores state information for each of the thumbnails, the state information comprising an indication of whether or not said each thumbnail is active, and wherein the processor, in response to the stored instructions, determines that one of the thumbnails is active if the cursor is located on said one thumbnail based on a cursor location and a current location of the one thumbnail.

25. The system of claim 24 wherein the processor, in response to the stored instructions:

generates a pop-up bar, based on descriptive textual information, for the active thumbnail; and

renders, through the video display device, the pop-up bar in the vicinity of the active thumbnail.

26. The system of claim 24 wherein the processor, in response to the stored instructions:

updates a state of the active thumbnail based on a selection input provided by the user through the user input device;

accesses, from the memory, a second, higher resolution, visual representation of the object represented by and associated with the active thumbnail;

generates a video output based on the higher resolution, visual representation of the object represented by and associated with the active thumbnail situated at a preferred viewing location in the environment; and

renders, through the video display device, the video output.

27. The system of claim 26 further comprising an audio output device, wherein the memory stores a first audio cue;

and wherein, when one of the thumbnails is selected by the user, the processor, in response to the stored instructions, provides the first audio cue to the audio output device.

28. The system of claim 26 wherein each of the plurality of thumbnails comprises a 64 pixel by 64 pixel bit map having 24 bit color and wherein each higher resolution, visual representation of the objects comprises a 512 pixel by 512 pixel bit map having 24 bit color.

29. The system of claim 26 wherein the processor, in response to the stored instructions, effects a video output based on an animation of the higher resolution, visual representation of the object represented by and associated with the active thumbnail moving from the corresponding one of the determined locations associated with the active thumbnail to a location at a foreground of the environment.

30. The system of claim 26 wherein the user input device comprises a mouse and the selection input comprises a mouse click.

31. The system of claim 26 wherein the processor, in response to the stored instructions and the move input:

updates the state information and location of the active thumbnail;

generates the video output based on the updated location of the active thumbnail; and

renders, through the display device, the video output.

32. The system of claim 31 wherein the user input device comprises a mouse and the move input comprises a left button mouse drag.

33. The system of claim 24 wherein the processor, in response to the stored instructions and for the active thumbnail:

updates the state information of the active thumbnail;

opens an application with which the object, associated with and represented by the active thumbnail, is related, so as to define an opened application;

loads the object into the opened application;

generates a video output based on the object loaded onto the opened application and a preferred viewing location; and

renders, through the video display device, the video output.

34. The system of claim 33 wherein the user input device comprises a mouse and the selection input comprises a mouse click.

35. The system of claim 23 wherein the memory stores virtual viewing point location information and the system further comprises a camera, and wherein the processor, in response to the executable instructions:

determines a head location of the user relative to the video display device so as to define a determined head location;

updates the virtual viewing point location information based on the determined head location so as to define an updated virtual viewing point location;

generates, in response to the updated virtual viewing point location, a video output which simulates parallax in the three-dimensional surface of the environment and in the thumbnails displayed in the environment; and

renders the video output through the video display device.