Methods of interfacing with multi-input devices and multi-input display systems employing interfacing techniques

• US 8,335,996 B2
• Filed: 04/08/2009
• Issued: 12/18/2012
• Est. Priority Date: 04/10/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: Davidson, Philip L., Han, Jefferson Y.
• Primary Examiner(s): Lo, Weilun
• Assistant Examiner(s): CHUONG, TRUC T
• Application Number:
  US12/420,493
• Publication Number:
  US 20090259967A1
• Time in Patent Office: 1,350 Days
• Field of Search: 715/799, 715/800, 715863-864
• US Class Current: 715/800
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Methods and systems for interfacing with multi-input devices employ various techniques for controlling the window framing of images. Such techniques provide control, including moving, sizing, and orientating, of one or more displayed window frames in which one or more images are displayed.

Claims

1. A method of interfacing with a multi-input display device, comprising the steps of:

displaying on a multi-input display device a graphical window, the displayed graphical window having an inner frame and an outer frame, the display device displaying only a first portion of an object, and the first portion of the object being displayed within the graphical window;

identifying an initial position of one or more elements contacting the inner frame or the outer frame of the displayed graphical window;

identifying a movement of the one or more elements;

changing at least one of a position, a shape and a size of the displayed graphical window as a function of the initial position and movement of the one or more elements; and

manipulating a display within the displayed graphical window in accordance with one or more of a plurality of modifications, the plurality of modifications comprising;

a first modification implemented when the initial position of the one or more elements is identified as contacting one of the inner frame and the outer frame of the displayed graphical window, the first modification comprising manipulating the display by displaying a second portion of the object within the changed graphical window, the second portion including portions of the object common to the first portion of the object and portions of the object not included in the first portion of the object, and the second portion being displayed such that positions on the display device of the common portions of the first and the second portions of the object are substantially the same, and

a second modification implemented when the initial position of the one or more elements is identified as contacting the other of the inner frame and the outer frame of the displayed graphical window, the second modification comprising manipulating the display by changing the first portion of the object by stretching or shrinking the horizontal or vertical extent of the first portion of the object in a same manner that the horizontal or vertical extent of the displayed graphical window is stretched or shrunk and displaying the changed first portion of the object within the stretched or shrunken graphical window.

2. The method of claim 1,

wherein identifying an initial position of the one or more elements comprises identifying the initial position of an initial one of the plurality of elements as contacting said one of the inner frame and the outer frame of the displayed graphical window,

wherein a plurality of elements contact the inner frame or the outer frame of the displayed graphical window,

wherein the first modification is implemented when the initial position of said initial one of the plurality of elements is identified as contacting said one of the inner frame and the outer frame of the displayed graphical window, and

wherein the second modification is implemented when the initial position of said initial one of the plurality of elements is identified as contacting the other of the inner frame and the outer frame of the displayed graphical window.

3. The method of claim 1, further comprising identifying a number of the one or more elements contacting the inner frame or the outer frame of the displayed graphical window;

and

wherein the changing step changes a characteristic of the displayed graphical window, the characteristic corresponding to a first characteristic when only one element is identified as contacting the inner frame or the outer frame of the displayed graphical window, the first characteristic being a position of the displayed graphical window on the display device; and

the characteristic corresponding to a second characteristic when a plurality of elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the second characteristic being at least one of a shape and a size of the displayed graphical window, the first and second characteristics being different.

4. The method of claim 1, further comprising identifying a number of plurality of elements contacting the inner frame or the outer frame of the displayed graphical window;

and

wherein the changing step changes the displayed graphical window in accordance with one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising;

a first graphical window modification implemented when only two elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the first graphical window modification changing at least one of a shape and a size of the displayed graphical window as a function of the identified initial positions and movement of the two elements, and

a second graphical window modification implemented when three elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the second graphical window modification changing at least one of a position, a shape and a size of the displayed graphical window as a function of the identified initial positions and movement of the three elements, the first and second graphical window modifications being different.

5. The method of claim 1, further comprising identifying a number of the one or more elements contacting the inner frame or the outer frame of the displayed graphical window;

and

wherein the changing step changes the displayed graphical window in accordance with a one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising;

a first graphical window modification implemented when only one element is identified as contacting the inner frame or the outer frame of the displayed graphical window, the first graphical window modification moving an edge of the displayed graphical window contacted by the one element in a direction of identified movement of the one element normal to an axis of the edge, and

a second graphical window modification implemented when a plurality of elements are identified as contacting the inner frame or the outer frame of the displayed graphical window along different edges of the displayed graphical window, the second graphical window modification moving each of the edges of the displayed graphical window contacted by a respective one of the elements in a direction of identified movement of the respective element normal to an axis of the respective edge.

6. The method of claim 1, wherein the changing step changes the displayed graphical window in accordance with one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising:

a first graphical window modification implemented when two elements are identified as contacting opposite edges of the displayed graphical window, the first graphical window modification changing at least one of a position and a shape of the displayed graphical window in accordance with the positions and movement of the two elements contacting the opposite edges,

a second graphical window modification implemented when two elements are identified as contacting adjacent edges of the displayed graphical window, the second graphical window modification changing at least one of a shape and a size of the displayed graphical window in accordance with the positions and movement of the two elements contacting the adjacent edges, and

a third graphical window modification implemented when two elements are identified as contacting a same edge of the displayed graphical window, the third graphical window modification changing at least one of a position and a shape of the displayed graphical window in accordance with the positions and movement of the two elements contacting the same edge.

13. The method of claim 5, wherein:

the first graphical window modification corresponds to moving the edge of the displayed graphical window contacted by the one element in a direction of identified movement of the one element normal to an axis parallel to the edge, and

the second graphical window modification corresponds to moving each of the edges of the displayed graphical window contacted by a respective one of the elements in a direction of identified movement of the respective element normal to an axis parallel to the respective edge.

7. A multi-input display system, comprising:

a display device configured to display on a display surface a graphical window, the displayed graphical window having an inner frame and an outer frame, the display device displaying only a first portion of an object, and the first portion of the object being displayed within the graphical window, and the display device being further configured to detect elements that are simultaneously in contact with the display surface; and

a controller adapted to identify an initial position of one or more elements contacting the inner frame or the outer frame of the displayed graphical window, to identify a movement of the one or more elements, to change at least one of a position, shape and size of the displayed graphical window as a function of the identified initial position and movement of the one or more elements; and

to manipulate a display within the displayed graphical window in accordance with one or more of a plurality of modifications, the plurality of modifications comprising;

a first modification implemented when the initial position of the one or more elements is identified as contacting one of the inner frame and the outer frame of the displayed graphical window, the first modification comprising manipulating the display by controlling the display device to display a second portion of the object within the changed graphical window, the second portion including portions of the object common to the first portion of the object and portions of the object not included in the first portion of the object, and the second portion being displayed such that positions on the display surface of the common portions of the first and the second portions of the object are substantially the same; and

a second modification implemented when the initial position of the one or more elements is identified as contacting the other of the inner frame and the outer frame of the displayed graphical window, the second modification comprising manipulating the display by changing the first portion of the object by stretching or shrinking the horizontal or vertical extent of the first portion of the object in a same manner that the horizontal or vertical extent of the displayed graphical window is stretched or shrunk and controlling the display device to display the changed first portion of the object within the stretched or shrunken graphical window.

8. The system of claim 7,

wherein the controller is adapted to identify an initial position of the one or more elements includes the controller being adapted to identify the initial position of an initial one of a plurality of elements as contacting said one of the inner frame and the outer frame of the displayed graphical window, and

wherein the controller is adapted to implement the first modification when the initial position of said initial one of the plurality of elements is identified as contacting said one of the inner frame and the outer frame of the displayed graphical window, and

wherein the controller is adapted to implement the second modification when the initial position of said initial one of the plurality of elements is identified as contacting the other of the inner frame and the outer frame of the displayed graphical window.

9. The system of claim 7,

wherein the controller is configured to identify a number of the one or more elements contacting the inner frame or the outer frame of the displayed graphical window, and

wherein the controller is configured to change a characteristic of the displayed graphical window, the characteristic corresponding to a first characteristic when only one element is identified as contacting the inner frame or the outer frame of the displayed graphical window, the first characteristic being a position of the displayed graphical window on the display surface;

and the characteristic corresponding to a second characteristic when a plurality of elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the second characteristic being at least one of a shape and a size of the displayed graphical window, the first and second characteristics being different.

10. The system of claim 7,

wherein the controller is configured to identify a number of plurality of elements contacting the inner frame or the outer frame of the displayed graphical window, and

wherein the controller is configured to control the displayed graphical window to change in accordance with one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising:

a first graphical window modification implemented when only two elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the first graphical window modification changing at least one of a shape and size of the displayed graphical window as a function of the identified initial positions and movement of the two elements, and

a second graphical window modification implemented when three elements are identified as contacting the inner frame or the outer frame of the displayed graphical window, the second graphical window modification changing at least one of a position, a shape and a size of the displayed graphical window as a function of the identified initial positions and movement of the three elements, the first and second graphical window modifications being different.

11. The system of claim 7,

wherein the controller is configured to identify a number of the one or more elements contacting the inner frame or the outer frame of the displayed graphical window, and

wherein the controller is configured to control the displayed graphical window to change in accordance with one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising:

a first graphical window modification implemented when only one element is identified as contacting the inner frame or the outer frame of the displayed graphical window, the first graphical window modification moving an edge of the displayed graphical window contacted by the one element in a direction of identified movement of the one element normal to an axis of the edge, and

a second graphical window modification implemented when a plurality of elements are identified as contacting the inner frame or the outer frame of the displayed graphical window along different edges of the displayed graphical window, the second graphical window modification moving each of the edges of the displayed graphical window contacted by a respective one of the elements in a direction of identified movement of the respective element normal to an axis of the respective edge.

12. The system of claim 7,

wherein the controller is configured to control the displayed graphical window to change in accordance with one or more of a plurality of graphical window modifications, the plurality of graphical window modifications comprising:

a first graphical window modification implemented when the controller identifies two elements as contacting opposite edges of the displayed graphical window, the first graphical window modification changing at least one of a position and a shape of the displayed graphical window in accordance with the positions and movement of the two elements contacting the opposite edges,

a second graphical window modification implemented when the controller identifies two elements as contacting adjacent edges of the displayed graphical window, the second graphical window modification changing at least one of a shape and a size of the displayed graphical window in accordance with the positions and movement of the two elements contacting the adjacent edges, and

a third graphical window modification implemented when the controller identifies two elements as contacting a same edge of the displayed graphical window, the third graphical window modification changing at least one of a position and a shape of the displayed graphical window in accordance with the positions and movement of the two elements contacting the same edge.

14. The system of claim 11, wherein:

the first graphical window modification corresponds to moving the edge of the displayed graphical window contacted by the one element in a direction of identified movement of the one element normal to an axis parallel to the edge, and

the second graphical window modification corresponds to moving each of the edges of the displayed graphical window contacted by a respective one of the elements in a direction of identified movement of the respective element normal to an axis parallel to the respective edge.

15. A method of interfacing with a multi-input display device, comprising the steps of:

displaying on a multi-input display device a first graphical window and a second graphical window, each of the displayed first and second graphical windows having an inner frame and an outer frame, the display device displaying only a first portion and a second portion of an object, the first portion of the object being displayed within the first graphical window and the second portion of the object being different from the first portion and being displayed within the second graphical window;

identifying an initial position of one or more first elements contacting the inner frame or the outer frame of the displayed first graphical window;

identifying a movement of the one or more first elements;

changing at least one of a position, a shape and a size of the displayed first graphical window as a function of the initial position and movement of the one or more first elements;

manipulating a first display within the displayed first graphical window in accordance with one or more of a first plurality of modifications, the first plurality of modifications comprising;

a first modification implemented when the initial position of the one or more first elements is identified as contacting one of the inner frame and the outer frame of the displayed first graphical window, the first modification comprising manipulating the first display by displaying a third portion of the object within the changed first graphical window, the third portion including portions of the object common to the first portion of the object and portions of the object not included in the first portion of the object, and the third portion being displayed such that positions on the display device of the common portions of the first and the third portions of the object are substantially the same, and

a second modification implemented when the initial position of the one or more first elements is identified as contacting the other of the inner frame and the outer frame of the displayed first graphical window, the second modification comprising manipulating the first display by changing the first portion of the object by stretching or shrinking the horizontal or vertical extent of the first portion of the object in a same manner that the horizontal or vertical extent of the displayed first graphical window is stretched or shrunk and displaying the changed first portion of the object within the stretched or shrunken first graphical window;

identifying an initial position of one or more second elements contacting the inner frame or the outer frame of the displayed second graphical window;

identifying a movement of the one or more second elements;

changing at least one of a position, a shape and a size of the displayed second graphical window as a function of the initial position and movement of the one or more second elements; and

manipulating a second display within the displayed second graphical window in accordance with one or more of a second plurality of modifications, the second plurality of modifications comprising;

a third modification implemented when the initial position of the one or more second elements is identified as contacting one of the inner frame and the outer frame of the displayed second graphical window, the third modification comprising manipulating the second display by displaying a fourth portion of the object within the changed second graphical window, the fourth portion including portions of the object common to the second portion of the object and portions of the object not included in the second portion of the object, and the fourth portion being displayed such that positions on the display device of the common portions of the second and the fourth portions of the object are substantially the same, and

a fourth modification implemented when the initial position of the one or more second elements is identified as contacting the other of the inner frame and the outer frame of the displayed second graphical window, the fourth modification comprising manipulating the second display by changing the second portion of the object by stretching or shrinking the horizontal or vertical extent of the second portion of the object in a same manner that the horizontal or vertical extent of the displayed second graphical window is stretched or shrunk and displaying the changed second portion of the object within the stretched or shrunken second graphical window.

17. The method of claim 15,

wherein identifying an initial position of one or more first elements comprises identifying the initial position of an initial one of the plurality of first elements as contacting said one of the inner frame and the outer frame of the displayed first graphical window,

wherein a plurality of first elements contact the inner frame or the outer frame of the displayed first graphical window,

wherein the first modification is implemented when the initial position of said initial one of the plurality of first elements is identified as contacting said one of the inner frame and the outer frame of the displayed first graphical window, and

wherein the second modification is implemented when the initial position of said initial one of the plurality of first elements is identified as contacting the other of the inner frame and the outer frame of the displayed first graphical window.

16. A multi-input display system, comprising:

a display device configured to display on a display surface a first graphical window and a second graphical window, each of the displayed first and second graphical windows having an inner frame and an outer frame, the display device displaying only a first portion and a second and different portion of an object, the first portion of the object being displayed within the first graphical window and the second portion of the object being different from the first portion and being displayed within the second graphical window, and the display device being further configured to detect one or more first and one or more second elements that are in contact with the display surface; and

a controller adapted to identify an initial position of one or more first elements contacting the inner frame or the outer frame of the displayed first graphical window, to identify a movement of the one or more first elements, to change at least one of a position, shape and size of the displayed first graphical window as a function of the identified initial position and movement of the one or more first elements; and

to manipulate a first display within the displayed first graphical window in accordance with one or more of a first plurality of modifications, the first plurality of modifications comprising;

a first modification implemented when the initial position of the one or more first elements is identified as contacting one of the inner frame and the outer frame of the displayed first graphical window, the first modification comprising manipulating the first display by controlling the display device to display a third portion of the object within the changed first graphical window, the third portion including portions of the object common to the first portion of the object and portions of the object not included in the first portion of the object, and the third portion being displayed such that positions on the display surface of the common portions of the first and the third portions of the object are substantially the same; and

a second modification implemented when the initial position of the one or more first elements is identified as contacting the other of the inner frame and the outer frame of the displayed first graphical window, the second modification comprising manipulating the first display by changing the first portion of the object by stretching or shrinking the horizontal or vertical extent of the first portion of the object in a same manner that the horizontal or vertical extent of the displayed first graphical window is stretched or shrunk and controlling the display device to display the changed first portion of the object within the stretched or shrunken first graphical window;

wherein the controller is further adapted to identify an initial position of one or more second elements contacting the inner frame or the outer frame of the displayed second graphical window, to identify a movement of the one or more second elements, to change at least one of a position, shape and size of the displayed second graphical window as a function of the identified initial position and movement of the one or more second elements; and

to manipulate a second display within the displayed second graphical window in accordance with one or more of a second plurality of modifications, the second plurality of modifications comprising;

a third modification implemented when the initial position of the one or more second elements is identified as contacting one of the inner frame and the outer frame of the displayed second graphical window, the third modification comprising manipulating the second display by controlling the display device to display a fourth portion of the object within the changed second graphical window, the fourth portion including portions of the object common to the second portion of the object and portions of the object not included in the second portion of the object, and the fourth portion being displayed such that positions on the display surface of the common portions of the second and the fourth portions of the object are substantially the same; and

a fourth modification implemented when the initial position of the one or more second elements is identified as contacting the other of the inner frame and the outer frame of the displayed second graphical window, the fourth modification comprising manipulating the second display by changing the second portion of the object by stretching or shrinking the horizontal or vertical extent of the second portion of the object in a same manner that the horizontal or vertical extent of the displayed second graphical window is stretched or shrunk and controlling the display device to display the changed second portion of the object within the stretched or shrunken second graphical window.