Methods of interfacing with multi-input devices and multi-input display systems employing interfacing techniques

• US 8,788,967 B2
• Filed: 04/08/2009
• Issued: 07/22/2014
• Est. Priority Date: 04/10/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: Davidson, Philip L., Han, Jefferson Y.
• Primary Examiner(s): Ng, Amy
• Assistant Examiner(s): Wong, William
• Application Number:
  US12/420,475
• Publication Number:
  US 20090259964A1
• Time in Patent Office: 1,931 Days
• Field of Search: 715/754, 715/788, 715/799, 715/800, 715/863, 345619-688
• US Class Current: 715/788
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Methods and systems for interfacing with multi-input devices employ various techniques for controlling the window framing of images. Such techniques provide control, including moving, sizing, and orientating, of one or more displayed window frames in which one or more images are displayed.

Claims

1. A method performed by a multi-input display device, the method comprising:

displaying on a multi-input display device a graphical window and at least a first portion of an image within the graphical window;

identifying a number, position and movement of one or more elements contacting the display device, wherein identifying a number and position of one or more elements comprises identifying a first element and a second element as contacting the display device initially at positions on a first edge of a frame of the displayed graphical window and identifying a third element as contacting the display device initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window;

changing a characteristic of the displayed graphical window in accordance with the identified number, position and movement of the one or more elements contacting the display device, the characteristic corresponding to a first characteristic when only one element is identified as contacting the display device, the first characteristic being a position of the displayed graphical window; and

the characteristic corresponding to a second characteristic when a plurality of elements are identified as contacting the display device, the second characteristic being at least one of a shape and size of the displayed graphical window, the first and second characteristics being different, wherein changing a characteristic comprises, when the first and the second elements are identified as contacting the display device initially at the positions on the first edge and the third element is identified as contacting the display device initially at the position on the second edge;

in response to identifying that the first and the second elements are contacting the display device initially at the positions on the first edge, locking the first edge in position;

moving the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element; and

maintaining the first edge of the frame at a fixed position; and

displaying on the display device at least a second portion of the image within the graphical window having the changed characteristic such that positions on the display device of common portions of the first and the second portions of the image remain stationary relative to the multi-input display device.

2. The method of claim 1,

wherein identifying a number and position of one or more elements comprises identifying two elements as contacting the display device initially at positions on the displayed graphical window, and

wherein changing a characteristic comprises, when the two elements are identified as contacting the display device initially at the positions on the displayed graphical window, uniformly scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

3. The method of claim 1,

wherein identifying a number and position of one or more elements comprises identifying two elements as contacting the display device initially at positions on the displayed graphical window, and

wherein changing a characteristic comprises, when the two elements are identified as contacting the display device initially at the positions on the displayed graphical window, moving and scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

4. The method of claim 3, wherein moving and scaling includes rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the two elements relative to one another.

5. The method of claim 1,

wherein identifying a number and position of one or more elements comprises identifying three elements as contacting the display device initially at positions on the frame of the displayed graphical window, and

wherein changing a characteristic comprises, when the three elements are identified as contacting the display device initially at the positions on the frame, manipulating a shape of the displayed graphical window based on changes in the identified positions of the three elements in accordance with a best fit methodology.

6. The method of claim 5, wherein manipulating the shape of the displayed graphical window based on changes in the identified positions of the three elements is a 5 degree of freedom operation.

7. The method of claim 1, wherein changing a characteristic comprises modifying the shape of the displayed graphical window by moving the first edge of the frame of the displayed graphical window in alignment with identified joint movement of the first and the second elements and maintaining the second edge of the frame in alignment with the identified position of the third element.

8. The method of claim 1, wherein changing a characteristic comprises changing a length of the first and the second edges of the frame of the displayed graphical window in accordance with a change in distance between the identified positions of the first and the second elements.

9. The method of claim 1, wherein changing a characteristic comprises rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the first and the second elements relative to one another so that the first edge of the frame of the displayed graphical window is aligned with an axis extending through the first and the second elements as the first and the second elements move.

10. The method of claim 1, wherein the displayed graphical window is rectangular in shape, and changing a characteristic comprises:

aligning the first edge of the frame of the displayed graphical window with an axis extending through the identified positions of the first and the second elements, wherein aligning the first edge includes at least one of keeping the first edge stationary, moving the first edge vertically, moving the first edge horizontally, and rotating the first edge;

changing a length of the first and the second edges of the frame in accordance with a change in distance between the identified positions of the first and the second elements; and

aligning the second edge of the flame with the identified position of the third element.

11. A method performed by a multi-input display device, the method comprising:

displaying on a multi-input display device a graphical window and at least a first portion of an image within the graphical window;

identifying a number, position and movement of a plurality of elements that are simultaneously in contact with the display device, wherein identifying a number and position of the plurality of elements comprises identifying a first element and a second element as being in simultaneous contact with the display device initially at positions on a first edge of a frame of the displayed graphical window and identifying a third element as being in simultaneous contact with the display device initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window;

manipulating the displayed graphical window in accordance with one or more of a plurality of modifications, the plurality of modifications comprising;

a first modification implemented when only two elements are identified as being simultaneously in contact with the display device at positions on the frame of the displayed graphical window, the first modification changing at least one of a shape and a size of the displayed graphical window in accordance with the identified positions and movement of the two elements, and

a second modification implemented when three elements are identified as being simultaneously in contact with the display device at positions on the frame of the displayed graphical window, the second modification changing at least one of a position, a shape and a size of the displayed graphical window in accordance with the identified positions and movement of the three elements, the first and the second modifications being different, wherein the second modification is implemented when the first and the second elements are identified as being in simultaneous contact with the display device initially at the positions on the first edge and the third element is identified as being in simultaneous contact with the display device initially at the position on the second edge, and wherein the second modification comprises;

in response to identifying the first and the second elements as being in simultaneous contact with the display device initially at the positions on the first edge, locking the first edge in position;

moving the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element; and

maintaining the first edge of the frame at a fixed position; and

displaying on the display device at least a second portion of the image within the manipulated graphical window such that positions on the display device of common portions of the first and the second portions of the image remain stationary relative to the multi-input display device.

12. The method of claim 11, wherein the first modification comprises uniformly scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

13. The method of claim 11, wherein the first modification comprises moving and scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

14. The method of claim 13, wherein the first modification includes rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the two elements relative to one another.

15. The method of claim 11, wherein the second modification comprises changing a shape of the displayed graphical window based on changes in the identified positions of the three elements in accordance with a best fit methodology.

16. The method of claim 15, wherein changing the shape of the displayed graphical window based on changes in the identified positions of the three elements is a 5 degree of freedom operation.

17. The method of claim 11, wherein the second modification comprises modifying the shape of the displayed graphical window by moving the first edge of the frame of the displayed graphical window in alignment with identified joint movement of the first and the second elements and maintaining the second edge of the frame in alignment with the identified position of the third element.

18. The method of claim 11, wherein the second modification comprises changing a length of the first and the second edges of the frame of the displayed graphical window in accordance with a change in distance between the identified positions of the first and the second elements.

19. The method of claim 11, wherein the second modification comprises rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the first and the second elements relative to one another so that the first edge of the frame of the displayed graphical window is aligned with an axis extending through the first and the second elements as the first and the second elements move.

20. The method of claim 11, wherein the displayed graphical window is rectangular in shape, and the second modification comprises:

aligning the first edge of the frame of the displayed graphical window with an axis extending through the identified positions of the first and the second elements, wherein aligning the first edge includes at least one of keeping the first edge stationary, moving the first edge vertically, moving the first edge horizontally, and rotating the first edge;

changing a length of the first and the second edges of the frame in accordance with a change in distance between the identified positions of the first and the second elements; and

aligning the second edge of the frame with the identified position of the third element.

21. A multi-input display system, comprising:

a display device configured to display on a display surface a graphical window and at least a first portion of an image within the graphical window and detect one or more elements contacting the display surface; and

a controller configured to;

identify a number, position and movement of the one or more elements contacting the display surface, wherein the controller being configured to identify a number and position of the one or more elements comprises the controller being configured to identify a first element and a second element as contacting the display device initially at positions on a first edge of a frame of the displayed graphical window, and to identify a third element as contacting the display device initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window; and

change a characteristic of the displayed graphical window in accordance with the identified number, position and movement of the one or more elements contacting the display surface, the characteristic corresponding to a first characteristic when only one element is identified as contacting the display surface, the first characteristic being a position of the displayed graphical window; and

the characteristic corresponding to a second characteristic when a plurality of elements are identified as contacting the display surface, the second characteristic being at least one of a shape and size of the displayed graphical window, the first and the second characteristics being different, wherein the controller being configured to change a characteristic comprises, when the first and the second elements are identified as contacting the display device initially at the positions on the first edge and the third element is identified as contacting the display device initially at the position on the second edge;

in response to identifying that the first and the second elements are contacting the display device initially at the positions on the first edge, the controller being configured to lock the first edge in position;

the controller being configured to move the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element; and

the controller being configured to maintain the first edge of the frame at a fixed position;

wherein the controller is configured to control the display device to display on the display surface at least a second portion of the image within the graphical window having the changed characteristic such that positions on the display surface of common portions of the first and the second portions of the image remain stationary relative to the display surface.

22. The system of claim 21,

wherein the controller being configured to identify a number and position of one or more elements comprises the controller being configured to identify two elements as contacting the display surface initially at positions on the displayed graphical window, and

wherein the controller being configured to change a characteristic comprises the controller being configured to, when the two elements are identified as contacting the display surface initially at the positions on the graphical window, uniformly scale the size of the displayed graphical window in accordance with the identified movement of the two elements.

23. The system of claim 21,

wherein the controller being configured to identify a number and position of one or more elements comprises the controller being configured to identify two elements as contacting the display surface initially at positions on the displayed graphical window, and

wherein the controller being configured to change a characteristic comprises the controller being configured to, when the two elements are identified as contacting the display surface initially at the positions on the graphical window, move and scale the size of the displayed graphical window in accordance with the identified movement of the two elements.

24. The system of claim 23, wherein the controller is configured to rotate the displayed graphical window in accordance with changes in orientation of the identified positions of the two elements relative to one another.

25. The system of claim 21,

wherein the controller being configured to identify a number and position of one or more elements comprises the controller being configured to identify three elements as contacting the display surface initially at positions on the frame of the displayed graphical window, and

wherein the controller being configured to change a characteristic comprises the controller being configured to, when the three elements are identified as contacting the display surface initially at positions on the frame of the graphical window, manipulate a shape of the displayed graphical window based on changes in the identified positions of the three elements in accordance with a best fit methodology.

26. The system of claim 25, wherein the controller is configured to manipulate the shape of the graphical window in accordance with a 5 degree of freedom operation.

27. The system of claim 21, wherein the controller is configured to modify the shape of the displayed graphical window by moving the first edge of the frame of the displayed graphical window in alignment with identified joint movement of the first and the second elements and maintaining the second edge of the frame in alignment with the identified position of the third element.

28. The system of claim 21, wherein the controller is configured to change a length of the first and second edges of the frame of the displayed graphical window in accordance with a change in distance between the identified positions of the first and the second elements.

29. The system of claim 21, wherein the controller is configured to rotate the displayed graphical window in accordance with changes in orientation of the identified positions of the first and the second elements relative to one another so that the first edge of the frame of the displayed graphical window is aligned with an axis extending through the first and the second elements as the first and the second elements move.

30. The system of claim 21, wherein the displayed graphical window is rectangular in shape, and the controller is adapted to:

align the first edge of the frame of the displayed graphical window with an axis extending through the identified positions of the first and the second elements, wherein aligning the first edge includes at least one of keeping the first edge stationary, moving the first edge vertically, moving the first edge horizontally, and rotating the first edge;

change a length of the first and the second edges of the frame in accordance with a change in distance between the identified positions of the first and the second elements; and

align the second edge of the frame with the identified position of the third element.

31. A multi-input display system, comprising:

a display device configured to display on a display surface a graphical window and at least a first portion of an image within the graphical window, and detect a plurality of elements that are simultaneously in contact with the display surface;

a controller configured to identify a number, position and movement of the plurality of elements that are simultaneously in contact with the display surface, wherein the controller being configured to identify a number and position of the plurality of elements comprises the controller being configured to identify a first element and a second element as being in simultaneous contact with the display surface initially at positions on a first edge of a frame of the displayed graphical window, and the controller being configured to identify a third element as being in simultaneous contact with the display surface initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window,

and the controller is configured to manipulate the displayed graphical window in accordance with one or more of a plurality of modifications, the plurality of modifications comprising;

a first modification implemented when only two elements are identified as being simultaneously in contact with the display surface at positions on the frame of the displayed graphical window, the first modification changing at least one of a shape and a size of the displayed graphical window in accordance with the identified positions and movement of the two elements and

a second modification implemented when three elements are identified as being simultaneously in contact with the display surface at positions on the frame of the displayed graphical window, the second modification changing at least one of a position, a shape and a size of the displayed graphical window in accordance with the identified positions and movement of the three elements, the first and the second modifications being different, wherein the controller is configured to implement the second modification when the first and the second elements are identified as being in simultaneous contact with the display surface initially at the positions on the first edge and the third element is identified as being in simultaneous contact with the display surface initially at the position on the second edge, and wherein the second modification comprises;

in response to identifying the first and the second elements as being in simultaneous contact with the display device initially at the positions on the first edge, locking the first edge in position;

moving the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element, and

maintaining the first edge of the frame at a fixed position,

wherein the controller is configured to control the display device to display on the display surface at least a second portion of the image within the manipulated graphical window such that positions on the display surface of common portions of the first and second portions of the image remain stationary relative to the display surface.

32. The system of claim 31, wherein the first modification comprises uniformly scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

33. The system of claim 31, wherein the first modification comprises moving and scaling the size of the displayed graphical window in accordance with the identified movement of the two elements.

34. The system of claim 33, wherein the first modification includes rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the two elements relative to one another.

35. The system of claim 31, wherein the second modification comprises changing a shape of the displayed graphical window based on changes in the identified positions of the three elements in accordance with a best fit methodology.

36. The system of claim 35, wherein changing the shape of the displayed graphical window based on changes in the identified positions of the three elements is a 5 degree of freedom operation.

37. The system of claim 31, wherein the second modification comprises modifying the shape of the displayed graphical window by moving the first edge of the frame of the displayed graphical window in alignment with identified joint movement of the first and the second elements and maintaining the second edge of the frame in alignment with the identified position of the third element.

38. The system of claim 37, wherein the second modification comprises changing a length of the first and the second edges of the frame of the displayed graphical window in accordance with a change in distance between the identified positions of the first and the second elements.

39. The system of claim 31, wherein the second modification comprises rotating the displayed graphical window in accordance with changes in orientation of the identified positions of the first and the second elements relative to one another so that the first edge of the frame of the displayed graphical window is aligned with an axis extending through the first and the second elements as the first and the second elements move.

40. The system of claim 31, wherein the displayed graphical window is rectangular in shape, and the second modification comprises:

aligning the first edge of the frame of the displayed graphical window with an axis extending through the identified positions of the first and second elements, wherein aligning the first edge includes at least one of keeping the first edge stationary, moving the first edge vertically, moving the first edge horizontally, and rotating the first edge;

changing a length of the first and the second edges of the frame in accordance with a change in distance between the identified positions of the first and the second elements; and

aligning the second edge of the frame with the identified position of the third element.

41. A method performed by a multi-input display device, the method comprising:

displaying a window on a display surface of the multi-input display device, the window being re-sizeable, re-shapeable, and movable relative to the display surface;

displaying a visible portion of an image within the window, the image having a stationary position relative to the display surface, wherein the visible portion of the image depends on a size, shape and position of the window relative to the display surface;

identifying a quantity, position, and movement of one or more elements contacting the display surface, wherein identifying a quantity and position of one or more elements comprises identifying a first element and a second element as contacting the display device initially at positions on a first edge of a frame of the displayed graphical window and identifying a third element as contacting the display device initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window; and

changing a characteristic of the displayed window in accordance with the identified quantity, position, and movement of the one or more elements contacting the display surface, wherein the changed characteristic corresponds to the position of the window relative to the display surface provided that only one element is identified as contacting the display surface, and the changed characteristic corresponds to at least one of the size and the shape of the window provided that a plurality of elements is identified as contacting the display surface, wherein changing a characteristic comprises, when the first and the second elements are identified as contacting the display device initially at the positions on the first edge and the third element is identified as contacting the display device initially at the position on the second edge;

in response to identifying that the first and the second elements are contacting the display device initially at the positions on the first edge, locking the first edge in position;

moving the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element; and

maintaining the first edge of the frame at a fixed position,

wherein changing the characteristic of the displayed window concurrently changes the visible portion of the image without changing the stationary position of the image relative to the display surface.

42. A method performed by a multi-input display device, the method comprising:

displaying a window on a display surface of the multi-input display device, the window being re-sizeable, re-shapeable, and movable relative to the display surface;

displaying a visible portion of an image within the window, the image having a stationary position relative to the display surface, wherein the visible portion of the image depends on a size, shape and position of the window relative to the display surface;

identifying a quantity, position, and movement of two or more elements contacting the display surface, wherein identifying a quantity and position of two or more elements comprises identifying a first element and a second element as contacting the display device initially at positions on a first edge of a frame of the displayed graphical window and identifying a third element as contacting the display device initially at a position on a second edge opposite the first edge of the frame of the displayed graphical window; and

changing a characteristic of the displayed window in accordance with the identified quantity, position, and movement of the two or more elements contacting the display surface, wherein the changed characteristic corresponds (i) to at least one of the size and the shape of the window relative to the display surface provided that two elements are identified as simultaneously contacting the display surface, and (ii) to at least one of the position, the size, and the shape of the window provided that three elements are identified as contacting the display surface, wherein changing a characteristic comprises, when the first and the second elements are identified as contacting the display device initially at the positions on the first edge and the third element is identified as contacting the display device initially at the position on the second edge;

in response to identifying that the first and the second elements are contacting the display device initially at the positions on the first edge, locking the first edge in position;

moving the second edge of the frame of the displayed graphical window in alignment with the identified movement of the third element; and

maintaining the first edge of the frame at a fixed position, and

wherein changing the characteristic of the displayed window concurrently changes the visible portion of the image without changing the stationary position of the image relative to the display surface.