Using physical objects to adjust attributes of an interactive display application

• US 7,743,348 B2
• Filed: 06/30/2004
• Issued: 06/22/2010
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Drucker, Steven M., Dehlin, Joel P., Wilson, Andrew D., Robbins, Daniel C., Nye, Kristen Nilsen
• Primary Examiner(s): Huynh; Ba
• Assistant Examiner(s): Augustine; Nicholas
• Application Number:
  US10/883,515
• Publication Number:
  US 20060001650A1
• Time in Patent Office: 2,183 Days
• Field of Search: 715/863, 715/767, 715/773, 715/810, 715/839
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Input is provided to an application using a plurality of physical objects disposed adjacent to an interactive display surface. A primary location is determined where a primary physical object, e.g., a finger or thumb of the user, is positioned adjacent to the interactive display surface. An additional location is determined where an additional physical object is positioned adjacent to the interactive display surface. The attribute might be a size of an image or selected portion of the image that will be retained after cropping. A change in position of at least one of the objects is detected, and the attribute is adjusted based on the change in position of one or both objects. A range of selectable options of the application can also be display by touching the interactive display surface with one'"'"'s fingers or other objects, and one of the options can be selected with another object.

Claims

1. In a computing environment including a computer system and an interactive display surface, a method for manipulating attributes of an object in a software application by providing physical inputs via the interactive display surface, the method comprising the steps of:

(a) executing the software application on the computer system, the application having at least one object with attributes that can be adjusted during execution of the application based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the object on the interactive display surface;

(b) determining a primary location adjacent to the interactive display surface where a primary physical object has been positioned by a user;

(c) determining a second location adjacent to the interactive display surface where a second physical object has been positioned by a user, the primary location and the secondary location forming a physical object layout on the interactive display surface;

(d) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface, which type of operation is to be initiated to adjust the attributes of the object, the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the object;

(e) detecting a change in position of at least one of the primary location and the second location; and

(f) initiating the determined type of operation to adjust the attribute according to the change in at least one of the primary and the second location, the change corresponding to at least one of the first and the second position.

2. The method of claim 1, wherein the steps of determining the primary location, the second location, and the change in position of at least one of the primary location and the second location comprise the steps of:

(a) transmitting infrared (IR) light through the interactive display surface toward a face of the interactive display surface adjacent to which the primary physical object and the secondary physical object are disposed;

(b) detecting the primary location by sensing IR light reflected from the primary physical object when the primary object has been positioned adjacent to the interactive display surface;

(c) detecting the second location by sensing IR light reflected from the second physical object when the secondary object has been positioned adjacent to the interactive display surface; and

(d) detecting the change in position of at least one of the primary location and the second location by determining any change in position of the primary physical object and the secondary physical object, based upon the IR light that was sensed.

3. The method of claim 1, wherein at least one of the primary physical object and the second physical object includes a digit on one of a user'"'"'s hands.

4. The method of claim 1, wherein the attribute includes a size of a virtual entity displayed on the interactive display surface, and the step of adjusting the attribute comprises the steps of:

(a) determining that one of the primary location and the second location is disposed at one of a boundary of the virtual entity and within the virtual entity;

(b) determining that the primary location and the second location are disposed in one of a substantially vertical alignment and a substantially horizontal alignment with each other; and

(c) resizing the virtual entity so that its extents in at least one dimension are defined by the primary location and the second location, a center of the virtual entity being positioned at a midpoint between the primary location and the second location.

5. The method of claim 4, wherein the virtual entity comprises an image.

6. The method of claim 4, wherein an aspect ratio of the virtual entity is preserved as the size of the virtual entity is changed.

7. The method of claim 4, wherein an aspect ratio of the virtual entity is changed as the size of the virtual entity is changed.

8. The method of claim 1, wherein the attribute includes a portion of a virtual entity displayed on the interactive display surface, and the step of adjusting the attribute comprises the steps of:

(a) determining that one of the primary location and the second location is disposed at one of a boundary of the virtual entity and within the virtual entity;

(b) determining that the primary location and the second location are disposed neither in a substantially vertical alignment nor a substantially horizontal alignment with each other; and

(c) cropping the virtual entity to eliminate all but a portion of the virtual entity with diagonally opposite corners corresponding to the primary location and the second location.

9. The method of claim 1, wherein the step of adjusting the attribute comprises the steps of:

(a) presenting a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(b) associating the second location with a property option selected from the range of property options, the property option selected being a property option closest to the second location; and

(c) adjusting the attribute in accordance with the property option selected.

10. The method of claim 9, further comprising the steps of:

(a) detecting a secondary location adjacent to the interactive display surface where a secondary physical object has been disposed; and

(b) adjusting a size of the property selection range to generally fit a space defined by the primary location and the secondary location.

11. The method of claim 10, wherein the primary physical object and the secondary physical object include digits on a user'"'"'s hand.

12. The method of claim 11, wherein the second physical object includes a digit on a user'"'"'s hand.

13. The method of claim 1, wherein the step of adjusting the attribute further comprises the steps of:

(a) presenting a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(b) associating the second location with a property option selected from the range of property options, the property option selected being closest to the second location;

(c) presenting an additional property selection range adjacent to the property selected, the additional property selection range including a range of additional property options associated with the property option selected, which are within the additional property selection range;

(d) associating an additional property option selected among the range of additional property options with a changed second location, the additional property option selected being closest to the changed second location; and

(e) adjusting the attribute in accordance with the additional property option selected.

47. The method of claim 1, further comprising upon determining type of operation that is to be performed, displaying dynamic feedback to the user representing the type of operation that is to be performed as a result of the placement of the second physical object.

14. A method for adjusting a presentation of an image on an interactive display surface that is coupled to a computer system, comprising the steps of:

(a) executing an image presentation application on the computer system, the image presentation application presenting at least one image on the interactive display surface, the image having attributes that can be adjusted based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the image on the interactive display surface;

(b) determining a primary location adjacent to the interactive display surface where a primary physical object has been disposed;

(c) determining a second location adjacent to the interactive display surface where a second physical object has been disposed, the primary location and the secondary location forming an object layout on the interactive display surface;

(d) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface, which type of operation is to be initiated to adjust the boundaries of the image the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the image on the interactive display surface;

(e) detecting any change in a position of at least one of the primary location and the second location; and

(f) initiating the determined type of operation to adjust the Previously Presented boundaries of the image according to the change in at least one of the primary and the second location, the change corresponding to at least one of the first and the second position.

15. The method of claim 14, wherein the steps of determining the primary location, the second location, and the change in position of at least one of the primary location and the second location comprise the steps of:

(a) transmitting infrared (IR) light through the interactive display surface toward a face of the interactive display surface adjacent to which the primary physical object and the secondary physical object are disposed;

(b) detecting the primary location by sensing IR light reflected from the primary physical object when the primary object has been positioned adjacent to the interactive display surface;

(c) detecting the second location by sensing IR light reflected from the second physical object when the secondary object has been positioned adjacent to the interactive display surface; and

(d) detecting the change in position of at least one of the primary location and the second location by determining any change in position of the primary physical object and the secondary physical object, based upon the IR light that was sensed.

16. The method of claim 14, wherein at least one of the primary physical object and the second physical object includes a digit on one of a user'"'"'s hands.

17. The method of claim 14, wherein the step of associating Previously Presented boundaries of the image comprises the step of changing a size of the image by:

(a) determining that the primary location and the second location are generally one of vertically aligned and horizontally aligned with each other;

(b) setting opposite boundaries of the image to correspond with the primary location and the second location, respectively; and

(c) moving at least one Previously Presented boundary of the image to correspond with any change in the position of the primary location and an opposite Previously Presented boundary to correspond with any change in position of the second location, to change the size of the image.

18. The method of claim 17, wherein a center of the image remains at a midpoint between the primary location and the second location as any change in at least one of the primary location and the second location occurs.

19. The method of claim 14, wherein an aspect of the ratio of the image is preserved as the size of the image is changed.

20. The method of claim 14, wherein an aspect ratio of the image is changed as the size of the image is changed.

21. The method of claim 14, further comprising the step of changing a portion of the image displayed on the interactive display surface by:

(a) determining that the primary location and the second location are neither generally vertically aligned nor generally horizontally aligned with each other;

(b) forming Previously Presented image boundaries with diagonally opposite corners corresponding to the primary location and the second location, respectively;

(c) adjusting at least one of the diagonally opposite corners to correspond to any change in the position of at least one of the primary location and the second location; and

(d) cropping the image to eliminate all but the portion of the image bounded by the Previously Presented image boundary.

22. A method for providing input to an application executing on a computer system coupled to an interactive display surface, comprising the steps of:

(a) executing an application on the computer system, the application having at least one object with attributes that can be adjusted during execution of the application based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the object on the interactive display surface;

(b) determining a primary location adjacent to the interactive display surface where a primary physical object has been disposed;

(c) presenting a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(d) determining a second location adjacent to the interactive display surface where a second physical object has been disposed, the primary location and the secondary location forming an object layout on the interactive display surface;

(e) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface, which type of operation is to be initiated to adjust the attributes of the object, the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the object; and

(f) adjusting the attribute in accordance with the selected property option.

23. The method of claim 22, wherein the steps of determining the primary location, the second location, and the change in position of at least one of the primary location and the second location, comprise the steps of:

(a) transmitting infrared (IR) light through the interactive display surface toward a face of the interactive display surface adjacent to which the primary physical object and the secondary physical object are disposed;

(b) detecting the primary location by sensing IR light reflected from the primary physical object when the primary object has been disposed adjacent to the interactive display surface;

(c) detecting the second location by sensing IR light reflected from the second physical object when the secondary object has been disposed adjacent to the interactive display surface; and

(d) detecting any change in position of at least one of the primary location and the second location by determining any change in the primary location and the second location, by determining any change in position of the primary physical object and the secondary physical object, based upon the IR light that was sensed.

24. The method of claim 22, further comprising the steps of:

(a) determining a secondary location adjacent to the interactive display surface where a secondary physical object has been disposed; and

(b) adjusting a size of the property selection range to generally fit a space defined by the primary location and the secondary location.

25. The method of claim 24, wherein the primary physical object, the second physical object, and the secondary physical object include digits on a user'"'"'s hand.

26. The method of claim 22, wherein the step of adjusting the attribute comprises the steps of:

(a) presenting a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(b) associating the second location with a property option selected among the range of property options, the property option being closest to the second location;

(c) presenting an additional property selection range adjacent to the property option selected, the additional property selection range including a range of additional property options associated with the property option selected, and included within the additional property selection range;

(d) associating an additional property option selected from the range of additional property options with a changed second location, the additional property option selected being closest to the changed second location; and

(e) adjusting the attribute in accordance with the additional property option selected.

27. A system for providing input to an application that is being executed, comprising:

(a) an interactive display surface adjacent to which a physical object is manipulated, said interactive display surface diffusing light, and having a processing side and an interactive side from which the image is viewed and adjacent to which the physical object can be disposed, the processing side being opposite to the interactive side;

(b) a projector that projects virtual entities onto the processing side of the interactive display surface, said virtual entities being visible from the interactive side;

(c) a light source disposed on the processing side of the interactive display surface, the light source emitting infrared (IR) light that is transmitted through the interactive display surface to the interactive side and reflected back through the interactive display surface by the physical object that is disposed adjacent to the interactive side of the interactive display surface;

(d) a light sensor disposed on the processing side of the interactive display surface, the light sensor sensing IR light reflected back from the physical object through the interactive display surface and imaging the interactive display surface to detect the physical object and its location;

(e) a processor in communication with the light sensor and the projector; and

(f) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) executing an application on the computer system, the application having at least one object with attributes that can be changed during execution of the application based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the object on the interactive display surface;

(ii) determining a primary location adjacent to the interactive display surface where a primary physical object has been positioned by a user;

(iii) determining a second location adjacent to the interactive display surface where a second physical object has been positioned by a user, the primary location and the secondary location forming a physical object layout on the interactive display surface;

(iv) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the object on the interactive display surface, which type of operation is to be initiated to adjust the attributes of the object, the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the object;

(v) detecting a change in position of at least one of the primary location and the second location; and

(vi) initiating the determined type of operation to adjust the attribute according to the change in at least one of the primary and the second location, the change corresponding to at least one of the first and the second position.

28. The system of claim 27, wherein at least one of the primary physical object and the second physical object includes a digit on one of a user'"'"'s hands.

29. The system of claim 27, wherein the attribute includes a size of the virtual entity displayed on the interactive display surface, and the machine language instructions further cause the processor to:

(a) determine that one of the primary location and the second location is disposed at one of a boundary of the virtual entity and within the virtual entity;

(b) determine that the primary location and the second location are disposed in one of a substantially vertical alignment and a substantially horizontal alignment with each other; and

(c) resize the virtual entity so that its extents in at least one dimension are defined by the primary location and the second location, a center of the virtual entity being positioned at a midpoint between the primary location and the second location.

30. The system of claim 29, wherein the virtual entity comprises an image.

31. The system of claim 29, wherein the machine language instructions further cause the processor to preserve an aspect ratio of the virtual entity as the size of the virtual entity is changed.

32. The system of claim 29, wherein the machine language instructions further cause the processor to change an aspect ratio of the virtual entity as the size of the virtual entity is changed.

33. The system of claim 27, wherein the attribute includes a portion of a virtual entity displayed on the interactive display surface, and the machine language instructions further cause the processor to

(a) determine that one of the primary location and the second location is disposed at one of a boundary of the virtual entity and within the virtual entity;

(b) determine that the primary location and the second location are disposed neither in a substantially vertical alignment nor a substantially horizontal alignment with each other; and

(c) crop the virtual entity to eliminate all but a portion of the virtual entity with diagonally opposite corners corresponding to the primary location and the second location.

34. The system of claim 27, wherein the machine language instructions further cause the processor to:

(a) present a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(b) associate the second location with a property option selected from the range of property options, the property option selected being closest to the second location; and

(c) adjust the attribute in accordance with the property option selected.

35. The system of claim 34, wherein the machine language instructions further cause the processor to:

(a) detect a secondary location adjacent to the interactive display surface where a secondary physical object has been disposed; and

(b) adjust a size of the property selection range to generally fit a space defined by the primary location and the secondary location.

36. The system of claim 35, wherein the primary physical object and the secondary physical object include digits on a user'"'"'s hand.

37. The system of claim 35, wherein the second physical object includes a digit on a user'"'"'s hand.

38. The system of claim 27, wherein the machine language instructions further cause the processor to:

(a) present a property selection range adjacent to the primary location, the property selection range including a range of property options within the property selection range;

(b) associate the second location with a property option selected from the range of property options, the property option selected being closest to the second location;

(c) present an additional property selection range adjacent to the property option selected, the additional property selection range including a range of additional property options associated with the property option selected, and included in the additional property selection range;

(d) associate an additional property option selected from the range of additional property option with a changed second location, the additional property option selected being closest to the changed second location; and

(e) adjust the attribute in accordance with the additional property option selected.

39. A system for adjusting a presentation of an image, comprising:

(a) an interactive display surface adjacent to which a physical object is manipulated, said interactive display surface diffusing light, and having a processing side and an interactive side from which the image is viewed and adjacent to which the physical object can be placed, the processing side being opposite to the interactive side;

(b) a projector that projects graphic images onto the processing side of the interactive display surface, said graphic images being visible from the interactive side;

(c) a light source disposed on the processing side of the display surface, the light source emitting infrared (IR) light that is transmitted through the display surface to the interactive side and reflected back through the interactive display surface by the physical object that is disposed adjacent to the interactive side of the interactive display surface;

(d) a light sensor disposed on the processing side of the interactive display surface, the light sensor sensing IR light reflected back from the physical object through the interactive display surface and imaging the interactive display surface to detect the physical object and its location;

(e) a processor in communication with the light sensor and the projector; and

(f) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) executing an image presentation application on the computer system, the image presentation application presenting at least one image on the interactive display surface using the projector, the image having attributes that can be adjusted based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the image on the interactive display surface;

(ii) using the light sensor for determining a primary location adjacent to the interactive display surface where a primary physical object has been disposed;

(iii) using the light sensor for determining a second location adjacent to the interactive display surface where a second physical object has been disposed, the primary location and the secondary location forming an object layout on the interactive display surface;

(iv) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface, which type of operation is to be initiated to adjust the attributes of the object, the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the image;

(v) using the light sensor for detecting any change in a position of at least one of the primary location and the second location; and

(vi) initiating the determined type of operation to adjust the Previously Presented boundaries of the image according to the change in at least one of the primary and the second location, the change corresponding to at least one of the first and the second position.

40. The system of claim 39, wherein at least one of the primary physical object and the second physical object includes a digit on one of a user'"'"'s hands.

41. The system of claim 39, wherein the attribute includes a size of an image displayed on the interactive display surface, and the machine language instructions further cause the processor to:

(a) determine that the primary location and the second location are generally one of vertically aligned and horizontally aligned with each other;

(b) set opposite boundaries of the image to correspond with the primary location and the second location, respectively; and

(c) move at least one Previously Presented boundary of the image to correspond with any change in the position of the primary location and an opposite Previously Presented boundary to correspond with any change in position of the second location, to change the size of the image.

42. The system of claim 41, wherein a center of the image remains at a midpoint between the primary location and the second location as any change in at least one of the primary location and the second location occurs.

43. The system of claim 41, wherein the machine language instructions further cause the processor to preserve an aspect ratio of the image as the size of the image is changed.

44. The system of claim 41, wherein the machine language instructions further cause the processor to change an aspect ratio of the image as the size of the image is changed.

45. The system of claim 39, wherein the attribute includes a portion of an image displayed on the interactive display surface, and the machine language instructions further cause the processor to:

(a) determine that the primary location and the second location are neither generally vertically aligned nor generally horizontally aligned with each other;

(b) form Previously Presented image boundaries with diagonally opposite corners corresponding to the primary location and the second location, respectively;

(c) adjust at least one of the diagonally opposite corners to correspond to any change in the position of at least one of the primary location and the second location; and

(d) crop the image to eliminate all but the portion of the image bounded by the Previously Presented image boundary.

46. In a computing environment including a computer system and an interactive display surface, a method for rotating an image in a software application by providing physical inputs via the interactive display surface, the method comprising the steps of:

(a) executing the software application on the computer system, the application having at least one image with attributes that can be adjusted during execution of the application based on physical inputs inputted by a computer system user, wherein the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface determines the manner in which the attributes are to be adjusted, such that two of the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the visual depiction of the image on the interactive display surface;

(b) determining a primary location adjacent to the interactive display surface where a primary physical object has been positioned by a user;

(c) determining a second location adjacent to the interactive display surface where a second physical object has been positioned by a user, the primary location and the secondary location forming a physical object layout on the interactive display surface;

(d) determining, based on the physical object layout, the order in which the primary and second physical objects were placed and the positioning of the physical inputs relative to the visual depiction of the image on the interactive display surface, which type of operation is to be initiated to adjust the attributes of the object, the operation type being variable and selectable from a plurality of operation types corresponding to various object layouts, wherein the same physical inputs can initiate different attribute adjustments without altering application input configuration settings based on where the two inputs are positioned relative to the image;

(e) detecting a change in position of at least one of the primary location and the second location; and

(f) initiating the determined type of operation to rotate the image according to the change in at least one of the primary and the second location, the change corresponding to at least one of the first and the second position.