Using a physical object to control an attribute of an interactive display application

• US 7,379,047 B2
• Filed: 06/30/2004
• Issued: 05/27/2008
• Est. Priority Date: 06/30/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dehlin, Joel, Wong, Curtis, Roseway, Asta, Drucker, Steven M.
• Primary Examiner(s): Hjerpe; Richard
• Assistant Examiner(s): Shapiro; Leonid
• Application Number:
  US10/883,253
• Publication Number:
  US 20060001645A1
• Time in Patent Office: 1,427 Days
• Field of Search: 345/156, 345/163, 345/175
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]

Abstract

An input to a computer system is made by manipulating a physical object disposed adjacent to an interactive display surface. An application having at least one changeable attribute is executed on the computer system. When the physical object is disposed adjacent to the interactive display surface, it is identified and its location is determined based upon a shape, a size, an infrared reflectance, or some other optical quality. The initial location and orientation of the physical object on the interactive display surface relative to the interactive display surface is identified. The attribute of the application is correlated with the physical object and is changed in response to movement of the physical object. The physical object can thus be used, for example, to select menu options and parameters within the application being executed.

Claims

1. A method for providing input to a computer system having an interactive display surface by manipulating a physical object disposed adjacent to the interactive display surface, the method comprising the steps of:

executing an application on the computer system, the application having at least one attribute changeable while the application is executing;

detecting a location and an orientation of the physical object relative to the interactive display surface based upon a characteristic of the physical object when the physical object is disposed adjacent to the interactive display surface, the characteristic including at least one of a shape, a size, and an optical quality as well as at least one of the orientation of the object and the location of the physical object, such that at least one of rotating and translating the physical object changes the attribute;

correlating the attribute with the orientation of the physical object;

detecting a change in the location of the physical object on the interactive display surface;

adjusting a scaling factor based on the change in the location of the physical object on the interactive display surface;

changing the attribute in response to at least one of the location and a movement of the physical object relative to the interactive display surface; and

applying the scaling factor to changes in the attribute associated with rotation of the physical object.

2. The method of claim 1, further comprising the step of determining the characteristic of the physical object, said step of determining the characteristic of the physical object comprising the steps of:

(a) transmitting infrared light through the interactive display surface toward a face of the interactive display surface adjacent to which the physical object is disposed; and

(b) determining the characteristic of the physical object by sensing infrared light reflected from the physical object back through the interactive display surface.

3. The method of claim 2, further comprising determining the characteristic of the physical object by creating an ellipse representative of the characteristic of the physical object, the ellipse having an equivalent statistical center, spatial moments of points on the physical object reflecting infrared light through the interactive display surface and thus defining the characteristic.

4. The method of claim 3, wherein the points defining the characteristic of the physical object include adjoining points meeting a predefined condition.

5. The method of claim 2, further comprising determining the characteristic of the physical object by creating an ellipse representative of the characteristic of the physical object, the ellipse having dimensions large enough to encompass an outermost point of all points defining the characteristic.

6. The method of claim 5, wherein the outermost points include points not having adjoining points in all considered directions.

7. The method of claim 1, wherein the attribute is correlated with the characteristic of the physical object by:

(a) associating the attribute with a region of the interactive display surface; and

(b) detecting when the physical object is disposed adjacent to the region of the interactive display surface associated with the attribute.

8. The method of claim 1, further comprising the steps of:

(a) displaying on the interactive display surface a plurality of indicators respectively representing a plurality of changeable program attributes; and

(b) enabling a user to select a desired attribute among the plurality of changeable program attributes by one of rotating the physical object and translating the physical object toward a desired indicator representing the desired secondary attribute.

9. The method of claim 8, further comprising the steps of:

(a) displaying on the interactive display surface a plurality of secondary indicators representing a plurality of secondary changeable program attributes associated with the desired indicator; and

(b) enabling a user to select a desired secondary attribute among the plurality of secondary changeable program attributes by one of rotating the physical object and translating the physical object toward a desired secondary indicator.

10. The method of claim 8, further comprising the steps of:

(a) detecting selection of the desired attribute; and

(b) enabling a user to change a value of the changeable program attribute selected by one of rotating the physical object and translating the physical object.

11. The method of claim 1, further comprising the steps of:

(a) associating a plurality of data elements with the physical object;

(b) depicting the data elements with representations displayed on the interactive display surface adjacent a position of the physical object; and

(c) at least one of revolving and moving the representations to correspond with at least one of a rotation and a translation of the physical object.

12. The method of claim 1, further comprising the step of generating an audio signal when the physical object is disposed adjacent to the interactive display surface, the audio signal generated being associated with one of the location and the shape of the physical object.

13. The method of claim 12, wherein the audio signal generated is generated from a predetermined list of audio signals, each being associated with a different shape of a physical object.

14. The method of claim 12, wherein the audio signal generated is varied according to the position on the interactive display surface where the physical object is disposed.

15. The method of claim 12, wherein the interactive display surface is partitioned into a plurality of regions, each of the regions being associated with a different category of audio signals associated with different shapes of physical objects.

16. The method of claim 12, further comprising the step of adjusting a property of the audio signal generated upon detecting at least one of a rotation and a translation of the physical object.

17. The method of claim 16, wherein the property of the audio signal includes at least one of a pitch, a tempo, a sustain, a repetition, and a harmonic quality.

18. A memory medium on which are stored machine instructions for carrying out the steps of claim 1.

19. A method for providing input to a computer system having an interactive display surface by manipulating a physical object disposed adjacent to the interactive display surface, the method comprising the steps of:

executing an application on the computer system, the application having at least one attribute changeable while the application is executing;

determining a characteristic of the physical object by identifying an ellipse representative of the characteristic of the physical object;

correlating the attribute with a position of the ellipse, the position of the ellipse including at least one of an orientation and a location relative to the interactive display surface;

determining an initial position of the ellipse; and

detecting a change in the location of the ellipse relative to the interactive display surface;

adjusting a scaling factor based on the change in the location of the ellipse relative to the interactive display surface;

changing the attribute in response to a change in position of the ellipse; and

applying the scaling factor to changes in the attribute associated with the rotation of the ellipse.

20. The method of claim 19, further comprising the step of determining the characteristic of the physical object, said step of determining the characteristic comprising the steps of:

(a) transmitting infrared light through the interactive display surface toward a face of the interactive display surface adjacent to which the physical object is disposed; and

(b) determining the characteristic of the physical object by sensing infrared light reflected from the physical object back through the interactive display surface.

21. The method of claim 20, wherein the ellipse representative of the characteristic of the physical object includes a statistical center and spatial moments equivalent to points on the physical object reflecting infrared light through the interactive display surface and thus defining the characteristic.

22. The method of claim 21, wherein the points defining the characteristic of the physical object include adjoining points meeting a predefined condition.

23. The method of claim 21, wherein the outermost points include points not having adjoining points in all considered directions.

24. The method of claim 20, wherein the ellipse representative of the characteristic of the physical object includes dimensions large enough to encompass an outermost point of all points defining the characteristic.

25. The method of claim 19, wherein the attribute is correlated with the characteristic of the physical object by:

(a) associating the attribute with a region of the interactive display surface; and

(b) detecting when the physical object is disposed adjacent to the region of the interactive display surface associated with the attribute.

26. The method of claim 19, further comprising the steps of:

(a) displaying on the interactive display surface a plurality of indicators representing a plurality of changeable program attributes; and

(b) enabling a user to select a desired attribute among the plurality of changeable program attributes by one of rotating the physical object and translating the physical object toward a desired indicator representing the desired attribute.

27. The method of claim 26, further comprising the steps of:

(a) displaying on the interactive display surface a plurality of secondary indicators representing a plurality of secondary changeable program attributes associated with the desired indicator; and

(b) enabling the user to select a desired secondary attribute among the plurality of secondary changeable program attributes by one of rotating the physical object and translating the physical object toward a desired secondary indicator.

28. The method of claim 26, further comprising the steps of:

(a) detecting selection of the desired attribute; and

(b) enabling the user to change a value of the changeable program attribute selected by one of rotating the physical object and translating the physical object.

29. The method of claim 19, further comprising the steps of:

(a) associating a plurality of data elements with the physical object;

(b) depicting the data elements with representations displayed on the interactive display surface adjacent a position of ellipse representing a characteristic of the physical object; and

(c) at least one of rotating and moving the representations to correspond with at least one of a rotation and a translation of the physical object.

30. The method of claim 19, further comprising the step of generating an audio signal when the physical object is disposed adjacent to the interactive display surface, the audio signal generated being associated with one of the location and the shape of the physical object.

31. The method of claim 30, wherein the audio signal is generated from a predetermined list of audio signals associated with corresponding different shapes of physical objects.

32. The method of claim 30, wherein a quality of the audio signal generated depends on the position on the interactive display surface where the physical object is disposed.

33. The method of claim 32, wherein the interactive display surface is partitioned into a plurality of regions, each of which is associated with a different category of audio signals.

34. The method of claim 30, further comprising the step of adjusting a property of the audio signal generated in response to at least one of a rotation or a translation of the physical object.

35. The method of claim 34, wherein the property of the audio signal includes at least one of a pitch, a tempo, a sustain, a repetition, and a harmonic quality.

36. A memory medium on which are stored machine instructions for carrying out the steps of claim 19.

37. A system for providing input to an application that is being executed, comprising:

(a) an interactive display surface adjacent to which the physical object is manipulated, said interactive display surface diffusing light, and having a processing side and an interactive side from which the image is viewed and adjacent to which the physical object can be placed, the processing side being opposite to the interactive side;

(b) a projector that projects graphic images onto the processing side of the interactive display surface, said graphic images being visible from the interactive side;

(c) a light source disposed on the processing side of the display surface, the light source emitting infrared light that is transmitted through the display surface to the interactive side and reflected back through the interactive display surface by the physical object that is disposed adjacent to the interactive side of the interactive display surface;

(d) a light sensor disposed on the processing side of the interactive display surface, the light sensor sensing infrared light reflected back from the physical object through the interactive display surface and imaging the interactive display surface to detect the physical object and its location;

(e) a processor in communication with the light sensor; and

(f) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) executing an application with the processor, the application having at least one attribute changeable while the application is executing;

(ii) detecting the location and an initial orientation of the physical object relative to the interactive display surface based upon a characteristic of the physical object when the physical object is disposed adjacent to the interactive display surface, the characteristic including at least one of a shape, a size, and an optical quality;

(iii) correlating the attribute with the characteristic of the physical object; and

(iv) changing the attribute in response to at least one of the location and a movement of the physical object relative to the interactive display surface.

38. The system of claim 37, wherein the machine instructions stored in the memory further cause the processor to determine the characteristic of the physical object by creating an ellipse representative of the characteristic of the physical object, the ellipse one of having an equivalent statistical center and spatial moments of points defining the characteristic, and having dimensions large enough to encompass all outermost points of the points defining the characteristic.

39. The system of claim 37, wherein the machine instructions stored in the memory further cause the processor to recognize the points defining the characteristic of the physical object as including adjoining points meeting a predefined condition.

40. The method of claim 39, wherein the outermost points include points not having adjoining points meeting the predefined condition.

41. The system of claim 37, wherein the characteristic includes at least one of the orientation of the object and the position of the physical object, such that at least one of rotating and translating the physical object changes the attribute.

42. The system of claim 41, wherein the machine instructions stored in the memory further cause the processor to correlate the attribute with the characteristic of the physical object by:

(a) associating the attribute with a region of the interactive display surface; and

(b) detecting when the physical object is disposed adjacent to the region of the interactive display surface associated with the attribute.

43. The system of claim 41, wherein the machine instructions stored in the memory further cause the processor to:

(a) correlate a value of the attribute with the orientation of the physical object;

(b) detect a change in the location of the physical object on the interactive display surface;

(c) adjust a scaling factor based on the change in the location of the physical object on the interactive display surface; and

(d) apply the scaling factor to changes in the attribute associated with rotation of the physical object.

44. The system of claim 41, wherein the machine instructions stored in the memory further cause the processor to:

(a) display on the interactive display surface a plurality of indicators representing a plurality of changeable program attributes; and

(b) enable a user to select a desired attribute among the plurality of changeable program attributes by one of rotating the physical object and translating the physical object toward a desired indicator representing the desired attribute.

45. The system of claim 44, wherein the machine instructions stored in the memory further cause the processor to:

(a) display on the interactive display surface a plurality of secondary indicators representing a plurality of secondary changeable program attributes associated with the desired indicator; and

(b) enable a user to select a desired secondary attribute among the plurality of secondary changeable program attributes by one of rotating the physical object and translating the physical object toward a desired secondary indicator representing the desired secondary attribute.

46. The system of claim 44, wherein the machine instructions stored in the memory further cause the processor to:

(a) detect a selection of the desired attribute; and

(b) enable a user to change a value of the changeable program attribute selected by one of rotating the physical object and translating the physical object.

47. The system of claim 41, wherein the machine instructions stored in the memory further cause the processor to:

(a) associate a plurality of data elements with the physical object;

(b) depict the data elements with representations displayed on the interactive display surface adjacent a position of the physical object; and

(c) at least one of revolve and move the representations to correspond with at least one of a rotation and a translation of the physical object.

48. The system of claim 41, wherein the machine instructions stored in the memory further cause the processor to cause an audio signal to be generated when the physical object is disposed adjacent to the interactive display surface, the audio signal generated being associated with one of the location and the shape of the physical object.

49. The system of claim 48, wherein the machine instructions stored in the memory further cause the processor to the audio signal generated to be varied according to the position on the interactive display surface where the physical object is disposed.

50. The system of claim 48, wherein the machine instructions stored in the memory further cause the processor to partition the interactive display surface into a plurality of regions, each of the regions being associated with a different category of audio signals associated with different shapes of physical objects.

51. The system of claim 48, wherein the machine instructions stored in the memory further cause the processor to adjust a property of the audio signal generated upon detecting at least one of a rotation and a translation of the physical object.

52. The system of claim 48, wherein the property of the audio signal includes at least one of a pitch, a tempo, a sustain, a repetition, and a harmonic quality.

53. The system of claim 48, wherein the machine instructions stored in the memory further cause the processor to cause the audio signal to be generated as define by a predetermined list of audio signals, each being associated with a different shape of a physical object.

54. A method for providing input to a computer system having an interactive display surface by manipulating a physical object disposed adjacent to the interactive display surface, the method comprising the steps of:

executing an application on the computer system, the application having a plurality of changeable program attributes that are changeable while the application is executing;

displaying a plurality of indicators representing a plurality of changeable program attributes on the interactive display surface;

determining a characteristic of the physical object by identifying an ellipse representative of the characteristic of the physical object;

determining an initial position of the ellipse;

enabling a user to select a desired attribute among the plurality of changeable program attributes by one of rotating the physical object and translating the physical object toward a desired indicator representing the desired attribute;

correlating the selected attribute with a position of the ellipse, the position of the ellipse including at least one of an orientation and a location relative to the interactive display surface;

changing the attribute in response to a change in position of the ellipse;

displaying on the interactive display surface a plurality of secondary indicators representing a plurality of secondary changeable program attributes associated with the desired indicator; and

enabling the user to select a desired secondary attribute among the plurality of secondary changeable program attributes by one of rotating the physical object and translating the physical object toward a desired secondary indicator.

55. A method for providing input to a computer system having an interactive display surface by manipulating a physical object disposed adjacent to the interactive display surface, the method comprising the steps of:

executing an application on the computer system, the application having at least one attribute changeable while the application is executing;

detecting a location and an orientation of the physical object relative to the interactive display surface based upon a characteristic of the physical object when the physical object is disposed adjacent to the interactive display surface, the characteristic including at least one of a shape, a size, and an optical quality;

correlating the attribute with the physical object;

changing the attribute in response to at least one of the location and a movement of the physical object relative to the interactive display surface, wherein changing the attribute includes;

generating an audio signal when the physical object is disposed adjacent to the interactive display surface, the audio signal generated being associated with one of the location and the shape of the physical object, wherein the audio signal is generated from a predetermined list of audio signals, each being associated with a different shape of a physical object.