Using size and shape of a physical object to manipulate output in an interactive display application

• US 7,511,703 B2
• Filed: 06/28/2004
• Issued: 03/31/2009
• Est. Priority Date: 06/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dehlin, Joel P., Wilson, Andrew D.
• Primary Examiner(s): Hjerpe; Richard
• Assistant Examiner(s): Sheng; Tom V
• Application Number:
  US10/879,872
• Publication Number:
  US 20060007123A1
• Time in Patent Office: 1,737 Days
• Field of Search: 345/173, 345/179, 345/175, 178/18.09
• US Class Current: 345/175
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

Output of a computer system is manipulated using a physical object disposed adjacent to an interactive display surface. A painting application produces an image in response to an object disposed adjacent to the interactive display surface. During each of a plurality of capture intervals, a set of points corresponding to the object is detected when the object is disposed adjacent to the interactive display surface. An image is projected onto the interactive display surface representing the set of points and is filled with a color or pattern. As successive sets of points are accumulated during each of a plurality of capture intervals, a composite image is displayed. An object can thus be used, for example, to “draw,” “paint,” or “stamp” images on the display surface. These images manifest characteristics of the object and its interaction and movement relative to the interactive display surface in a realistic manner.

Claims

1. A method for interactively producing an image on an interactive display surface by manipulating a physical object relative to the interactive display surface, comprising the steps of:

(a) detecting a set of points presented by at least a portion of the physical object disposed adjacent to the interactive display surface during a capture interval;

(b) generating an image corresponding to the set of points detected;

(c) displaying the image on the interactive display surface;

(d) detecting sets of points presented by said at least the portion of the physical object disposed adjacent to the interactive display surface during a plurality of capture intervals that occur successively over a time;

(e) generating images for the sets of points, where one image is generated for each set of points detected during a different capture interval;

(f) accumulating the images corresponding to the sets of points presented by said at least the portion of the physical object during the plurality of capture intervals;

(g) generating a composite image for each set of points by determining at least one connected component corresponding to said at least the portion of the physical object, the connected component having a plurality of adjoining points meeting a predefined condition and being bounded by outermost points not having adjoining points in all considered directions; and

(h) presenting the composite image comprising the images accumulated for the sets of points.

2. The method of claim 1, wherein the physical object is moved over the interactive display surface during the plurality of capture intervals.

3. The method of claim 1, further comprising the step of enhancing the composite image by blending each of the images at an offset to the images that were accumulated.

4. The method of claim 1, wherein the step of detecting the set of points presented by said at least the portion of the physical object during a capture interval includes the steps of:

(a) transmitting infrared light through the interactive display surface toward a face of the interactive display surface adjacent to which the surface of physical object is disposed; and

(b) detecting the set of points presented by said at least the portion of the physical object by sensing infrared light reflected from the face of the physical object back through the interactive display surface.

5. The method of claim 1, wherein the step of generating the image comprises the step of creating an ellipse representative of each connected component, the ellipse having a major axis and a minor axis.

6. The method of claim 5, wherein the ellipse representative of each connected component is created to have dimensions sufficiently large to encompass the outermost points of the connected component.

7. The method of claim 5, wherein the ellipse representative of each connected component has an equivalent statistical center and spatial moments, said spatial moment comprising the points included in the connected component.

8. The method of claim 5, further comprising the steps of:

(a) generating a plurality of -ellipses over the plurality of capture frames, each connected component having a corresponding ellipse generated for it;

(b) connecting ends of major axes and ends of minor axes of the ellipses generated for a connected component over the plurality of capture frames, with continuing lines to create outlined spaces; and

(c) filling in the outlined spaces with at least one of a color and a pattern.

9. The method of claim 8, further comprising the step of enabling a user to select said at least one of the color and the pattern from a plurality of available fill attributes presented in a portion of the interactive display surface.

10. The method of claim 9, wherein the pattern includes a plurality of lines.

11. The method of claim 9, wherein the portion of the interactive display surface includes a plurality of regions representing a palette of available patterns and colors.

12. The method of claim 1, further comprising the step of fading the images that were accumulated, each of the images being successively faded as a function of time passing since the capture interval during which the set of points, from which the image was generated, was detected.

13. The method of claim 1, further comprising the step of altering the images generated when more than one image occupies a same location on the interactive display surface.

14. The method of claim 1, further comprising the step of enabling a user to undo creation of an image previously created.

15. The method of claim 14, further comprising the step of enabling the user to redo creation of an image previously removed during a previous undo.

16. The method of claim 1, further comprising the step of enabling a user to pause changes in an image presented on the interactive display surface.

17. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 1.

18. A method for enabling a user to employ a physical object to produce an image on an interactive display surface, wherein the image corresponds to a portion of the physical object that is adjacent to an interactive display surface, comprising the steps of:

(a) detecting a set of points corresponding to the portion of the physical object disposed adjacent to the interactive display surface during each of a plurality of capture intervals, wherein the step of determining the set of points presented by the physical object includes the steps of;

(i) transmitting infrared light though the interactive display surface toward the portion of the physical object that is adjacent to the interactive display surface;

(ii) sensing infrared light reflected from the portion of the physical object back through the interactive display surface, producing a signal; and

(iii) detecting the set of points corresponding to the portion of the physical object based upon the signal;

(b) generating an image on the interactive display surface representative of the set of points detected, for each of the plurality of capture intervals, wherein the step of generating the image comprises the step of;

(i) determining at least one connected component corresponding to the portion of the physical object that is adjacent to the interactive display surface, the connected component having a plurality of adjoining points meeting a predefined condition, said connected component being bounded by outermost points not having adjoining points in all considered directions;

(c) accumulating images representative of sets of points presented by the physical object during each of the plurality of capture intervals; and

(d) displaying a composite image including the accumulated images of sets of points.

19. The method of claim 18, further comprising the steps of:

(a) eliciting a user selection of at least one of a color and a pattern for filling the composite image displayed on the interactive display surface; and

(b) filling the composite image with said one of the color and the pattern.

20. The method of claim 19, further comprising the step of generating each image with an ellipse representative of the at least one connected component, the ellipse having a major axis and a minor axis.

21. The method of claim 20, wherein the ellipse representative of each connected component has dimensions sufficiently large to encompass the outermost points included in the connected component.

22. The method of claim 20, wherein the ellipse representative of each connected component has an equivalent statistical center and spatial moments comprising the points included in the connected component.

23. The method of claim 20, further comprising the steps of:

(a) generating a plurality of ellipses over the plurality of capture frames, each connected component having a corresponding ellipse generated for it;

(b) connecting ends of major axes and ends of minor axes of the ellipses generated for a connected component over the plurality of capture frames, with continuing lines to create outlined spaces; and

(c) filling in the outlined spaces with at least one of a color and a pattern.

24. The method of claim 19, wherein the step of eliciting the user selection comprises the step of presenting available pattern attributes in a portion of the interactive display surface.

25. The method of claim 24, further comprising the step of enabling the user to select one of the color and the pattern with one of the physical object and a second physical object that is placed adjacent to said one of the color and the pattern being selected from within the portion of the interactive display surface.

26. The method of claim 19, wherein the pattern comprises a plurality of lines.

27. The method of claim 19, wherein the portion of the display screen includes a plurality of regions representing a palette of available colors and patterns.

28. The method of claim 18, further comprising the step of enhancing the composite image by blending each of the images at an offset to the images that were accumulated.

29. The method of claim 18, further comprising the step of fading the images accumulated, each of the images being successively faded as a function of time passing since a capture interval during which the set of points from which the image was generated was detected.

30. The method of claim 18, further comprising the step of altering the images generated when more than one image occupies a same location on the interactive display surface.

31. The method of claim 18, further comprising the step of enabling a user to undo creation of an image previously created.

32. The method of claim 31, further comprising the step of enabling the user to redo creation of an image previously removed during a previous undo.

33. The method of claim 18, further comprising the step of enabling a user to pause changes in an image presented on the interactive display surface.

34. A memory medium on which are stored machine executable instructions for carrying out the steps of claim 18.

35. A system for providing input to an application that is being executed, to create an image in response to a physical object, comprising:

(a) an interactive display surface adjacent to which a physical object is manipulated, said interactive display surface diffusing light, and having a processing side and an interactive side from which the image is viewed and adjacent to which the physical object can be placed, the processing side being opposite to the interactive side and parallel thereto;

(b) a projector that projects graphic images onto the processing side of the interactive display surface, said graphic images being visible from the interactive side;

(c) a light source disposed on the processing side of the display surface, the light source emitting infrared light that is transmitted through the display surface to the interactive side and reflected back through the interactive display surface by the physical object that is disposed adjacent to the interactive side of the interactive display surface;

(d) a light sensor disposed on the processing side of the interactive display surface, the light sensor sensing infrared light reflected back from the physical object through the processing side of the interactive display surface and imaging the interactive display surface to detect the physical object and its location;

(e) a processor in communication with the light sensor; and

(f) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) detecting sets of points presented by at least a portion of a physical object disposed adjacent to the interactive display surface during a plurality of capture intervals that occur successively over a time;

(ii) generating images for each of the sets of points detected during a different capture interval;

(iii) accumulating the images corresponding to the sets of points presented by said at least the portion of physical object during the plurality of capture intervals;

(iv) generating a composite image comprising the images accumulated for the sets of points

(v) enhancing the composite image by blending each of the images at an offset to the images that were accumulated; and

(vi) displaying the composite image on the interactive display surface.

36. The system of claim 35, wherein the machine language instructions further cause the processor to generate the image corresponding to each set of points by determining at least one connected component corresponding to said at least the portion of a physical object, the connected component having a plurality of adjoining points meeting a predefined condition and being bounded by outermost points not having adjoining points in all considered directions.

37. The system of claim 36, wherein the machine language instructions further cause the processor to create an ellipse representative of each connected component, the ellipse having a major axis and a minor axis.

38. The system of claim 37, wherein the ellipse representative of the set of points includes an ellipse having dimensions large enough to encompass an outermost point of all points included in the connected component.

39. The system of claim 37, wherein the ellipse representative of each connected component is created to have dimensions sufficiently large to encompass the outermost points of the connected component.

40. The system of claim 37, wherein the machine language instructions further cause the processor to:

(a) generate a plurality of ellipses over the plurality of capture frames, each connected component having a corresponding ellipse generated for it;

(b) connect ends of major axes and ends of minor axes of the ellipses generated for a connected component over the plurality of capture frames, with continuing lines to create outlined spaces; and

(c) fill in the outlined spaces with at least one of a color and a pattern.

41. The system of claim 40, wherein the machine instructions further cause the processor to enable a user to select said at least one of the color and the pattern from a plurality of available fill attributes presented in a portion of the interactive display surface.

42. The system of claim 41, wherein the pattern includes a plurality of lines.

43. The system of claim 41, wherein the portion of the interactive display surface includes a plurality of regions representing a palette of available colors and patterns.

44. The system of claim 35, wherein the machine language instructions further cause the processor to fade the images that were accumulated, each of the images being successively faded as a function of time passing since the capture interval during which the set of points, from which the image was generated, was detected.

45. The system of claim 35, wherein the machine language instructions further cause the processor to alter the images generated when more than one image occupies a same location on the interactive display surface.

46. The system of claim 35, wherein the machine language instructions further cause the processor to enable a user to undo creation of an image previously created.

47. The system of claim 46, wherein the machine language instructions further cause the processor to enable the user to redo creation of an image previously removed during a previous undo.

48. The system of claim 35, wherein the machine language instructions further cause the processor to enable a user to pause changes in an image presented on the interactive display surface.

49. A system for enabling a user to employ a physical object produce an image, comprising:

(a) an interactive display surface adjacent to which a physical object is manipulated, said interactive display surface diffusing light, and having a processing side and an interactive side from which the image is viewed and adjacent to which the physical object can be placed, the processing side being opposite to the interactive side;

(b) a projector that projects graphic images onto the processing side of the interactive display surface, said graphic images being visible from the interactive side;

(c) a light source disposed on the processing side of the display surface, the light source emitting infrared light that is transmitted through the display surface to the interactive side and reflected back through the interactive display surface by the physical object that is disposed adjacent to the interactive side of the interactive display surface;

(d) a light sensor disposed on the processing side of the interactive display surface, the light sensor sensing infrared light reflected back from the physical object through the interactive display surface and imaging the interactive display surface to detect the physical object and its location;

(e) a processor in communication with the light sensor; and

(f) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) detecting a set of points corresponding to a portion of a physical object disposed adjacent to the interactive display surface during each of a plurality of capture intervals, wherein the set of points is detected in a signal produced by the light sensor in response to infrared from the light source that is reflected from the portion of the a physical object disposed adjacent to the interactive display surface;

(ii) generating an image on the interactive display surface representative of the set of points detected, for each of the plurality of capture intervals, wherein the machine language instructions generate the image by determining at least one connected component corresponding to the portion of a physical object that is adjacent to the interactive display surface, the connected component having a plurality of adjoining points meeting a predefined condition, said connected component being bounded by outermost points not having adjoining points in all considered directions;

(iii) accumulating images representative of sets of points presented by the physical object during each of the plurality of capture intervals; and

(iv) displaying a composite image including the accumulated images of sets of points.

50. The system of claim 49, wherein the machine instructions further cause the processor to:

(a) elicit a user selection of at least one of a color and a pattern for filling the composite image displayed on the interactive display surface; and

(b) fill the composite image with said one of the color and the pattern.

51. The system of claim 50, wherein the machine instructions further cause the processor to present available pattern attributes in a portion of the interactive display surface when eliciting the user selection of at least one of the color and the pattern.

52. The system of claim 51, wherein the machine language instructions further cause the processor to enable a user to select one of the color and the pattern with one of the physical object and a second physical object that is placed adjacent to said one of the color and the pattern being selected from within the portion of the interactive display surface.

53. The system of claim 50, wherein the pattern comprises a plurality of lines.

54. The system of claim 50, wherein the portion of the display screen includes a plurality of regions representing a palette of available colors and patterns.

55. The system of claim 49, wherein the machine language instructions further cause the processor to enhance the composite image by blending each of the images at an offset to the images that were accumulated.

56. The system of claim 49, wherein the machine language instructions further cause the processor to generate each image with an ellipse representative of the at least one connected component, the ellipse having a major axis and a minor axis.

57. The system of claim 56, wherein the ellipse representative of each connected component has dimensions sufficiently large to encompass the outermost points included in the connected component.

58. The system of claim 56, wherein the ellipse representative of each connected component has an equivalent statistical center and spatial moments comprising the points included in the connected component.

59. The system of claim 56, wherein the machine language instructions further cause the processor to:

(a) generating a plurality of ellipses over the plurality of capture frames, each connected component having a corresponding ellipse generated for it;

(b) connecting ends of major axes and ends of minor axes of the ellipses generated for a connected component over the plurality of capture frames, with continuing lines to create outlined spaces; and

(c) filling in the outlined spaces with at least one of a color and a pattern.

60. The system of claim 49, wherein the machine language instructions further cause the processor to fade the images accumulated, each of the images being successively faded as a function of time passing since a capture interval during which the set of points from which the image was generated was detected.

61. The system of claim 49, wherein the machine language instructions further cause the processor to alter the images generated when more than one image occupies a same location on the interactive display surface.

62. The system of claim 49, wherein the machine language instructions further cause the processor to enable a user to undo creation of an image previously created.

63. The system of claim 62, wherein the machine language instructions further cause the processor to enabling the user to redo creation of an image previously removed during a previous undo.

64. The system of claim 49, wherein the machine language instructions further cause the processor to enable a user to pause changes in an image presented on the interactive display surface.