Media Integration Layer

• US 20050140694A1
• Filed: 10/23/2003
• Published: 06/30/2005
• Est. Priority Date: 10/23/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goel, Rajat, Stokes, Michael, Schechter, Greg D., Subramanian, Sriram, Smith, Adam M., Curtis, Donald B., Calkins, Matthew W., Vandenberg, Eric S., Blanco, Leonardo E., Gallo, Kevin T., Beda, Joseph S., Schneider, Gerhard A.
• Granted Patent: US 7,511,718 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/545 where tasks reside in different layers, e.g. user- and kernel-space
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 17/005 Tree description, e.g. octree, quadtree
• CPC Class Codes: G06T 2210/08 Bandwidth reduction
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/0414 Vertical resolution change
• CPC Class Codes: G09G 2340/0421 Horizontal resolution change
• CPC Class Codes: G09G 2340/0428 Gradation resolution change
• CPC Class Codes: G09G 2340/0464 Positioning

Abstract

A media integration layer including an application programming interface (API) and an object model allows program code developers to interface in a consistent manner with a scene graph data structure in order to output graphics. Via the interfaces, program code adds child visuals to other visuals to build up a hierarchical scene graph, writes Instruction Lists such as geometry data, image data, animation data and other data for output, and may specify transform, clipping and opacity properties on visuals. The media integration layer and API enable programmers to accomplish composition effects within their applications in a straightforward manner, while leveraging the graphics processing unit in a manner that does not adversely impact normal application performance. A multiple-level system includes the ability to combine different media types (such as 2D, 3D, Video, Audio, text and imaging) and animate them smoothly and seamlessly.

Claims

1. A method for arranging computer graphics data for processing into an output, comprising:

receiving a function call via an interface of a media integration layer, the function call corresponding to graphics-related data; and

causing data in a scene graph data structure to be modified based on the function call.

2. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to initialize a new instance of a visual class.

3. The method of claim 2 further comprising, receiving a function call via an interface corresponding to a transform associated with the visual.

4. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to initialize a new instance of a drawing visual class.

5. The method of claim 4 further comprising, receiving a function call via an interface to open the drawing visual instance for rendering, and in response, causing a drawing context to be returned, the drawing context providing a mechanism for rendering into the drawing visual.

6. The method of claim 1 further comprising, receiving brush data in association with the function call, and wherein causing data in a scene graph data structure to be modified comprises invoking a brush function to modify a data structure in the scene graph data structure such that when a frame is rendered from the scene graph, an area will be filled with visible data corresponding to the brush data.

7. The method of claim 6 wherein receiving brush data comprises receiving data corresponding to a solid color.

8. The method of claim 6 wherein receiving brush data comprises receiving data corresponding to a linear gradient brush and a stop collection comprising t least one stop.

9. The method of claim 6 wherein receiving brush data comprises receiving data corresponding to a radial gradient brush.

10. The method of claim 6 wherein receiving brush data comprises receiving data corresponding to an image.

11. The method of claim 10 further comprising, receiving a function call via an interface corresponding to an image effect to apply to the image.

12. The method of claim 1 further comprising, receiving pen data in association with the function call, and wherein causing data in a scene graph data structure to be modified comprises invoking a pen function that defines an outline of a shape.

13. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a geometry-related function to represent an ellipse in the scene graph data structure.

14. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a geometry-related function to represent a rectangle in the scene graph data structure.

15. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a geometry-related function to represent a path in the scene graph data structure.

16. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a geometry-related function to represent a line in the scene graph data structure.

17. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function related to hit-testing a visual in the scene graph data structure.

18. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function related to transforming coordinates of a visual in the scene graph data structure.

19. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function related to calculating a bounding box of a visual in the scene graph data structure.

20. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function via a common interface to a visual object in the scene graph data structure.

21. The method of claim 1 further comprising invoking a visual manager to render a tree of at least one visual object to a rendering target.

22. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place a container object in the scene graph data structure, the container object configured to contain at least one visual object.

23. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place image data into the scene graph data structure.

24. The method of claim 23 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place an image effect object into the scene graph data structure that is associated with the image data.

25. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place data corresponding to text into the scene graph data structure.

26. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to provide a drawing context in response to the function call.

27. The method of claim 26 wherein the function call corresponds to a retained visual, and further comprising, calling back to have the drawing context of the retained visual returned to the scene graph data structure.

28. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place a three-dimensional visual into the scene graph data structure.

29. The method of claim 28 wherein causing data in a scene graph data structure to be modified comprises mapping a two-dimensional surface onto the three dimensional visual.

30. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place animation data into the scene graph data structure.

31. The method of claim 30 further comprising communicating timeline information corresponding to the animation data to a composition engine at another layer of the media integration layer.

32. The method of claim 31 wherein the composition engine interpolates graphics data based on the timeline to animate an output corresponding to an object in the scene graph data structure.

33. The method of claim 1 wherein receiving a function call via an interface of a media integration layer comprises receiving markup, and wherein causing data in a scene graph data structure to be modified comprises parsing the markup into a call to an interface of an object.

34. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises invoking a function to place an object corresponding to audio and/or video data into the scene graph data structure.

35. The method of claim 1 wherein causing data in a scene graph data structure to be modified comprises changing a mutable value of an object in the scene graph data structure.

37. The system of claim 26 wherein at least one function is invoked to place a tree of visual objects into the scene graph data structure.

38. The system of claim 37 further comprising a visual manager that when invoked renders the tree of visual objects to a rendering target.

39. The system of claim 37 wherein the tree of visual objects is contained in a visual collection object.

40. The system of claim 26 wherein at least one function is invoked to place a visual object into the scene graph data structure.

41. The system of claim 40 wherein at least one function is invoked to associate a brush with the visual object.

42. The system of claim 40 wherein at least one function is invoked to associate a geometry with the visual object.

43. The system of claim 42 wherein the geometry comprises at least one of a set containing an ellipse geometry, a rectangle geometry, a line geometry and a path geometry.

44. The system of claim 40 wherein at least one function is invoked to associate a transform with the visual object.

45. The system of claim 44 wherein the transform comprises a rotate transform for changing a perceived angle of the visual object.

46. The system of claim 44 wherein the transform comprises a scale transform for changing a perceived size of the visual object.

47. The system of claim 44 wherein the transform comprises a translate transform for changing a perceived position of the visual object.

48. The system of claim 44 wherein the transform comprises a skew transform for changing a perceived skew of the visual object.

49. The system of claim 44 further comprising animation information associated with the transform, and wherein the animation information causes transformation data associated with the transform to change over time thereby animating the transformation of the visual object over time.

50. The system of claim 40 wherein at least one function is invoked to associate a color with the visual object.

51. The system of claim 40 wherein at least one function is invoked to associate gradient data with the visual object.

52. The system of claim 40 wherein at least one function is invoked to associate a tile brush with the visual object.

53. The system of claim 40 wherein at least one function is invoked to associate an image with the visual object.

54. The system of claim 40 wherein at least one function is invoked to associate three-dimensional data with the visual object.

55. The system of claim 40 wherein at least one function is invoked to associate a drawing comprising drawing primitives with the visual object.

56. The system of claim 40 wherein at least one function is invoked to associate audio and/or video media d with the visual object.

57. The system of claim 40 wherein at least one function is invoked to associate an image effect with the visual object.

58. The system of claim 40 wherein at least one function is invoked to associate a pen with the visual object, to describe how a shape is outlined.

59. The system of claim 40 wherein at least one function is invoked to obtain a drawing context associated with the visual object.

60. The system of claim 40 wherein at least one function is invoked to associate hit testing data with the visual object.

61. The system of claim 40 wherein at least one function is invoked to associate a rectangle with the visual object.

62. The system of claim 61 wherein at least one function is invoked to describe how a source rectangle should be stretched to fit a destination rectangle corresponding to the visual object.

63. The system of claim 61 wherein at least one function is invoked to describe how content is positioned vertically within a container corresponding to the visual object.

64. The system of claim 61 wherein at least one function is invoked to describe how content is positioned horizontally within a container corresponding to the visual object.

36. In a computing environment, a system comprising:

a scene graph data structure of a layered system for containing data that can be rendered into output that for subsequent integrated output that can be viewed; and

an object model including objects and other data that can be contained in the scene graph data structure, at least some of the objects of the object model having interfaces for invoking functions to modify contents of the scene graph data structure.

65. In a computing environment, a system comprising:

interface means for receiving function calls;

high-level composition means for integrating graphics-related data and and/or media-related data received via the interface means into a scene graph; and

rendering means for converting the scene graph into output that may be transmitted or displayed.

66. The system of claim 65 wherein the rendering means includes low-level composition means for constructing a frame for viewing based on data received from the high-level composition engine.

67. The system of claim 65 further comprising animation means, the high-level composition engine providing timeline data to the low-level composition means for interpolating the appearance of visible data across at least two frames to animate the visible data over time.