Methods and apparatuses for providing a hardware accelerated web engine

• US 20080120626A1
• Filed: 11/17/2006
• Published: 05/22/2008
• Est. Priority Date: 11/17/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Blanchard, Richard, Calhoun, Kevin, Hyatt, Dave, Graffagnino, Peter, Melton, Don, Stachowiak, Maciej, Drieu, Gilles, Adler, Darin
• Granted Patent: US 8,234,392 B2
• Time in Patent Office: Days
• US Class Current: 719/320
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

Methods and apparatuses for providing hardware acceleration of a web browser are disclosed. In one embodiment, a method of operating a web browser on a computer system includes analyzing a data stream having a plurality of fragments. The method further includes determining what fragments of the data stream should be rendered for storage into separate backing stores. The method further includes rendering the fragments into raster images intended for hardware acceleration. The method further includes storing the raster images in the backing stores located in a graphics processing unit.

Claims

1. A method for hardware accelerating a web browser on a computer system, the method comprising:

analyzing a data stream having a plurality of fragments, the data stream being processed by the web browser;

determining what fragments of the data stream should be rendered for storage into separate backing stores;

rendering the fragments into raster images intended for hardware acceleration; and

causing the storage of the raster images in the backing stores located in a graphics processing unit, wherein the web browser provides information identifying the raster images to the graphics processing unit.

2. The method of claim 1, further comprising:

compositing the raster images, which each represent a decomposed portion of a complete image, into a complete image displayed as a web page in order to hardware accelerate the operation of the web browser.

3. The method of claim 1, wherein the data stream comprises a web page having a plurality of elements.

4. The method of claim 1, wherein determining what fragments of the data stream should be rendered for storage into separate backing stores occurs by assigning a backing store for each fragment of the data stream.

5. The method of claim 3, further comprising:

determining what elements of the web page should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each element having a specific attribute associated with the element, the specific attribute comprises opacity, explicit position, relative position, or explicit Z ordering.

6. The method of claim 5, wherein determining what elements of the web page should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each element that provides scripting of the web page.

7. The method of claim 5, wherein determining what elements of the web page should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each element associated with an explicit indication that the element should be an independent layer.

8. The method of claim 5, wherein determining what elements of the web page should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each element having a combination of specific attributes which individually each specific attribute is not sufficient to receive an independent layer.

9. The method of claim 1, wherein determining what fragments of the data stream should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each fragment associated with a plug-in model.

10. The method of claim 1, wherein determining what fragments of the data stream should be rendered for storage into separate backing stores occurs by allocating an independent layer associated with a backing store for each fragment requiring animation.

11. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

analyzing a data stream having a plurality of fragments, the data stream being processed by the web browser;

determining what fragments of the data stream should be rendered for storage into separate backing stores;

rendering the fragments into raster images intended for hardware acceleration; and

causing the storage of the raster images in the backing stores located in a graphics processing unit, wherein the web browser provides information identifying the raster images to the graphics processing unit.

12. A method of hardware accelerating a web browser on a computer system, the method comprising:

constructing a data structure having a plurality of nodes associated with elements of a web page;

monitoring a state for each node of the data structure; and

determining whether a change of a state of a node requires rendering the node into a raster image, which represents a decomposed portion of a complete image.

13. The method of claim 12, further comprising:

rendering the node into a raster image if rendering is necessary;

storing the raster image in a retained backing store located in a graphics processing unit if rendering is necessary; and

compositing raster images located in retained backing stores associated with the data structure into a complete image in order to hardware accelerate the operation of the web browser.

14. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

constructing a data structure having a plurality of nodes associated with elements of a web page;

monitoring a state for each node of the data structure; and

determining whether a change of a state of a node requires rendering the node into a raster image, which represents a decomposed portion of a complete image.

15. The machine-readable medium of claim 14, further comprising:

rendering the node into a raster image if rendering is necessary;

storing the raster image in a retained backing store located in a graphics processing unit if rendering is necessary; and

compositing raster images located in retained backing stores associated with the data structure into a complete image in order to hardware accelerate the operation of the web browser.

16. A method for hardware accelerating a web browser on a computer system, the method comprising:

constructing a first data structure by grouping elements of a web page into a first plurality of nodes;

determining what elements should be rendered for storage into separate backing stores;

constructing a second data structure separate from the first data structure, the second data structure comprises a second plurality of nodes each representing an independent layer associated with a raster image intended for hardware acceleration;

rendering all nodes of the first data structure to create raster images, which each represent a decomposed portion of a complete image, stored in the backing stores located in a graphics processing unit (GPU); and

compositing the raster images stored in the backing stores to construct a first final image displayed as a web page in order to hardware accelerate the operation of the web browser.

17. The method of claim 16, further comprising:

receiving an input to animate a portion of the web page associated with a raster image stored in a backing store;

determining if animation of the portion of the web page requires re-rendering of any node of the first data structure; and

instructing the GPU to re-composite the raster images into a second final image displayed as a web page if no re-rendering of any node of the first data structure is required.

18. The method of claim 17, wherein animations of the portion of the web page that do not require re-rendering of any nodes of the first data structure comprise changing at least one of:

position, Z order, opacity, and scale.

19. The method of claim 17, further comprising:

re-rendering a node of the first data structure associated with the portion of the web page if re-rendering is required; and

re-compositing the raster images to construct a third final image displayed as a web page.

20. The method of claim 19, wherein animations of the portion of the web page that do require re-rendering at least one node of the first data structure comprise changing shape or updating a video frame.

21. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for each element having a specific attribute associated with the element, the specific attribute comprises opacity, explicit position, relative position, or explicit Z ordering.

22. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for each element that provides scripting of the web page.

23. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for each element associated with an explicit indication that the element should be an independent layer.

24. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for each element having a combination of specific attributes which individually each specific attribute is not sufficient to receive an independent layer.

25. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for a plug-in model associated with the element.

26. The method of claim 16, wherein determining what elements should be rendered for storage into separate backing stores occurs by allocating an independent layer for each element requiring animation.

27. The method of claim 16, wherein each independent layer is associated with a raster image stored in a single retained backing store.

28. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

constructing a first data structure by grouping elements of a web page into a first plurality of nodes;

determining what elements should be rendered for storage into separate backing stores;

constructing a second data structure separate from the first data structure, the second data structure comprises a second plurality of nodes each representing an independent layer associated with a raster image intended for hardware acceleration;

rendering all nodes of the first data structure to create raster images, which each represent a decomposed portion of a complete image, stored in the backing stores located in a graphics processing unit (GPU); and

compositing the raster images stored in the backing stores to construct a first final image displayed as a web page in order to hardware accelerate the operation of the web browser.

29. The machine-readable medium of claim 28, further comprising:

receiving an input to animate a portion of the web page associated with a raster image stored in a backing store;

determining if animation of the portion of the web page requires re-rendering of any node of the first data structure; and

instructing the GPU to composite the raster images into a second final image displayed as a web page if no re-rendering of any node of the first data structure is required.

30. The machine-readable medium of claim 29, further comprising:

re-rendering a node associated with the portion of the web page if re-rendering is required; and

re-compositing the raster images to construct a third final image displayed as a web page.

31. A data processing system, comprising:

a first microprocessor coupled to a bus;

main memory coupled to the bus with the main memory configured to store a first data structure representing a render tree having nodes for grouping elements of a web page; and

a graphics processing unit coupled to the bus, the graphics processing unit having a second microprocessor and a plurality of backing stores with each backing store configured to store raster images, which represent a decomposed portion of a complete image, associated with nodes of the render tree.

32. The method of claim 31, wherein the nodes comprise an image element, a graphic element, a text element, a vector element, and a media element.

33. The method of claim 32, wherein each of the nodes are defined by a property selected from the group consisting of a position, a coordinate system, a transform, and a relationship.

34. A method of operating a software application on a computer system, the method comprising:

analyzing a data stream having a plurality of fragments, the data stream being encoded in a markup language;

determining what fragments of the data stream should be rendered for storage into separate backing stores;

rendering the fragments into raster images, which each represent a decomposed portion of a complete image; and

causing the storage of the raster images in the backing stores located in a graphics processing unit, wherein the software application provides information identifying the raster images to the graphics processing unit.

35. The method of claim 34, further comprising:

compositing the raster images into a complete image in order to hardware accelerate the operation of the software application.

36. The method of claim 34, wherein the information identifying the raster images comprises location and attributes of the raster images.

37. The method of claim 36, wherein the attributes of the raster images comprise opacity and position of each raster image within a complete image.

38. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

analyzing a data stream having a plurality of fragments, the data stream being encoded in a markup language;

determining what fragments of the data stream should be rendered for storage into separate backing stores;

rendering the fragments into raster images, which each represent a decomposed portion of a complete image; and

causing the storage of the raster images in the backing stores located in a graphics processing unit, wherein the web browser provides information identifying the raster images to the graphics processing unit.

39. A method of operating a graphics processing unit on a computer system, the method comprising:

receiving raster images, which each represent a decomposed portion of a complete image, associated with a data stream having a plurality of fragments, the data stream being encoded in a markup language;

storing each raster image in a separate backing store located in the graphics processing unit; and

compositing the raster images into a complete image in order to hardware accelerate the operation of a web browser.

40. The method of claim 39, further comprising:

receiving information identifying the raster images, the information received from the web browser.

41. The method of claim 40, wherein the data stream comprises a web page having a plurality of elements.

42. The method of claim 40, wherein receiving information identifying the raster images indicates a data structure containing a hierarchical representation of the raster images.

43. The method of claim 40, wherein receiving information identifying the raster images indicates the location of the raster images in the computer system.

44. The method of claim 40, wherein receiving information identifying the raster images indicates attributes of each raster image.

45. The method of claim 44, wherein the attributes of the raster images comprise opacity and position of each raster image within a complete image.

46. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

receiving raster images, which each represent a decomposed portion of a complete image, associated with a data stream having a plurality of fragments, the data stream being encoded in a markup language;

storing each raster image in a separate backing store located in the graphics processing unit; and

compositing the raster images into a final display image in order to hardware accelerate the operation of a web browser.

47. A method of expressing animation in a data stream, the method comprising:

defining animation states in the data stream with each state having at least one property such that properties are animated as a group.

48. The method of claim 47, wherein the properties comprise opacity, x and y coordinate position, Z ordering, width, and height.

49. A method of expressing animation in a data stream, the method comprising:

declaring how a change in state should be processed via an animation rule.

50. The method of claim 49, wherein an animation rule comprises defining a duration for each property that is expected to change.

51. The method of claim 49, wherein an animation rule comprises defining a transition for each property that is expected to change.

52. The method of claim 51, wherein the transition associated with a property comprises an interpolation function for the property with a numeric value.

53. The method of claim 51, wherein the transition associated with a property comprises a cutoff transition for the property with a non-numeric value.

54. The method of claim 49, wherein declaring how a change in state should be processed via an animation rule occurs with a sequence of states.

55. A method of expressing animation in a data stream, the method comprising:

defining animation states in the data stream with each state having at least one property such that properties are animated as a group; and

declaring how a change in state should be processed via an animation rule.

56. The method of claim 55, wherein an animation rule comprises defining a duration for each property that is expected to change.

57. The method of claim 55, wherein an animation rule comprises defining a transition for each property that is expected to change.

58. The method of claim 57, wherein the transition associated with a property comprises an interpolation function for the property with a numeric value.

59. The method of claim 57, wherein the transition associated with a property comprises a cutoff transition for the property with a non-numeric value.

60. The method of claim 55, wherein declaring how a change in state should be processed via an animation rule occurs with a sequence of states.

61. The method of claim 55, wherein the data stream is written in cascading style sheets.

62. A machine-readable medium having executable instructions to cause a machine to perform a method comprising:

defining animation states in the data stream with each state having at least one property such that properties are animated as a group; and

declaring how a change in state should be processed via an animation rule.

63. In an environment with a web browser application and a graphics processing unit for hardware accelerating the operation of the web browser application, an application programming interface (API), comprising:

transferring a first call with redraw instructions from a subclass of a first data structure having a plurality of render nodes to a second data structure having a plurality of layer nodes; and

transferring a second call with render instructions from the second data structure to the subclass of the first data structure.

64. The method of claim 63, wherein the subclass of the first data structure creates the second data structure via the API.

65. The method of claim 64, wherein changes to the render nodes of the first data structure create equivalent changes to the layer nodes of the second data structure via the API.

66. The method of claim 65, wherein the graphical processing unit accelerates changes to the layer nodes of the second data structure.

67. A web browser comprising:

a set of instructions to determine data for graphics acceleration on a graphics processing unit.

68. The web browser of claim 67, further comprising:

a set of instructions to transfer the data to the graphics processing unit.