Multimedia documents integrating and displaying system

• US 6,741,242 B1
• Filed: 07/17/2000
• Issued: 05/25/2004
• Est. Priority Date: 03/23/2000
• Status: Expired due to Fees

• Current Assignee: Famotik Kabushikikaisha, Kunio Yarita
• Original Assignee: Famotik Kabushikikaisha
• Inventors: Itoh, Hiroyuki, Yoshida, Takashi
• Primary Examiner(s): Jankus, Almis R.
• Application Number:
  US09/617,885
• Time in Patent Office: 1,408 Days
• Field of Search: 345/419, 345/420, 345/421, 345/422, 345/423, 345/424, 345/426, 345/427, 345/428, 345/581, 345/606, 345/744, 345/760
• US Class Current: 345/419
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2210/61 Scene description

Abstract

A multimedia documents integrating and displaying system, comprising, a framework function section for operating in close communication with a parser, interpreting various drawing tags in a document described in a markup language, generating a framework for synchronizing and operating various drawing primitives, and generating a two-dimensional drawing command and a three-dimensional drawing command based on the framework, a two-dimensional rendering function section for receiving the two-dimensional drawing command from the framework function section, and creating a two-dimensional canvas capable of executing the two-dimensional drawing command on a Web browser, and a three-dimensional rendering function section for receiving the three-dimensional drawing command from the framework function section, and creating a three-dimensional canvas capable of executing the three-dimensional drawing command on the Web browser.

Claims

1. A multimedia documents integrating and displaying system, comprising:

a framework means for operating in close communication with a parser, interpreting various drawing tags in a document described in a markup language, generating a framework for synchronizing and operating various drawing primitives, and generating a two-dimensional drawing command and a three-dimensional drawing command based on said framework;

a two-dimensional rendering means for receiving said two-dimensional drawing command from said framework means, and creating a two-dimensional canvas capable of executing the two-dimensional drawing command on a Web browser; and

a three-dimensional rendering means for receiving said three-dimensional drawing command from said framework means, and creating a three-dimensional canvas capable of executing the three-dimensional drawing command on the Web browser.

2. A system as set forth in claim 1,

wherein said two-dimensional rendering means and said three-dimensional rendering means generate graphic display data for describing a two-dimensional image or a three-dimensional image using a two-dimensional drawing command set in Java AWT, and passing the graphic display data to the Web browser.

3. A system as set forth in claim 1,

wherein said framework means comprises:

an execution request generation means, which is activated by the parser, for receiving call-back data corresponding to a predetermined DTD from the parser, and generating execution request description data including a description type to which a parameter is added; and

a manager means for generating a two-dimensional drawing command and a three-dimensional drawing command based on said execution request description data.

4. A system as set forth in claim 3,

wherein said execution request generation means comprises:

an SVG means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of SVG, and generating execution request description data which indicates a two-dimensional drawing, sound, a moving image, an image, text or a hyperlink.

5. A system as set forth in claim 3,

wherein said execution request generation means further comprises:

a CMW means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of a CMW (Cross-Media World), and generating execution request description data which indicates a material of a three-dimensional graphic form, a texture, a camera, a light, a font, an interpolator, a timer, a sensor, image mapping, an XVL file or a VRML file.

6. A system as set forth in claim 3,

wherein said execution request generation means further comprises:

a TVX means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD in TVX (TVML with XML), and generating execution request description data which indicates sound, a moving image, an image, a material of a three-dimensional graphic form, a texture, a camera, a light, an interpolator, a timer, a sensor or a VRML file.

7. A system as set forth in claim 3,

wherein said manager means comprises the means of:

(a) structuring said execution request description data to thereby create a tree structure for an entire document described in a markup language; and

(b) generating a two-dimensional drawing command or a three-dimensional drawing command according to the execution request description data in said tree structure to thereby express the document.

8. A system as set forth in claim 7,

wherein said manager means further comprises the means of:

(a) mixing a two-dimensional drawing description and a three-dimensional drawing description according to a location of the execution request description data in said tree structure; and

(b) generating a two-dimensional embedding command for embedding a two-dimensional canvas provided by said two-dimensional rendering means in a three-dimensional canvas provided by said three-dimensional rendering means.

9. A system as set forth in claim 3,

wherein said manager means further comprises the step of:

converting an operation event received through the Web browser to a two-dimensional drawing command or a three-dimensional drawing command.

10. A system as set forth in claim 1,

wherein said framework means, the two-dimensional rendering means and the three-dimensional rendering means comprise the means of:

(a) being activated by the Web browser; and

(b) being executed after being read as objects.

11. A system as set forth in claim 1,

wherein said two-dimensional rendering means comprises:

a means for receiving a two-dimensional drawing command from said manager means, and generating a drawing method call according to a content of the two-dimensional drawing command;

a two-dimensional drawing means for receiving said drawing method call, and converting the drawing method call to media operation data; and

a two-dimensional canvas means for receiving the media operation data from the two-dimensional drawing means, generating a two-dimensional canvas on the Web browser, and executing a graphic display according to the media operation data.

12. A system as set forth in claim 11,

wherein said two-dimensional canvas means comprises:

a two-dimensional object managing means for generating a rendering request after receiving the media operation data and transferring a two-dimensional file to be used by said two-dimensional drawing means, and requesting image data generation for an entire two-dimensional canvas based on the media operation data and the two-dimensional file; and

a two-dimensional object display means for receiving the rendering request from said two-dimensional object managing means, and setting a two-dimensional display attribute in a two-dimensional canvas according to the rendering request.

13. A system as set forth in claim 12,

wherein said two-dimensional canvas means further comprises:

a two-dimensional media operation means for receiving an operation event for a two-dimensional operation from the Web browser, and passing a rendering request to said two-dimensional object display means according to the operation event.

14. A system as set forth in claim 13,

wherein said two-dimensional object display means comprises the means of:

(a) passing a partial rendering request to said two-dimensional drawing means according to the rendering request received from said two-dimensional media operation means; and

(b) receiving the media operation data from the two-dimensional drawing means.

15. A system as set forth in claim 11,

wherein said two-dimensional canvas means further comprises:

a means for calculating transformation data for a two-dimensional canvas if the two-dimensional canvas is pasted to a three-dimensional canvas, and setting a two-dimensional display attribute on a transformed two-dimensional canvas according to the transformation data.

16. A system as set forth in claim 1,

wherein said three-dimensional rendering means comprises:

a three-dimensional scene graph generation means for reading three-dimensional data based on three-dimensional drawing commands received from said manager means, and generating three-dimensional scene graphs for three-dimensional displays according to the three-dimensional drawing commands;

a three-dimensional scene drawing means for retaining a scene graph tree, which is an accumulation of the above three-dimensional scene graphs, and generating scene drawing data by traversing the scene graph tree; and

a three-dimensional canvas means for generating a three-dimensional canvas on a predetermined drawing area on the Web browser, and concomitantly generating graphic display data to be displayed on said three-dimensional canvas based on three-dimensional scene drawing data from said three-dimensional scene drawing means.

17. A system as set forth in claim 16,

wherein said three-dimensional scene graph generation means comprises:

a means, which is activated by a three-dimensional drawing command from said manager means, for setting or changing three-dimensional attributes including material, texture, camera, light and font style in the scene graph, and generating a scene graph with a three-dimensional attribute setting to thereby reflect a change in the three-dimensional attributes in a three-dimensional space.

18. A system as set forth in claim 16,

wherein said three-dimensional scene graph generation means further comprises:

a means, which is activated by a three-dimensional drawing command from said manager means, for setting or changing a behavior of a scene graph, expressing a moving image animation from part of a scene or an entire scene according to a time, an event and/or the like, and generating a scene graph from a three-dimensional behavior setting to thereby reflect a change of the three-dimensional behavior in the three-dimensional space.

19. A system as set forth in claim 18,

wherein the three-dimensional drawing command from said manager means comprises:

information on a sensor, a timer and an interpolation for defining a behavior of a three-dimensional animation.

20. A system as set forth in claim 16,

wherein said three-dimensional scene graph generation means further comprises:

a two-dimensional embedding means for receiving a three-dimensional drawing command from said manager means for embedding two-dimensional image data, and generating a scene graph in which the two-dimensional image data becomes embedded.

21. A system as set forth in claim 20,

wherein said two-dimensional embedding means comprises the means of:

(a) receiving from said manager means a three-dimensional drawing command which includes an image mapping setting for mapping an image to a three-dimensional image; and

(b) generating a scene graph for setting a texture for image mapping in the scene graph tree retained by said three-dimensional scene drawing means.

22. A system as set forth in claim 20,

wherein said two-dimensional embedding means comprises:

a hyperlink setting means for receiving hyperlink data as well as two-dimensional image data as a three-dimensional drawing command if a hyperlink is set in a two-dimensional image to be embedded, and generating a scene graph for embedding two-dimensional data which includes hyperlink information therein.

23. A system as set forth in claim 16,

wherein said three-dimensional scene graph generation means further comprises:

an XVL loader means, which is activated by a three-dimensional drawing command from said manger means, for reading an external XVL file, retrieving a lattice-shaped three-dimensional element using a Lattice kernel, converting the lattice-shaped three-dimensional element into a rounded three-dimensional element, and generating a scene graph.

24. A system as set forth in claim 16,

wherein said three-dimensional scene graph generation means further comprises:

a VRML loader means, which is activated by a three-dimensional drawing command from said manager means, for reading an external VRML file, analyzing a content of the external VRML file, retrieving a three-dimensional basic element, and a relationship and a movement of the three-dimensional basic element, and generating a scene graph.

25. A system as set forth in claim 16,

wherein said three-dimensional scene drawing means comprises:

a three-dimensional scene graph construction means for receiving a plurality of scene graphs according to a content of a three-dimensional drawing command from said scene graph generation means, generating a scene graph tree of scenes to be displayed using structured group nodes and reel nodes based on the scene graphs, and concomitantly performing an addition, a deletion or a change for the scene graphs in said scene graph tree; and

a means for receiving said scene graph tree from said three-dimensional scene graph construction means, scanning (traversing) all nodes in this scene graph tree from its root node, converting a coordinate system of polygon shape data (a modeling coordinate system), defined in each node, to a three-dimensional coordinate system in a three-dimensional space (a world coordinate system), and thereby generating scene drawing data.

26. A system as set forth in claim 25,

wherein said scene drawing data comprises:

rendering mode data generated from non-polygon shape data retained in the scene graph tree.

27. A system as set forth in claim 25,

wherein there is comprised:

a view control means for receiving a scene graph indicating a content of a camera in the scene graph tree, converting a coordinate to change a viewpoint in a three-dimensional scene, and passing the scene graph with a changed viewpoint to said scene drawing means.

28. A system as set forth in claim 16,

wherein said three-dimensional canvas means comprises:

a view coordinate conversion means for receiving scene drawing data including the polygon shape data from said three-dimensional scene drawing means, and normalizing coordinates of the polygon shape data to draw the polygon shape data in two dimensions on said three-dimensional canvas; and

a means for receiving normalized projection scene drawing data from said view coordinate conversion means, and generating graphic display data while buffering image drawing data to display images eliminated by hidden surface removal in said three-dimensional canvas.

29. A system as set forth in claim 28,

wherein said three-dimensional canvas means further comprises:

a means for receiving scene drawing data excluding the polygon shape data from said three-dimensional scene drawing means, and generating a rendering mode according to the scene drawing data.

30. A system as set forth in claim 28,

wherein said three-dimensional canvas means further comprises:

a means for generating three-dimensional canvas information based on an operation event from the Web browser for the three-dimensional canvas, and passing the three-dimensional canvas information to said scene drawing means.

31. A computer program product for integrating and displaying multimedia documents using a computer computer program product, comprising:

a framework means for operating in close communication with a parser, interpreting various drawing tags in a document described in a markup language, generating a framework for synchronizing and operating various drawing primitives, and generating a two-dimensional drawing command and a three-dimensional drawing command based on said framework;

a two-dimensional rendering means for receiving said two-dimensional drawing command from said framework means, and creating a two-dimensional canvas capable of executing the two-dimensional drawing command on a Web browser; and

a three-dimensional rendering means for receiving said three-dimensional drawing command from said framework means, and creating a three-dimensional canvas capable of executing the three-dimensional drawing command on the Web browser.

32. A computer program product as set forth in claim 31,

wherein said two-dimensional rendering means and said three-dimensional rendering means generate graphic display data for describing a two-dimensional image or a three-dimensional image using a command set in Java AWT, and passing the graphic display data to the Web browser.

33. A computer program product as set forth in claim 31,

wherein said framework means comprises:

an execution request generation means, which is activated by the parser, for receiving call-back data corresponding to a predetermined DTD from the parser, and generating execution request description data including a description type to which a parameter is added; and

a manager means for generating a two-dimensional drawing command and a three-dimensional drawing command based on said execution request description data.

34. A computer program product as set forth in claim 33,

wherein said execution request generation means comprises:

an SVG means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of SVG, and generating execution request description data which indicates a two-dimensional drawing, sound, a moving image, an image, text or a hyperlink.

35. A computer program product as set forth in claim 33,

wherein said execution request generation means further comprises:

a CMW means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of a CMW (Cross-Media World), and generating execution request description data which indicates a material of a three-dimensional graphic form, a texture, a camera, a light, a font, an interpolator, a timer, a sensor, image mapping, an XVL file or a VRML file.

36. A computer program product as set forth in claim 33,

wherein said execution request generation means further comprises:

a TVX means for receiving from said parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD in TVX (TVML with XML), and generating execution request description data which indicates sound, a moving image, an image, a material of a three-dimensional graphic form, a texture, a camera, a light, an interpolator, a timer, a sensor or a VRML file.

37. A computer program product as set forth in claim 33,

wherein said manager means comprises the means of:

(a) structuring said execution request description data to thereby create a tree structure for an entire document described in a markup language; and

(b) generating a two-dimensional drawing command or a three-dimensional drawing command according to the execution request description data in said tree structure to thereby express the document.

38. A computer program product as set forth in claim 37,

wherein said manager means further comprises the means of:

(a) mixing a two-dimensional drawing description and a three-dimensional drawing description according to a location of the execution request description data in said tree structure; and

(b) generating a two-dimensional embedding command for embedding a two-dimensional canvas provided by said two-dimensional rendering means in a three-dimensional canvas provided by said three-dimensional rendering means.

39. A computer program product as set forth in claim 33,

wherein said manager means further comprises the step of:

converting an operation event received through the Web browser to a two-dimensional drawing command or a three-dimensional drawing command.

40. A computer program product as set forth in claim 31,

wherein said framework means, the two-dimensional rendering means and the three-dimensional rendering means comprise the means of:

(a) being activated by the Web browser; and

(b) being executed after being read as objects.

41. A computer program product as set forth in claim 31,

wherein said two-dimensional rendering means comprises:

a means for receiving a two-dimensional drawing command from said manager means, and generating a drawing method call according to a content of the two-dimensional drawing command;

a two-dimensional drawing means for receiving said drawing method call, and converting the drawing method call to media operation data; and

a two-dimensional canvas means for receiving the media operation data from the two-dimensional drawing means, generating a two-dimensional canvas on the Web browser, and executing a graphic display according to the media operation data.

42. A computer program product as set forth in claim 41,

wherein said two-dimensional canvas means comprises:

a two-dimensional object managing means for generating a rendering request after receiving the media operation data and transferring a two-dimensional file to be used by said two-dimensional drawing means, and requesting image data generation for an entire two-dimensional canvas based on the media operation data and the two-dimensional file; and

a two-dimensional object display means for receiving the rendering request from said two-dimensional object managing means, and setting a two-dimensional display attribute in a two-dimensional canvas according to the rendering request.

43. A computer program product as set forth in claim 42,

wherein said two-dimensional canvas means further comprises:

a two-dimensional media operation means for receiving an operation event for a two-dimensional operation from the Web browser, and passing a rendering request to said two-dimensional object display means according to the operation event.

44. A computer program product as set forth in claim 43,

wherein said two-dimensional object display means comprises the means of:

(a) passing a partial rendering request to said two-dimensional drawing means according to the rendering request received from said two-dimensional media operation means; and

(b) receiving the media operation data from the two-dimensional drawing means.

45. A computer program product as set forth in claim 41,

wherein said two-dimensional canvas means further comprises:

a means for calculating transformation data for a two-dimensional canvas if the two-dimensional canvas is pasted to a three-dimensional canvas, and setting a two-dimensional display attribute on a transformed two-dimensional canvas according to the transformation data.

46. A computer program product as set forth in claim 31,

wherein said three-dimensional rendering means comprises:

a three-dimensional scene graph generation means for reading three-dimensional data based on three-dimensional drawing commands received from said manager means, and generating three-dimensional scene graphs for three-dimensional displays according to the three-dimensional drawing commands;

a three-dimensional scene drawing means for retaining a scene graph tree, which is an accumulation of the above three-dimensional scene graphs, and generating scene drawing data by traversing the scene graph tree; and

a three-dimensional canvas means for generating a three-dimensional canvas on a predetermined drawing area on the Web browser, and concomitantly generating graphic display data to be displayed on said three-dimensional canvas based on three-dimensional scene drawing data from said three-dimensional scene drawing means.

47. A computer program product as set forth in claim 46,

wherein said three-dimensional scene graph generation means comprises:

a means, which is activated by a three-dimensional drawing command from said manager means, for setting or changing three-dimensional attributes including material, texture, camera, light and font style in the scene graph, and generating a scene graph with a three-dimensional attribute setting to thereby reflect a change in the three-dimensional attributes in a three-dimensional space.

48. A computer program product as set forth in claim 46,

wherein said three-dimensional scene graph generation means further comprises:

a means, which is activated by a three-dimensional drawing command from said manager means, for setting or changing a behavior of a scene graph, expressing a moving image animation from part of a scene or an entire scene according to a time, an event and/or the like, and generating a scene graph from a three-dimensional behavior setting to thereby reflect a change of the three-dimensional behavior in the three-dimensional space.

49. A computer program product as set forth in claim 48,

wherein the three-dimensional drawing command from said manager means comprises:

information on a sensor, a timer and an interpolation for defining a behavior of a three-dimensional animation.

50. A computer program product as set forth in claim 46,

wherein said three-dimensional scene graph generation means further comprises:

a two-dimensional embedding means for receiving a three-dimensional drawing command from said manager means for embedding two-dimensional image data, and generating a scene graph in which the two-dimensional image data becomes embedded.

51. A computer program product as set forth in claim 50,

wherein said two-dimensional embedding means comprises the means of:

(a) receiving from said manager means a three-dimensional drawing command which includes an image mapping setting for mapping an image to a three-dimensional image; and

(b) generating a scene graph for setting a texture for image mapping in the scene graph tree retained by said three-dimensional scene drawing means.

52. A computer program product as set forth in claim 50,

wherein said two-dimensional embedding means comprises:

a hyperlink setting means for receiving hyperlink data as well as two-dimensional image data as a three-dimensional drawing command if a hyperlink is set in a two-dimensional image to be embedded, and generating a scene graph for embedding two-dimensional data which includes hyperlink information therein.

53. A computer program product as set forth in claim 46,

wherein said three-dimensional scene graph generation means further comprises:

an XVL loader means, which is activated by a three-dimensional drawing command from said manger means, for reading an external XVL file, retrieving a lattice-shaped three-dimensional element using a Lattice kernel, converting the lattice-shaped three-dimensional element into a rounded three-dimensional element, and generating a scene graph.

54. A computer program product as set forth in claim 46,

wherein said three-dimensional scene graph generation means further comprises:

a VRML loader means, which is activated by a three-dimensional drawing command from said manager means, for reading an external VRML file, analyzing a content of the external VRML file, retrieving a thee-dimensional basic element, and a relationship and a movement of the three-dimensional basic element, and generating a scene graph.

55. A computer program product as set forth in claim 46,

wherein said three-dimensional scene drawing means comprises:

a three-dimensional scene graph construction means for receiving a plurality of scene graphs according to a content of a three-dimensional drawing command from said scene graph generation means, generating a scene graph tree of scenes to be displayed using structured group nodes and reel nodes based on the scene graphs, and concomitantly performing an addition, a deletion or a change for the scene graphs in said scene graph tree; and

a means for receiving said scene graph tree from said three-dimensional scene graph construction means, scanning (traversing) all nodes in this scene graph tree from its root node, converting a coordinate system of polygon shape data (a modeling coordinate system), defined in each node, to a three-dimensional coordinate system in a three-dimensional space (a world coordinate system), and thereby generating scene drawing data.

56. A computer program product as set forth in claim 55,

wherein said scene drawing data comprises:

rendering mode data generated from non-polygon shape data retained in the scene graph tree.

57. A computer program product as set forth in claim 55,

wherein there is comprised:

a view control means for receiving a scene graph indicating a content of a camera in the scene graph tree, converting a coordinate to change a viewpoint in a three-dimensional scene, and passing the scene graph with a changed viewpoint to said scene drawing means.

58. A computer program product as set forth in claim 46,

wherein said three-dimensional canvas means comprises:

a view coordinate conversion means for receiving scene drawing data including the polygon shape data from said three-dimensional scene drawing means, and normalizing coordinates of the polygon shape data to draw the polygon shape data in two dimensions on said three-dimensional canvas; and

a means for receiving normalized projection scene drawing data from said view coordinate conversion means, and generating graphic display data while buffering image drawing data to display images eliminated by hidden surface removal in said three-dimensional canvas.

59. A computer program product as set forth in claim 58,

wherein said three-dimensional canvas means further comprises:

a means for receiving scene drawing data excluding the polygon shape data from said three-dimensional scene drawing means, and generating a rendering mode according to the scene drawing data.

60. A computer program product as set forth in claim 58,

wherein said three-dimensional canvas means further comprises:

a means for generating three-dimensional canvas information based on an operation event from the Web browser for the three-dimensional canvas, and passing the three-dimensional canvas information to said scene drawing means.