Visual content browsing using rasterized representations

• US 20020015042A1
• Filed: 11/29/2000
• Published: 02/07/2002
• Est. Priority Date: 08/07/2000
• Status: Active Grant

• Current Assignee: DigiMedia Tech, LLC (IP Investments Group LLC)
• Original Assignee: Zframe, Inc.
• Inventors: Weiss, Howard P., Robotham, John S., Johnson, Charles Lee
• Granted Patent: US 6,704,024 B2
• Time in Patent Office: Days
• US Class Current: 345/581
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine
• CPC Class Codes: G09G 5/39 Control of the bit-mapped memory
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The present invention relates to display of visual content on a client device using server-side rasterization of visual content. Visual content is rendered on a server system, transformed into bitmaps compatible with the display attributes of a client device, and transmitted for display on the client device. The invention allows the server to perform, in effect, as a remote browser for displaying Web pages, e-mail, e-mail attachments, electronic document and forms, database queries and results, drawings, presentations, and images at the client device. The approach is “remote” because the server does the rendering and the client provides the interface; “multi-level” because rendered visual content is represented as a multi-level set of raster representations; and constitutes a “browsing system” because the client and server share data about the source visual content element being browsed, and the client performs a specific browsing function assisted by the server.

Claims

1. A method of displaying visual content on a client device, comprising the steps of:

establishing a communications link between the client device and a server;

determining a display attribute of the client device;

representing a visual content element on a proxy display surface of the server;

transforming the represented visual content element to a bitmap based on the display attribute of the client device; and

transmitting the bitmap to the client device for display thereon.

3. The method of claim 1 further comprising the step of defining a data element describing an attribute of the visual content element.

4. The method of claim 3 further comprising the step of transmitting the data element to the client device.

5. The method of claim 3 further comprising the step of storing the data element at the client device.

6. The method of claim 1 wherein the attribute comprises data describing the source of the visual content.

7. The method of claim 1 wherein the attribute comprises data describing a constituent component of the visual content element.

8. The method of claim 7 in which the constituent component is localized to a sub-region of a proxy display surface.

9. The method of claim 8 further comprising the step of transmitting a sub-region coordinate of the constituent component to the client device.

10. The method of claim 9 further comprising the step of modifying a sub-region of a user interface of the client device in response to the sub-region coordinate.

11. The method of claim 3 further comprising the step of receiving the bitmap and the data element at the client device.

12. The method of claim 11 further comprising the step of modifying a caching mechanism of the client device in response to the data element.

13. The method of claim 11 further comprising the step of modifying a display mechanism of the client device in response to the data element.

14. The method of claim 7 wherein the data describing the constituent component comprises location coordinates in the proxy display surface.

15. The method of claim 11 further comprising the step of generating a display surface in a display memory at the client device in response to the bitmap and the data element.

16. The method of claim 11 further comprising the step of generating a display surface in an allocated client viewport 16 at the client device in response to the bitmap and the data element.

17. The method of claim 11 further comprising the step of generating a display at the client device in response to the bitmap and the data element.

18. The method of claim 11 further comprising the step of generating a user interface at the client device in response to the data element.

19. The method of claim 18 further comprising the steps of:

producing a user interface action at the user interface; and

generating a response in response to the user interface action and the data element.

20. The method of claim 1 further comprising the step of transmitting a request from the client device to the server requesting a cache refresh.

21. The method of claim 1 further comprising the step of transmitting from the server to the client a cache refresh.

44. The method of claim 1 further comprising the step of inserting a watermark in the bitmap prior to the step of transmitting the bitmap.

45. The method of claim 44 wherein the watermark is not visible to a user of the client device.

46. The method of claim 44 wherein the watermark is an active visual control.

48. The method of claim 47 wherein the selection bookmark comprises a composite selection bookmark.

49. The method of claim 47 wherein the selection bookmark specifies a prioritization of the type of rendering for application to the selected pixel region.

2. A method of displaying visual content on a client device, comprising the steps of:

establishing a communications link between the client device and a server;

determining a display attribute of the client device;

rasterizing a visual content element on a proxy display surface of the server as a bitmap based on the display attribute of the client device; and

transmitting the bitmap to the client device for display thereon.

22. A method for generating on a server a multi-level set of bitmaps from an input bitmap, comprising the steps of:

determining a display attribute of a client device;

selecting a pixel region of an input bitmap;

in response to the display attribute of the client device, performing a first transform operation on the pixel region to generate a first bitmap;

defining a data element corresponding to a second bitmap, the data element defining a relationship of the first bitmap to the second bitmap; and

transmitting to the client the first bitmap, the second bitmap representation, and the data element.

23. The method of claim 22 wherein the step of selecting comprises recalling a stored selection identifying the pixel region.

25. The method of claim 24 wherein the step of transmitting comprises transmitting to the client in a predetermined order the plurality of bitmaps and the plurality of data elements.

26. The method of claim 22 wherein said step of performing a first transform operation comprises performing a scaling operation on the pixel region.

27. The method of claim 26 further comprising the step of performing, in response to the display attribute of the client device, a second transform operation on the pixel region to generate the second bitmap.

28. The method of claim 22 wherein the second bitmap is the input bitmap.

29. The method of claim 22 further comprising the step of generating a first display of at least a portion of one of the first bitmap and the second bitmap on a client device.

30. The method of claim 29 further comprising the step of selecting from the first display at least a portion of the other of the first bitmap and the second bitmap and generating a second display in response thereto.

47. The method of claim 22 further comprising the step of generating a selection bookmark in response to the selection of a pixel region.

50. The method of claim 22 further comprising the step of generating a transaction bookmark in response to the selection of a pixel region, the transaction bookmark being associated with an executable action including transmission of data from the client to a server.

51. The method of claim 50 wherein the transaction bookmark is a composite transaction bookmark including specifications for at least two transactions related to at least one visual content element.

52. The method of claim 50 wherein the transaction bookmark comprises a transaction trigger, the transaction trigger initiating the executable action based on one of an associated input action applied to a pixel location and a value defined within an input filed.

53. The method of claim 52 wherein the transaction trigger comprises an automatic trigger, the associated input action being automatically generated to initiate the executable transaction.

54. The method of claim 50 wherein the transaction bookmark comprises an executable input action to be applied to a rendered reference visual content element.

24. A method for generating on a server a multi-level set of bitmaps from an input bitmap, comprising the steps of:

determining a display attribute of a client device;

identifying a plurality of pixel regions of an input bitmap;

in response to the display attribute of the client device, performing a first transform operation on each of the plurality of pixel regions to generate a plurality of bitmaps;

defining a plurality of data elements, each data element corresponding to a respective one of the plurality of bitmaps, each of the data elements defining a relationship of the respective bitmap to the input bitmap; and

transmitting to the client the plurality of bitmaps and the plurality of data elements.

31. A server for providing a visual content element to a client device comprising:

a proxy module for representing a proxy display;

a transformation module in communication with the proxy module, the transformation module generating a bitmap in response to an input bitmap and a display attribute of the client device; and

a communications module in communication with the transformation module, the communications module transmitting the bitmap to the client device for display thereon.

32. The server of claim 31 wherein the transformation module comprises a scaling module.

33. The server of claim 31 further comprising a client attribute module in communication with the transformation module, the client attribute module determining the display attribute of the client device.

34. The server of claim 31 further comprising a data element generator in communication with the transformation module, the data element generator producing a data element based on the display attribute.

35. The server of claim 31 wherein the communications module is configured to transmit the data element to the client device.

36. The server of claim 31 wherein the display attribute comprises data describing a constituent component of the visual content element.

37. The server of claim 36 wherein the communications module is configured to transmit a sub-region coordinate of the constituent component.

39. The server of claim 38 wherein the communications module is configured to transmit the first bitmap, the second bitmap and the data element to the client device.

40. The server of claim 38 wherein the transformation module comprises a scaling module.

41. The server of claim 38 wherein the transformation module is configured to perform a second transform operation on the pixel region to generate the second bitmap.

38. A server for generating a multi-level set of bitmaps from an input bitmap comprising:

a proxy module for representing a proxy display;

a communications module configured to receive pixel region selection data from a client device; and

a transformation module in communication with the proxy module and the communications module, the transformation module performing a transform operation on a pixel region of the input bitmap to generate a first bitmap in response to the pixel region selection data, the transformation module generating a data element corresponding to a second bitmap, the data element defining a relationship of the first bitmap to the second bitmap.

42. A method for generating on a server a multi-level set of bitmaps from an input bitmap, comprising the steps of:

receiving an input bitmap at a client device;

selecting a pixel region of the input bitmap;

determining a display attribute of the client device;

in response to the display attribute of the client device, performing a first transform operation on the pixel region to generate a first bitmap; and

transmitting to the client device the first bitmap.

43. The method of claim 42 further comprising the steps of:

defining a data element corresponding to a second bitmap, the data element defining a relationship of the first bitmap to the second bitmap; and

transmitting to a client the second bitmap representation and the data element.

55. A method for generating on a server a composite rasterized image from a set of input images, comprising the steps of:

determining a display attribute of a client device;

selecting a pixel region of a first input bitmap;

selecting a pixel region of a second input bitmap;

in response to the display attribute of the client device, performing a transform operation on the first and second input bitmaps to generate a composite bitmap; and

transmitting the composite bitmap to the client device for display thereon.

57. The method of claim 56 wherein the second part of the represented visual content is text-related content.

58. The method of claim 56 further comprising the step of transcoding the second part of the represented visual content element prior to transmission.

56. A method of displaying visual content on a client device, comprising the steps of:

establishing a communications link between the client device and a server;

determining a display attribute of the client device;

representing a visual content element on a proxy display surface of the server;

transforming a first part of the represented visual content element to a first bitmap based on the display attribute of the client device;

transmitting the first bitmap to the client device for display thereon;

transmitting a second part of the represented visual content element to the client device; and

transforming the second part of the represented visual content element to a second bitmap at the client device for display thereon.

59. A method of displaying visual content on a client device, comprising the steps of:

establishing a communications link between the client device and a server;

determining a display attribute of the client device;

representing a visual content element on a proxy display surface of the server;

performing a first transformation on a first portion of the represented visual content element to generate a first bitmap based on the display attribute of the client device;

performing a second transformation on a second portion of the represented visual content element to generate a second bitmap based on the display attribute of the client device; and

transmitting the first and second bitmaps to the client device for display thereon.