Managing Multimodal Annotations Of An Image

• US 20100083153A1
• Filed: 09/29/2008
• Published: 04/01/2010
• Est. Priority Date: 12/07/2007
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Indrani, Menaka, Li, Jun, Jain, Jhilmil
• Granted Patent: US 8,898,558 B2
• Time in Patent Office: Days
• US Class Current: 715/765
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

Methods and computer readable mediums storing computer executable programs for managing multimodal annotations of an image are disclosed. A first communication channel is established between a first machine and a mobile device. A first hierarchical data structure generated by the mobile device is received at the first machine. The first hierarchical data structure includes an image annotated with at least one multimodal annotation. A representation of the first hierarchical data structure is rendered at the first machine. At least one multimodal annotation modification associated with the image is received at the first machine. The at least one multimodal annotation modification is incorporated into the first hierarchical data structure thereby generating a second hierarchical data structure.

Claims

1. A method of managing multimodal annotations of an image, the method comprising:

establishing a first communication channel between a first machine and a mobile device;

receiving a first hierarchical data structure generated by the mobile device at the first machine, the first hierarchical data structure including an image annotated with at least one multimodal annotation;

rendering a representation of the first hierarchical data structure at the first machine;

receiving at least one multimodal annotation modification associated with the image at the first machine; and

incorporating the at least one multimodal annotation modification into the first hierarchical data structure thereby generating a second hierarchical data structure.

2. The method of claim 1, further comprising providing a rendering module at the first machine, the rendering module being operable to render the representation of the first hierarchical data structure at the first machine.

3. The method of claim 1, further comprising providing an image annotation module at the first machine, the image annotation module being operable to incorporate the at least one multimodal annotation modification into the first hierarchical data structure thereby generating a second hierarchical data structure at the first machine.

4. The method of claim 1, wherein the first and second hierarchical data structures each include at least one multimodal annotation having a multimodal annotation type selected from a group consisting of an image annotation, a voice annotation, a video annotation, a structured text annotation, a barcode annotation, a free hand text annotation, a free hand sketch annotation, and an audio annotation.

5. The method of claim 1, wherein the first hierarchical data structure including an image annotated with at least one multimodal annotation comprises a first hierarchical data structure including an image annotated with a first multimodal annotation associated with a selected area of the image.

6. The method of claim 1, wherein the at least one multimodal annotation modification is selected from a group consisting of a multimodal annotation addition, a multimodal annotation deletion, and a multimodal annotation revision.

7. The method of claim 1, further comprising:

establishing a second communication channel between the first machine and the mobile device; and

transmitting the second hierarchical data structure from the local machine to the mobile device via the second communication channel.

8. The method of claim 1, further comprising storing the second hierarchical data structure at the first machine.

9. The method of claim 1, further comprising:

establishing a second communication channel between the first machine and the backend system; and

transmitting the received first hierarchical data structure from the first machine to the backend system via the second communication channel for storage at the backend system.

10. The method of claim 9, further comprising providing a rendering module at the backend system, the rendering module being operable to render the representation of the first hierarchical data structure at the first machine.

11. The method of claim 10, further comprising:

receiving the at least one multimodal annotation modification from the first machine at the backend system;

providing an image annotation module at the backend system, the image annotation module being operable to incorporate the at least one multimodal annotation modification into the first hierarchical data structure thereby generating a second hierarchical data structure;

employing the rendering module to render a representation of the second hierarchical data structure at the first machine; and

storing the second hierarchical data structure at the backend system.

12. The method of claim 11, further comprising:

establishing a third communication channel between a second machine and the backend system; and

employing the rendering module to render a representation of the second hierarchical data structure at the second machine.

13. A computer readable medium for storing a computer executable program for managing multimodal annotations of an image, the computer readable medium comprising:

computer readable code for establishing a first communication channel between a first machine and a mobile device;

computer readable code for receiving a first hierarchical data structure generated by the mobile device at the first machine, the first hierarchical data structure including an image annotated with at least one multimodal annotation;

computer readable code for rendering a representation of the first hierarchical data structure at the first machine;

computer readable code for receiving at least one multimodal annotation modification associated with the image at the first machine; and

computer readable code for incorporating the at least one multimodal annotation modification into the first hierarchical data structure thereby generating a second hierarchical data structure.

14. The computer readable medium of claim 13, wherein the computer readable code for receiving a first hierarchical data structure comprises computer readable code for receiving a first hierarchical data structure including an image annotated with a first multimodal annotation having a multimodal annotation type selected from a group consisting of an image annotation, a voice annotation, a video annotation, a structured text annotation, a barcode annotation, a free hand text annotation, a free hand sketch annotation, and an audio annotation.

15. The computer readable medium of claim 13, wherein the computer readable code for receiving a first hierarchical data structure generated by the mobile device at the first machine comprises computer readable code for receiving a first hierarchical data structure generated by the mobile device at the first machine, the first hierarchical data structure including an image annotated with a first multimodal annotation associated with a selected area of the image.

16. The computer readable medium of claim 13, wherein the computer readable code for receiving at least one multimodal annotation modification associated with the image at the first machine comprises computer readable code for receiving a first multimodal annotation modification selected from a group consisting of a multimodal annotation addition, a multimodal annotation deletion, and a multimodal annotation revision.

17. The computer readable medium of claim 13, further comprising:

computer readable code for establishing a second communication channel between the first machine and the mobile device; and

computer readable code for transmitting the second hierarchical data structure from the local machine to the mobile device via the second communication channel.

18. The computer readable medium of claim 13, further comprising computer readable code for storing the second hierarchical data structure at the first machine.

19. The computer readable medium of claim 13, further comprising:

computer readable code for establishing a second communication channel between the first machine and the backend system; and

computer readable code for transmitting the received first hierarchical data structure from the first machine to the backend system via the second communication channel for storage at the backend system.

20. The computer readable medium of claim 19, further comprising computer readable code for receiving a rendering of the representation of the first hierarchical data structure from the backend system at the first machine.

21. The computer readable medium of claim 20, further comprising:

computer readable code for transmitting the at least one multimodal annotation modification from the first machine to the backend system, the backend system being operable to incorporate the at least one multimodal annotation modification into the first hierarchical data structure thereby generating a second hierarchical data structure and to store the second hierarchical data structure; and

computer readable code for receiving a rendering of the second hierarchical data structure from the backend system at the first machine.

22. The computer readable medium of claim 21, further comprising:

computer readable code for establishing a third communication channel between a second machine and the backend system; and

computer readable code for receiving a rendering of a representation of the second hierarchical data structure at the second machine.

23. A method of managing multimodal annotations of an image, the method comprising:

receiving an image at a mobile device;

receiving at least one multimodal annotation of the image at the mobile device;

generating a first hierarchical data structure including the image and the at least one multimodal annotation at the mobile device;

transmitting the first hierarchical data structure from the mobile device to a first machine;

receiving a second hierarchical data structure including the image and at least one multimodal annotation of the image from the first machine at the mobile device, the second hierarchical data having been generated by the incorporation of at least one multimodal annotation modification received at the first machine into the first hierarchical data structure.

24. The method of claim 23, wherein further comprising:

establishing a communication channel with a backend system via the first machine; and

transmitting the first hierarchical data structure from the mobile device to the backend system via the communication channel for storage at the backend system.

25. The method of claim 23, wherein receiving at least one multimodal annotation of an image at the mobile device comprises receiving a first multimodal annotation associated with a selected area of the image.