System and method for visual annotation and knowledge representation

• US 20060061595A1
• Filed: 05/31/2003
• Published: 03/23/2006
• Est. Priority Date: 05/31/2002
• Status: Active Grant

• Current Assignee: XIFIN, Inc.
• Original Assignee: University of Utah Research Foundation (State of Utah)
• Inventors: Cochella, Christopher, Goede, Patricia Anne, Lauman, Jason R
• Granted Patent: US 7,453,472 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

A method and system for visually annotating an image. Annotations and notes to images, such as digital medical and healthcare images, may be stored in a structured vector representation alongside image information in a single, non-volatile and portable file or in a separate file from the image. The annotations may be composed of point, line and polygon drawings and text symbols, labels or definitions and captions or descriptions. The annotations may be structured in a manner that facilitates grouping and manipulation as user defined groups. The annotations may be related to an image but not inextricably bound such that the original image is completely preserved. Annotations may further be selectively displayed on the image for context appropriate viewing. The annotations may be retrieved for purposes such as editing, printing, display, indexing and reporting for example, and may be displayed on an image for interactive use with an embedded self-contained user interface.

Claims

1. A method for visually annotating a digital image, said digital image being raster based, said method comprising the steps of:

providing a digital image;

annotating said digital image;

storing electronically said annotations as vector based information linked to said digital image such that the digital image remains in a substantially unaltered condition; and

rendering said digital image with the annotations for display such that the annotations can be perceived by a user.

2. The method of claim 1, wherein the digital image is an image of at least a portion of a human body.

3. The method of claim 2, wherein the step of annotating the digital image includes the step of defining a region of interest and adding textual information.

4. The method of claim 1, wherein the step of storing comprises storing said annotations in a second file and said digital image is stored in a first file.

5. The method of claim 1, wherein the step of storing comprises storing said annotations in the same file which contains the digital image.

6. The method of claim 1, wherein step of storing comprises storing said annotations as text information.

7. The method of claim 6, wherein the annotations are stored in an XML compatible format.

8. The method of claim 1, wherein said annotations comprise at least one member selected from the group consisting of a region of interest, a pointer, a symbol, a caption, a label and an abbreviation.

9. The method of claim 8, further comprising the step of hierarchically organizing the annotations into logical groupings pursuant to a user defined structure.

10. The method of claim 8, wherein at least part of the annotation utilizes user defined lexicons.

11. The method of claim 1, wherein metadata is also stored along with the annotations.

12. The method of claim 1, wherein the annotation comprises a region of interest, said region of interest being defined by one selected from the group consisting of:

a set of points, a polygon, and a polyline.

13. The method of claim 1 wherein the step of rendering the image with the annotations comprises the step of displaying the annotations in a context appropriate display.

14. The method of claim 1 wherein each of the annotations can be selectively chosen for display on the digital image during the rendering step.

15. The method of claim 1 wherein each of the annotations can be displayed according to a predefined user grouping.

16. The method of claim 1 wherein the annotations are uniquely identified.

17. The method of claim 1 wherein the digital image comprises a sequence of digital images.

18. A method for interactively displaying annotations linked to a digital image, the annotations stored as vector information, said method comprising the steps of:

selecting and displaying the digital image on a display; and

generating a graphical user interface on the display to accept user preferences, said graphical user interface allowing a user to toggle on and off any grouping of the annotations through the graphical user interface, the annotations being visible when toggled on and hidden from view when toggled off.

19. The method of claim 18 wherein the digital image is a raster based image.

20. The method of claim 18 wherein the graphical user interface further allows a user to activate and deactivate spatial contextual labels.

21. The method of claim 18 wherein the graphical user interface allows a user to pan the digital image and zoom in and out.

22. The method of claim 18 wherein the graphical user interface allows a user to toggle on and off annotations from a particular author.

23. The method of claim 18 wherein the graphical user interface allows a user to toggle on and off annotations from a particular specialty.

24. The method of claim 18 wherein the groupings are predefined by the user.

25. The method of claim 18 wherein the digital image remains in a substantially unaltered format.

26. The method of claim 18 wherein said annotations comprise at least one member selected from the group consisting of a region of interest, a pointer, a symbol, a caption, a label and an abbreviation.

27. The method of claim 26 wherein the annotations comprise entries from a previously defined lexicon.

28. A method for annotating a digital image, said method comprising the steps of:

displaying a digital image on a computer display;

annotating said digital image;

grouping the annotations hierarchically pursuant to a user defined structure; and

storing said annotations as vector based information linked to said digital image such that the digital image remains in a substantially unaltered condition.

29. The method of claim 28 wherein the step of grouping the annotations hierarchically allows the annotation to be displayed in context appropriately.

30. The method of claim 28, wherein the annotations are saved in a format that can be electronically queried.

31. The method of claim 30, wherein the annotations are capable of being retrieved for interactive display.

32. The method of claim 28 wherein metadata is stored along with the annotations.

33. The method of claim 32 wherein the metadata uniquely identifies the annotations.

34. The method of claim 32 wherein the metadata comprises at least one selected from the group consisting of:

the name of the author or creator of each of the annotations, a date indicating when annotation was added to the digital image, a title, a subject, a description, and an area of specialty of the author or creator.

35. A method for managing digital images, said method comprising:

inputting annotations for each of the digital images;

saving each of the annotations in a textual format linked to the digital image such that the digital image remains substantially unaltered;

generating a query, said query querying the annotations;

retrieving the appropriate annotations and their linked digital images, if any, in response to the query; and

displaying the retrieved digital images and their associated annotations.

36. The method of claim 35 wherein the step of inputting annotations comprises defining a region of interest on each of the associated digital images.

37. The method of claim 35 further comprising the step of hierarchically grouping the annotations pursuant to a user defined structure and the step of generating a query utilizing said user defined structure.

38. The method of claim 35 wherein the step of displaying the retrieved digital images comprises the step of interactively displaying the annotations associated with the retrieved digital images.

39. The method of claim 38 wherein the annotations can be toggled on and off, said annotations being visible when toggled on and hidden from view when toggled off.

40. The method of claim 39 wherein annotations can be displayed pursuant to a hierarchal grouping.

41. A system for visually annotating digital data stored in a electronic file, the system comprising:

an extraction means for extracting data from the electronic file to create a visual representation of the digital data;

an organizing means for assembling the extracted data into a visual representation that reflects the digital data;

a constructing means for permitting a human user to manually create dimensionally appropriate annotations;

a storing means for electronically storing the annotations as vector information such that the digital data stored in the electronic file is substantially unaltered; and

a presenting means for displaying the visual representation of the digital data along with the annotations.

42. The system of claim 41, wherein the extraction means identifies the format of the electronic file based upon the electronic file extension, MIME type as embedded into the electronic file or provided externally as metadata, or user input.

43. The system of claim 41, wherein the visual representation of the digital data is a single visual data element that has the dimensions of the original data.

44. The system of claim 41, wherein the visual representation of the digital data is a sequence of visual data elements that each has the dimension of the original digital data, but has an additional dimension that defines a sequence axis.

45. The system of claim 41, wherein the visual representation of the digital data is a set of visual data elements that each has the dimension of the original digital data, but has an additional channel dimension where each visual data element represents the same conceptual space.

46. The system of claim 41, wherein the visual representation of the digital data is a group of visual data elements that each has the dimension of the original digital data, but each visual data element is unique in the conceptual space that merits a grouping to visualize the similarities or differences of the visual elements.

47. The system of claim 41 further comprising an identifier means for marking each annotation with user identifying information to track authorship of the respective annotation.

48. The system of claim 41 wherein the presenting means is capable of restricting the display of at least a portion of the annotations based upon previously defined viewing rights assigned to each user.

49. The system of claim 41 wherein the presenting means is capable of restricting the display of at least a portion of the annotations based upon user input.

50. A system for annotating an image, said system comprising:

means for storing a raster based digital image in a file;

means for displaying said raster based digital image;

means for creating vector based annotations related to said image; and

means for linking said vector based annotations to said digital image such that said annotations are related to said digital image and said digital image is preserved in its original condition.

51. The system of claim 50, further comprising a presenting means for displaying said digital image and associated annotations.

52. The system of claim 51 wherein the presenting means further allows a user to selectively display the annotations.

53. The system of claim 52 wherein the annotations can be selectively displayed pursuant to a predefined hierarchal relationship.

54. The system of claim 52 wherein the annotations can be selectively displayed pursuant to by specialty.

55. The system of claim 52 wherein the annotations can be selectively displayed pursuant to user input received through a graphical user interface.

56. The system of claim 51 wherein the presenting means is an embedded self-contained user interface.