Visually augmenting a graphical rendering of a chemical structure representation or biological sequence representation with multi-dimensional information

• US 8,854,361 B1
• Filed: 10/21/2013
• Issued: 10/07/2014
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Revvity Signals Software Incorporated
• Original Assignee: Cambridgesoft Corporation (Perkinelmer Incorporated)
• Inventors: Smith, Robin Y.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wu, Sing-Wai
• Application Number:
  US14/058,865
• Publication Number:
  US 20140267240A1
• Time in Patent Office: 351 Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2380/08 Biomedical applications
• CPC Class Codes: G16B 45/00 ICT specially adapted for bioinformatics-related data visualisation, e.g. displaying of maps or networks
• CPC Class Codes: G16B 5/00 ICT specially adapted for modelling or simulations in systems biology, e.g. gene-regulatory networks, protein interaction networks or metabolic networks
• CPC Class Codes: G16C 10/00 Computational theoretical chemistry, i.e. ICT specially adapted for theoretical aspects of quantum chemistry, molecular mechanics, molecular dynamics or the like
• CPC Class Codes: G16C 20/80 Data visualisation

Abstract

In certain embodiments, the invention relates to systems, methods, and apparatus that allow a user to visually augment a graphical rendering of either a chemical structure representation or a biological sequence representation with multi-dimensional information. A user captures a video image using a computing device such as a hand-held smart phone, computerized eye glasses or tablet computer. The video image includes information regarding at least one of a chemical structure and a biological sequence. A processor identifies, within the video image, a graphical representation of at least one of a chemical structure and a biological structure. The processor augments the graphical representation with additional information and provides the video data for presentation upon a display controlled by the computing device. The computing device presents the video data in substantially real time in relation to the capture of the video data by the computing device.

Claims

1. A method comprising:

receiving video data captured by a first computing device, wherein the video data comprises a sequence of images comprising a first video frame;

identifying, within the first video frame, by a processor of a second computing device, a graphical representation of one of a chemical structure and a biological sequence;

matching, by the processor, additional information to the graphical representation, wherein the additional information comprises at least one of

(i) a three-dimensional representation of the chemical structure or biological sequence,

(ii) a name of

a) a chemical compound represented by the chemical structure, or

b) a biological species comprising a DNA molecule, an RNA molecule, or a polypeptide represented by the biological sequence, and

(iii) a plurality of properties of the chemical compound or biological species;

rendering, by the processor, augmentation data corresponding to the additional information, wherein the augmentation data comprises one or both of (i) and (ii);

(i) graphics corresponding to at least a portion of the additional information, and

(ii) one or more graphical controls configured, upon selection, to present at least a portion of the additional information; and

merging, by the processor, the video data with the rendered augmentation data to positionally link the augmentation data to the graphical representation of the chemical structure or the biological sequence identified within the first video frame and providing the video data for presentation upon a display controlled by the first computing device, wherein the video data is presented by the first computing device in substantially real time in relation to the capture of the video data by the first computing device.

2. The method of claim 1, wherein the first computing device comprises the second computing device.

3. The method of claim 1, wherein the graphical representation of the chemical structure comprises a two-dimensional drawing of the chemical structure.

4. The method of claim 1, wherein merging the video data with the rendered augmentation data comprises:

determining an orientation of the first computing device; and

rendering the three-dimensional representation based in part upon the orientation of the first computing device.

5. The method of claim 1, wherein the additional information further comprises (iv) a storage location.

6. The method of claim 5, wherein matching additional information to the graphical representation comprises determining the storage location, wherein determining the storage location comprises identifying the chemical compound or biological species within a catalog.

7. The method of claim 1, wherein the rendering and the merging comprises rendering a first graphical control of the one or more graphical controls as a semi-transparent image and overlaying the semi-transparent image upon the first video frame.

8. The method of claim 1, wherein merging the video data with the rendered augmentation data comprises superimposing the graphical representation with at least a portion of the additional information.

9. The method of claim 1, wherein matching the additional information to the graphical representation comprises comparing the graphical representation to one or more stored graphical representations.

10. The method of claim 9, wherein matching the additional information to the graphical representation comprises digitally refining the portion of the graphical representation prior to comparing the graphical representation to the one or more stored graphical representations.

11. The method of claim 1, further comprising:

receiving, by the processor, an indication of selection of a first graphical control of the one or more graphical controls;

rendering, by the processor, augmentation data corresponding to the additional information related to the first graphical control; and

merging, by the processor, subsequent video data with the portion of the rendered augmentation data related to the first graphical control.

12. The method of claim 11, wherein merging the subsequent video data comprises adding audio data to the video data, wherein the audio data comprises a verbal description of the additional information.

13. The method of claim 1, wherein merging the video data with the rendered augmentation data comprises positioning the augmentation data in proximity to the graphical representation of the chemical structure or the biological sequence identified within the first video frame.

14. A system comprising:

a processor; and

a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to;

receive video data captured by a computing device, wherein the video data comprises a sequence of images comprising a first video frame;

identify, within the first video frame, a visual identification of a chemical structure or a biological sequence;

match additional information to the visual identification, wherein the additional information comprises at least one of

(i) a graphical representation of the chemical structure or biological sequence,

(ii) a name of

a) a chemical compound represented by the chemical structure, or

b) a biological species comprising a DNA molecule, an RNA molecule, or a polypeptide represented by the biological sequence, and

(iii) a plurality of properties of the chemical compound or biological species;

render augmentation data corresponding to the additional information, wherein the augmentation data comprises one or both of (i) and (ii);

(i) graphics corresponding to at least a portion of the additional information, and

(ii) one or more graphical controls configured, upon selection, to present at least a portion of the additional information; and

merge the video data with the rendered augmentation data to positionally link the augmentation data to the visual representation of the chemical structure or the biological sequence identified within the first video frame, wherein

merging the video data with the rendered augmentation data comprises positionally linking the at least one of (i) the portion of the additional information and (ii) the one or more graphical controls to the visual identification such that, upon movement of the visual identification within the first video frame, the at least one of (i) the portion of the additional information and (ii) the one or more graphical controls undergo a corresponding motion; and

provide the video data for presentation upon a display;

wherein the video data is presented in substantially real time in relation to the capture of the video data by the computing device.

15. The system of claim 14, wherein the computing device comprises the processor.

16. The system of claim 14, wherein the visual identification comprises a two-dimensional drawing of a chemical structure or biological sequence.

17. The system of claim 14, wherein the instructions, when executed, cause the processor to:

receive, from a user of the computing device, an indication of video freeze; and

responsive to the indication of video freeze, provide for presentation a still image comprising a most recent augmented video data frame.

18. The system of claim 17, wherein the instructions, when executed, cause the processor to:

receive, after providing the still image, selection of a first graphical control of the one or more graphical controls;

apply, responsive to selection of the first graphical control, the portion of the additional information related to the first graphical control to the still image to determine a second still image; and

provide, for presentation upon the display, the second still image.

19. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to:

receive video data destined for display upon a computing device, wherein the video data comprises a sequence of images comprising a first video frame;

identify, within the first video frame, a graphical representation of one of a chemical structure and a biological sequence;

match additional information to the graphical representation, wherein the additional information is arranged in a plurality of feature categories;

render augmentation data corresponding to the additional information, wherein the augmentation data comprises one or more graphical controls configured, upon selection, to present at least a portion of the additional information belonging to a first category of the plurality of feature categories;

merge the video data with the one or more graphical controls to positionally link the augmentation data to the graphical representation of the chemical structure or the biological sequence identified within the first video frame;

after augmenting the video data, provide the video data for presentation upon a display controlled by the computing device;

receive, responsive to selection of a first graphical control of the one or more graphical controls by a user of the computing device, an indication corresponding to the first category; and

responsive to the indication, augment subsequent video data with at least a portion of the additional information belonging to the first category; and

after augmenting the subsequent video data, provide the subsequent video data for presentation upon the display controlled by the computing device.

20. The computer readable medium of claim 19, wherein the computing device comprises the processor.

21. The computer readable medium of claim 19, wherein:

the video data was captured by the computing device; and

the video data is presented by the computing device in substantially real time in relation to the capture of the video data by the computing device.