Personalizing medical conditions with augmented reality

• US 9,256,962 B2
• Filed: 01/23/2013
• Issued: 02/09/2016
• Est. Priority Date: 01/23/2013
• Status: Active Grant

• Current Assignee: Cochlear Limited
• Original Assignee: Orca Health, Inc.
• Inventors: Berry, Matthew M., Lyman, Daniel D., Robinson, Gary L.
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US13/747,595
• Publication Number:
  US 20140204118A1
• Time in Patent Office: 1,112 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Augmented reality is used to simulate the impact of medical conditions on body parts and other objects within images taken of the objects. The simulations enable a user to see how a medical condition can affect the user by dynamically simulating the impact of the medical condition on captured images of body parts associated with the user in real-time. A user can select different medical conditions that are associated with different body parts. These objects are then identified within images containing the body parts using image recognition algorithms and/or user input. Thereafter, the images are modified so as to render the body parts as though the body parts were being impacted by the medical condition. The modifications are made by blending image data of the captured image with condition image data available to the processing system.

Claims

1. A storage device having stored computer-executable instructions which, when executed by at least one computing processor, implement a method for dynamically modifying an image of an anatomical object that is displayed on a display screen of a computing system that includes a camera that is used to capture the image that is displayed and modified according to the method, the method comprising:

an act of the computer system identifying a medical condition in response to received user input, the identified medical condition being associated with an anatomical object;

an act of the computer system, in response to identifying the medical condition, determining the anatomical object that is associated with the medical condition;

using a camera of the computing system to capture an image containing the anatomical object;

using the computing system to automatically detect the presence of the anatomical object within the captured image;

displaying the captured image on a display screen associated with the computing system, the captured image being rendered in real-time with the automatically detected anatomical object and an indicator that the automatically detected anatomical object has been automatically detected; and

an act of the computer system, in response to detecting user input manipulating a control for rendering a simulation at the computing system, rendering a simulation of the medical condition with the automatically detected anatomical object at a display screen, wherein the simulation dynamically reflects the simulated impact of the medical condition, over time, on the corresponding anatomical object.

2. The storage device recited in claim 1, wherein the computing system is a mobile computing system.

3. The storage device recited in claim 2, wherein the mobile computing system includes the camera used to capture the image containing the anatomical object, and wherein the mobile computing system also includes the display screen wherein the simulation is rendered.

4. The storage device recited in claim 1, wherein the method further includes receiving user input manipulating the control at the computing system subsequent to displaying the captured image and prior to rendering the simulation.

5. The storage device recited in claim 4, wherein the simulation is rendered in response to receiving user input selecting a play button.

6. The storage device recited in claim 1, wherein simulation of the medical condition consists of modifying a single display frame rendering the captured image as a static image.

7. The storage device recited in claim 1, wherein the simulation of the medical condition comprises the modifying of a plurality of sequential display frames used to render a plurality of the captured images on the display screen, including the captured image, as an animation.

8. The storage device recited in claim 7, wherein similar modifications are made to correspondingly related portions of the plurality of sequential display frames, the correspondingly related portions of the plurality of sequential display frames rendering similar image data associated with the anatomical object at different portions on the display screen while the anatomical object is moved relative to the camera.

9. The storage device recited in claim 7, wherein different modifications are made to correspondingly similar portions of the plurality of sequential display frames to reflect a dynamic change in the simulation of the medical condition, while the anatomical object remains in a same position relative to the camera such that the correspondingly similar portions of the plurality of sequential display frames rendering similar image data associated with the anatomical object.

10. The storage device recited in claim 1, wherein the medical condition is a degenerative or harmful condition.

11. The storage device recited in claim 1, wherein the medical condition is a treatment or beneficial condition.

12. The storage device recited in claim 1, wherein the rendering of the simulation of the medical condition with the anatomical object includes blending image data of the captured image with stored condition image data associated with the identified condition.

13. The storage device recited in claim 1, wherein the indicator comprises a highlighting element that highlights the automatically detected anatomical object in the captured image.

14. The storage device recited in claim 1, wherein receiving the user input for identifying the medical condition includes receiving input selecting the medical condition from a user interface menu.

15. The storage device recited in claim 1, wherein the act of automatically detecting the presence of the anatomical object includes automatically distinguishing the automatically detected anatomical object from at least one other anatomical object that is included in the captured image.

16. The storage device recited in claim 15, wherein the at least one other anatomical object is associated with a different medical condition.

17. The storage device recited in claim 15, wherein the at least one other anatomical object is associated with the identified medical condition.

18. The storage device recited in claim 1, wherein the storage device further stores one or more data structures correlating the identified medical condition with the anatomical object and the anatomical object with image characteristics associated with the anatomical object.

19. The storage device recited in claim 1, wherein the anatomical object comprises at least one of an eye and a mouth.

20. The storage device recited in claim 1, wherein the simulation dynamically reflects the simulated impact of the medical condition over time by displaying different, progressive stages of the medical condition over time and transitioning between the different, progressive stages of the medical condition over time to show the progression of the medical condition over time.

21. The storage device recited in claim 1, wherein the simulated impact of the medical condition on the corresponding automatically detected anatomical object is based on a user selected severity of the medical condition.

22. The storage device recited in claim 1, wherein the medical condition is at least one of conjunctivitis, gingival recession, gingivitis, or periodontitis.

23. A computer implemented method for dynamically modifying an image of an anatomical object that is displayed on a display screen of a computing system that includes a camera that is used to capture the image that is displayed and modified according to the method, the method comprising:

identifying a medical condition in response to received user input, the identified medical condition being associated with an anatomical object;

in response to identifying the medical condition, determining the anatomical object that is associated with the medical condition;

initiating an image capture mode of a camera associated with the computing system in response to identifying the medical condition;

using the camera to capture an image containing the anatomical object;

using the computing system to automatically detect the presence of the anatomical object within the captured image;

displaying the captured image on a display screen associated with the computing system, the captured image being rendered with the automatically detected anatomical object and an indicator that the automatically detected anatomical object has been automatically detected; and

in response to detecting user input manipulating a control for rendering a simulation at the computer system, rendering a simulation of the medical condition with the automatically detected anatomical object at a display screen, wherein the captured image and simulation are displayed with a control for selectably controlling the rendering of the medical condition as an animation, and the animation dynamically reflects the simulated impact of the medical condition, over time, on the corresponding anatomical object.

24. A computing device comprising:

at least one hardware processor;

a camera;

a display screen; and

one or more storage media having stored computer-executable instructions which, when executed by the at least one hardware processor, implement a method for dynamically modifying an image of a body part that is displayed on a display screen associated with the computing device, the method comprising;

identifying a medical condition in response to received user input, the identified medical condition being associated with a body part;

using the camera to capture an image containing the body part;

displaying the captured image on the display screen, the captured image being rendered with the body part in a way that highlights the body part and in such a way as to indicate that the body part has been automatically detected within the captured image; and

rendering augmented reality on the display screen, the augmented reality comprising a dynamic simulation of the medical condition that modifies at least a portion of the captured image of the body part and that is rendered as an animation with a plurality of different display frames which reflects the simulated impact of the medical condition, overtime, on the associated body part, wherein the rendering occurs at substantially a same time that the image containing the body part is captured and displayed by the computing system.