Personalizing medical conditions with augmented reality

• US 9,715,753 B2
• Filed: 02/04/2016
• Issued: 07/25/2017
• Est. Priority Date: 01/23/2013
• Status: Active Grant

• Current Assignee: Cochlear Limited
• Original Assignee: Orca Health, Inc.
• Inventors: Berry, Matthew M., Lyman, Daniel D., Robinson, Gary L.
• Primary Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US15/016,116
• Publication Number:
  US 20160232695A1
• Time in Patent Office: 537 Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Augmented reality is used to simulate the impact of medical conditions on body parts and other objects within images taken of the objects. The simulations enable a user to see how a medical condition can affect the user by dynamically simulating the impact of the medical condition on captured images of body parts associated with the user in real-time. A user can select different medical conditions that are associated with different body parts. These objects are then identified within images containing the body parts using image recognition algorithms and/or user input. Thereafter, the images are modified so as to render the body parts as though the body parts were being impacted by the medical condition. The modifications are made by blending image data of the captured image with condition image data available to the processing system.

Claims

1. A storage device having stored computer-executable instructions which, when executed by at least one computing processor, implement a method for dynamically modifying an image containing an anatomical object that is displayed on a display screen of a computing system that includes a camera that is used to capture the image that is displayed and modified according to the method, the method comprising:

using the camera of the computing system to capture the image containing the anatomical object;

using the computing system to automatically detect the presence of the anatomical object within the captured image, the anatomical object being associated with a medical condition;

displaying the captured image on a display screen associated with the computing system, the captured image being rendered in real-time with the automatically detected anatomical object; and

an act of the computer system, in response to detecting user input manipulating a control for rendering a simulation at the computing system, rendering the simulation of the medical condition associated with the automatically detected anatomical object at the display screen, wherein the simulation dynamically reflects a simulated impact of the medical condition associated with the corresponding anatomical object by displaying a progressive stage of the medical condition from the capture image.

2. The storage device recited in claim 1, wherein the computing system is a mobile computing system, wherein the mobile computing system includes the camera used to capture the image containing the anatomical object, and wherein the mobile computing system also includes the display screen wherein the simulation is rendered.

3. The storage device recited in claim 1, wherein the method further includes receiving user input manipulating the control at the computing system subsequent to displaying the captured image and prior to rendering the simulation.

4. The storage device recited in claim 3, wherein the simulation is rendered in response to receiving user input selecting a play button.

5. The storage device recited in claim 1, wherein simulation of the medical condition consists of modifying a single display frame rendering the captured image as a static image.

6. The storage device recited in claim 1, wherein the simulation of the medical condition comprises the modifying of a plurality of sequential display frames used to render a plurality of the captured images on the display screen, including the captured image, as an animation.

7. The storage device recited in claim 6, wherein similar modifications are made to correspondingly related portions of the plurality of sequential display frames, the correspondingly related portions of the plurality of sequential display frames rendering similar image data associated with the anatomical object at different portions on the display screen while the anatomical object is moved relative to the camera.

8. The storage device recited in claim 6, wherein different modifications are made to correspondingly similar portions of the plurality of sequential display frames to reflect a dynamic change in the simulation of the medical condition, while the anatomical object remains in a same position relative to the camera such that the correspondingly similar portions of the plurality of sequential display frames rendering similar image data associated with the anatomical object.

9. The storage device recited in claim 1, wherein the medical condition is a degenerative or harmful condition, and wherein the simulation dynamically reflects the simulated impact of the medical condition over time by displaying different, progressive stages of the medical condition over time and transitioning between the different, progressive stages of the medical condition over time to show the progression of the medical condition over time.

10. The storage device recited in claim 9, wherein the medical condition is at least one of conjunctivitis, gingival recession, gingivitis, or periodontitis.

11. The storage device recited in claim 1, wherein the medical condition is a treatment or beneficial condition, and wherein the simulation dynamically reflects the simulated impact of the medical condition over time by displaying different, progressive stages of the medical condition over time and transitioning between the different, progressive stages of the medical condition over time to show the progression of the medical condition over time.

12. The storage device recited in claim 1, wherein the rendering of the simulation of the medical condition with the anatomical object includes blending image data of the captured image with stored condition image data associated with the identified condition.

13. The storage device recited in claim 1, wherein the indicator comprises a highlighting element that highlights the automatically detected anatomical object in the captured image.

14. The storage device recited in claim 1, wherein the method further includes receiving user input for identifying the medical condition.

15. The storage device recited in claim 1, wherein the simulated impact of the medical condition on the corresponding automatically detected anatomical object is based on a user selected severity of the medical condition.

16. The storage device recited in claim 1, wherein the medical condition is at least one of conjunctivitis, gingival recession, gingivitis, or periodontitis.

17. The storage device recited in claim 1, wherein the storage device further stores one or more data structures correlating the identified medical condition with the anatomical object and the anatomical object with image characteristics associated with the anatomical object.

18. The storage device recited in claim 1, wherein the anatomical object comprises at least one of an eye and a mouth.

19. A computer implemented method for dynamically modifying an image of an anatomical object that is displayed on a display screen of a computing system, the method comprising:

obtaining an image containing an anatomical object;

using the computing system to automatically detect the presence of the anatomical object within the captured image, the anatomical object being associated with a medical condition;

displaying the captured image on a display screen associated with the computing system, the captured image being rendered with the automatically detected anatomical object and an indicator that the automatically detected anatomical object has been automatically detected; and

an act of the computer system, in response to detecting user input manipulating a control for rendering a simulation at the computing system, rendering the simulation of the medical condition associated with the automatically detected anatomical object at the display screen, wherein the simulation dynamically reflects a simulated impact of the medical condition over time by displaying, with the simulation, different progressive stages of the medical condition over time and transitioning between the different progressive stages of the medical condition over time to show the progression of the medical condition over time.

20. A computing device comprising:

at least one hardware processor;

a display screen; and

one or more storage media having stored computer-executable instructions which, when executed by the at least one hardware processor, implement a method for dynamically modifying an image of a body part that is displayed on a display screen associated with the computing device, the method comprising;

obtaining an image containing an anatomical object comprising a body part;

detecting the presence of the body part within the image, the anatomical object being associated with a medical condition;

displaying the image on a display screen associated with the computing system, the image being rendered with the body part in a way that highlights the body part and in such a way as to indicate that the body part has been detected within the image; and

rendering augmented reality on the display screen, the augmented reality comprising a dynamic simulation of the medical condition that modifies at least a portion of the captured image of the body part and that is rendered as an animation with a plurality of different display frames which reflects the simulated impact of the medical condition, over time, on the image of the body part.