Methods of using an imaging apparatus in augmented reality, in medical imaging and nonmedical imaging

• US 10,607,420 B2
• Filed: 08/29/2018
• Issued: 03/31/2020
• Est. Priority Date: 08/30/2017
• Status: Active Grant

• Current Assignee: Dermagenesis, LLC
• Original Assignee: Dermagenesis, LLC
• Inventors: Vogel, Richard, Xu, Tony, Steinberg, Alexander, Cohen, Ran
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US16/116,211
• Publication Number:
  US 20190066390A1
• Time in Patent Office: 580 Days
• Field of Search: 345633
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30052 Implant; Prosthesis
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

With inventive processing making use of surface-reconstruction and capping steps, more imagery acquired by 3C cameras can be put to use in augmented reality applications, especially applications, such as medical reconstruction, in which a certain theoretical ideal fit might be wanted but can be difficult or seemingly impossible to achieve due to highly complex, irregular shapes, perimeters and surfaces involved. The inventive technology is especially useful for ongoing wound measurement and comparative analysis and characterization of a wound over time, as well as working with anatomical reconstruction. The inventive technology also extends to non-medical augmented reality applications, and provides robust data sets representing a range of real-world objects, such as zoo animals, family pets, etc. susceptible of being imaged and stored as robust data sets that provide better verisimilitude when used in gaming or other virtual-world contexts as compared to a raw data set from a camera.

Claims

1. An augmented or virtual reality method of wound imaging and reconstruction, comprising:

operating an imaging device to acquire a 3D image of a wound that is real;

producing a depth image from the 3D image;

detecting a wound from the depth image, including

producing a preliminary wound boundary formed of pixels,

producing a final wound boundary from the preliminary wound boundary, including, for each pixel in the preliminary wound boundary,

searching for a maximum value of a directional second derivative of the depth image along a direction orthogonal to the preliminary wound boundary,

setting a pixel of the final wound boundary to coordinates corresponding with the maximum value, subject to a size control function to avoid breaking continuity of the final wound boundary; and

using the final wound boundary, performing at least one augmented reality processing step, virtual reality processing step, authentic reality processing step, or mixed reality processing step, wherein the step is performed by a computer or a processor.

2. The method of claim 1, wherein the operating step comprises imaging a human patient.

3. The method of claim 1, wherein the operating step comprises imaging a real animal.

4. The method of claim 3, wherein the real animal is selected from the group consisting of:

a farm animal;

a household pet;

a zoo animal.

5. The method of claim 1, wherein the operating step is performed to image a patient in a first geographic location, and wherein the 3D image is simultaneously accessible to both a first medical professional in a second geographic location and a second medical professional in a third geographic location, wherein the first geographic location, second geographic location, and third geographic location are remote from each other.

6. The method of claim 1, comprising a step of solving a Laplace equation with Dirichlet boundary conditions.

7. The method of claim 1, wherein the method steps exclude any RGB data processing having been performed and without any other color-information data processing having been performed.

8. The method of claim 1, further comprising constructing a virtual skin surface using the acquired 3D image using capping or an interpolation method on a 2-dimensional grid.

9. The method of claim 1, further comprising calculating a measurement from the final wound boundary.

10. The method of claim 1, further comprising:

generating a Z-buffer, wherein the depth image is produced by a conversion of the Z-buffer;

defining a region of interest U for the wound detection step.