METHOD OF RENDERING AND MANIPULATING ANATOMICAL IMAGES ON MOBILE COMPUTING DEVICE

• US 20130135312A1
• Filed: 11/13/2012
• Published: 05/30/2013
• Est. Priority Date: 11/10/2011
• Status: Active Grant

• Current Assignee: 7D Surgical ULC (Orthofix Medical Inc.)
• Original Assignee: Michael Leung, Victor Yang
• Inventors: YANG, Victor, STANDISH, Beau Anthony, RZESZUTEK, Richard Jeffrey, LEUNG, Michael, MARIAMPILLAI, Adrian, ANDROUTSOS, Dimitrios
• Granted Patent: US 8,933,935 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality

Abstract

Methods are provided for rendering an image of an anatomical object on a display of a mobile computing device. Sensors of the mobile computing device are interrogated to determine an orientation, or a change in orientation, of the computing device. Transformations are determined for rotating and positioning the image of the anatomical object such that the image appears to be stationary after a change in the orientation of the mobile computing device. Additional image data associated with a surgical tool or a surgical plan may also be rendered on the device for planning and/or training simulations.

Claims

1. A computer implemented method of dynamically displaying an anatomical object on a mobile computing device, the mobile computing device including sensors for determining an orientation of the mobile computing device and a display for displaying the anatomical object, the method comprising the steps of:

a) obtaining three-dimensional image data associated with the anatomical object;

b) determining a virtual camera vector defining a distance between a virtual camera and the anatomical object upon rendering of the anatomical object;

c) determining a reference vector associated with the anatomical object, such that when rendering the anatomical object on the display, the reference vector is aligned with a gravity vector when the mobile computing device is oriented in a reference orientation;

d) interrogating the sensors of the mobile computing device and determining an orientation of the mobile computing device;

e) determining a transformation such that the reference vector is aligned with the gravity vector, and such that the anatomical object is perceived as residing within the display at a location corresponding to the virtual camera vector;

f) applying the transformation to three-dimensional image data and obtaining transformed three-dimensional image data such that when rendered on the display, the anatomical object image appears to be stationary after a change in the orientation of the mobile computing device; and

g) rendering an image of the anatomical object on the display of the mobile computing device according to the transformed three-dimensional image data.

2. The method according to claim 1 further comprising the step of repeating steps d) to g) one or more times.

3. The method according to claim 1 further wherein the transformation is an affine transformation.

4. The method according to claim 1 wherein the transformation is an affine transformation.

5. The method according to claim 1 wherein the transformation is a net transformation, and wherein the step of determining the net transformation comprises the steps of:

determining an orientational transformation such that the reference vector is aligned with the gravity vector; and

determining a translational transformation such that the anatomical object is perceived as residing within the display at a location corresponding to the virtual camera vector.

6. The method according to claim 1 wherein the reference vector associated with the anatomical object is selected such that the reference vector is aligned with the gravity vector when the display is positioned in a horizontal orientation.

7. The method according to claim 1 wherein the three-dimensional image data is provided as a mesh.

8. The method according to claim 7 wherein the transformation is applied to each point in the mesh.

9. The method according to claim 1 further comprising the steps of:

receiving additional three-dimensional image data associated with a surgical device to be inserted into the anatomical object;

determining an initial position and an initial orientation of the surgical device relative to that of the anatomical object, thereby establishing an initial relative position of the surgical device;

rendering an image of the surgical device on the display in accordance with the initial relative position.

10. The method according to claim 9 wherein the initial position is associated with a pre-selected point of entry for the surgical device to enter the anatomical object.

11. The method according to claim 10 wherein the anatomical object includes a bone structure, and surgical device is a screw.

12. The method according to claim 9 wherein the surgical device is positioned by the steps of:

receiving input defining a translation of the surgical device; and

generating an additional transformation associated with the translation;

employing the additional transformation to determine an updated relative position of the surgical device; and

rendering an updated image of the surgical device on the display in accordance with the updated relative position.

13. The method according to claim 12 wherein the step of receiving the input includes the steps of:

detecting a touchless gesture; and

inferring the translation based on the touchless gesture.

14. The method according to claim 12 further comprising the steps of:

obtaining boundary three-dimensional image data defining a forbidden region for insertion of the surgical device;

determining whether or not the updated relative position of the surgical device is within the forbidden region.

15. The method according to claim 14 wherein in the event that a portion of the surgical device is determined to lie within the forbidden region, notifying the user.

16. The method according to claim 14 further comprising the steps of:

receiving placement information identifying one or more suitable locations for placement of the surgical device;

determining an accuracy of the placement of the surgical device by comparing the updated relative position of the surgical device with the placement information.

17. The method according to claim 16 further comprising providing feedback associated with the accuracy of the placement of the surgical device.

18. The method according to claim 1 further comprising the steps of:

receiving three-dimensional surgical plan data associated with a surgical plan;

rendering an image of the surgical plan on the display;

wherein a relative orientation of the three-dimensional surgical plan data and the three-dimensional image data is preserved when rendering the image of the surgical plan.

19. A computer implemented method of dynamically displaying an anatomical object on a mobile computing device, the mobile computing device including sensors for determining an orientation of the mobile computing device and a display for displaying the anatomical object, the method comprising the steps of:

a) obtaining three-dimensional image data associated with the anatomical object;

b) interrogating the sensors of the mobile computing device and determining a change in an orientation of the mobile computing device;

c) rendering an image of the anatomical object on the display, such that the anatomical object image appears to be stationary after the change in the orientation of the mobile computing device.

20. The method according to claim 19 further comprising the step of repeating steps b) and c) one or more times.

21. A computer-readable storage medium comprising instructions for dynamically displaying an anatomical object on a mobile computing device, the mobile computing device including sensors for determining an orientation of the mobile computing device and a display for displaying the anatomical object, wherein execution of the instructions by one or more processors causes the one or more processors to carry out the steps of:

a) obtaining three-dimensional image data associated with the anatomical object;

b) determining a virtual camera vector defining a distance between a virtual camera and the anatomical object upon rendering of the anatomical object;

c) determining a reference vector associated with the anatomical object, such that when rendering the anatomical object on the display, the reference vector is aligned with a gravity vector when the mobile computing device is oriented in a reference orientation;

d) interrogating the sensors of the mobile computing device and determining an orientation of the mobile computing device;

e) determining a transformation such that the reference vector is aligned with the gravity vector, and such that the anatomical object is perceived as residing within the display at a location corresponding to the virtual camera vector;

f) applying the transformation to three-dimensional image data and obtaining transformed three-dimensional image data such that when rendered on the display, the anatomical object image appears to be stationary after a change in the orientation of the mobile computing device; and

g) rendering an image of the anatomical object on the display of the mobile computing device according to the transformed three-dimensional image data.