Automated panning in robotic surgical systems based on tool tracking

• US 10,432,921 B2
• Filed: 07/12/2018
• Issued: 10/01/2019
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Hoffman, Brian D., Hasser, Christopher J., Nowlin, William C.
• Primary Examiner(s): Koval, Melissa J
• Assistant Examiner(s): Nguyen, Trung
• Application Number:
  US16/034,004
• Publication Number:
  US 20180324414A1
• Time in Patent Office: 446 Days
• Field of Search: 324210, 324304-309, 324310, 324317, 324300, 324347, 324375, 324522, 324 72, 324133, 324457, 32475023, 3247542, 324 3, 32476002, 324 7666, 324 7611, 382254-308, 348 71- 74
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

In one embodiment, a digital zoom and panning system for digital video is disclosed including an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

Claims

1. A digital zoom and panning system, the system comprising:

an endoscopic camera device to capture digital video images of a surgical site;

an image buffer coupled to the endoscopic camera device, the image buffer to store one or more frames of the digital video images as source pixels;

a first display device having first pixels to display images;

a first user interface displayed on the first display device to accept a first user input to display a fovea, the first user input including selection of a first source pixel array of source pixels within a first frame of the digital video images with reference to the surgical site and selection of a first target pixel array of target pixels within a subset of the first pixels of the first display device;

a first digital mapping and filtering device coupled between the image buffer and the first display device, the first digital mapping and filtering device to selectively map and filter source pixels in a first region of interest from the image buffer into target pixels in a first destination rectangle for the first display device; and

a tracking system to track a position of at least one object to digitally pan and display the fovea in the first destination rectangle based on one or more tracked positions of the at least one object.

2. The digital zoom and panning system of claim 1, wherein

the tracking system is a gaze detection system to detect a gaze of a user'"'"'s left and right eyes to determine the one or more tracked positions and digitally pan and display the fovea in the first destination rectangle.

3. The digital zoom and panning system of claim 1, wherein

the tracking system is a tool tracking system to detect a position of one or more tools to determine the one or more tracked positions and digitally pan and display the fovea in the first destination rectangle.

4. The digital zoom and panning system of claim 1, wherein

the endoscopic camera is mechanically panned with the first source pixel array within the first frame;

and

the first digital mapping and filtering device continues to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

5. The digital zoom and panning system of claim 4, wherein

when the endoscopic camera is mechanically panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.

6. The digital zoom and panning system of claim 1, wherein

the endoscopic camera is mechanically panned such that the first source pixel array is outside the first frame;

and

the first digital mapping and filtering device to digitally pan from the first source pixel array to a second source pixel array in a second frame of the digital video images with reference to the surgical site and to selectively map and filter source pixels in the second source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

7. The digital zoom and panning system of claim 6, wherein

when the endoscopic camera is mechanically panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.

8. The digital zoom and panning system of claim 6, wherein

when the first digital mapping and filtering device digitally pans from the first source pixel array to the second source pixel array, one or more source pixels outside the first source pixel array are brought within the second source pixel array and one or more source pixels within the first source pixel array are moved outside the second source pixel array.

9. A digital zoom and panning method for digital video, the method comprising:

capturing digital video images of a surgical site with an endoscopic camera device;

storing one or more frames of the digital video images as source pixels;

displaying images on a first display device, the first display device including a first plurality of pixels to display the images;

accepting a first user input through a first user interface displayed on the first display device, the first user input to selectively display a fovea on the first display device, the first input including a selection of a first source pixel array of source pixels within frames of the digital video images and a selection of a first target pixel array of target pixels within the first display device to display images;

selectively mapping and filtering source pixels in the first source rectangle from the image buffer into the target pixels in the first target pixel array within the first display device;

tracking a position of at least one object to determine one or more tracked positions of the at least one object; and

digitally panning and displaying the fovea in the first destination rectangle based on the one or more tracked positions of the at least one object.

10. The method of claim 9, wherein

the at least one object is a user'"'"'s left and right eyes to determine the one or more tracked positions thereof.

11. The method of claim 9, wherein

the at least one object is one or more tools to determine the one or more tracked positions thereof.

12. The method of claim 9, further comprising:

mechanically panning the endoscopic camera with the first source pixel array within the first frame; and

continuing to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

13. The method of claim 12, wherein

when the endoscopic camera is mechanically panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.