Apparatus for automated panning and zooming in robotic surgical systems

• US 10,038,888 B2
• Filed: 06/06/2017
• Issued: 07/31/2018
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Hoffman, Brian D., Hasser, Christopher J., Nowlin, William C.
• Primary Examiner(s): Koval, Melissa
• Assistant Examiner(s): Nguyen, Trung
• Application Number:
  US15/615,830
• Publication Number:
  US 20170339399A1
• Time in Patent Office: 420 Days
• Field of Search: 324210, 324304-309, 324310, 324317, 324300, 324347, 324375, 324522, 324 72, 324133, 324457, 32475023, 32475423, 32476002, 324 7666, 324 7611, 600410-427
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

In one embodiment of the invention, a digital zoom and panning system for digital video is disclosed including an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

Claims

1. A digital zoom and panning system, the system comprising:

an endoscopic camera device to capture digital video images of a surgical site;

an image buffer coupled to the endoscopic camera device, the image buffer to store one or more frames of the digital video images as source pixels;

a first display device having first pixels to display images;

a first user interface displayed on the first display device to accept a first user input to display a fovea, the first user input including selection of a first source pixel array of source pixels within a first frame of the digital video images with reference to the surgical site and selection of a first target pixel array of target pixels within a subset of the first pixels of the first display device; and

a first digital mapping and filtering device coupled between the image buffer and the first display device, and based on the first user input, the first digital mapping and filtering device to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to display the fovea on the first display device.

2. The digital zoom and panning system of claim 1, wherein

the first digital mapping and filtering device digitally pans the fovea within the first frame of the digital video images based on a user input;

and

the first digital mapping and filtering device continues to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

3. The digital zoom and panning system of claim 2, wherein

during the digital panning, the first digital mapping and filtering device brings one or more source pixels outside the first source pixel array within the first source pixel array and moves one or more source pixels within the first source pixel array outside the first source pixel array.

4. The digital zoom and panning system of claim 1, wherein

the first digital mapping and filtering device digitally pans from the first source pixel array to a second source pixel array, one or more source pixels outside the first source pixel array are brought within the second source pixel array and one or more source pixels within the first source pixel array are moved outside the second source pixel array.

5. The digital zoom and panning system of claim 2, wherein

the fovea nears an edge of the first frame of the digital video images;

the endoscopic camera is mechanically panned within the surgical site; and

the first digital mapping and filtering device continues to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

6. The digital zoom and panning system of claim 1, wherein

the endoscopic camera is mechanically panned such that the first source pixel array is outside the first frame;

and

the first digital mapping and filtering device to digitally pan from the first source pixel array to a second source pixel array in a second frame of the digital video images with reference to the surgical site and to selectively map and filter source pixels in the second source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

7. The digital zoom and panning system of claim 6, wherein

when the endoscopic camera is mechanically panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.

8. The digital zoom and panning system of claim 6, wherein

the first digital mapping and filtering device digitally pans from the first source pixel array to a second source pixel array, one or more source pixels outside the first source pixel array are brought within the second source pixel array and one or more source pixels within the first source pixel array are moved outside the second source pixel array.

9. A digital zoom and panning method for digital video, the method comprising:

capturing digital video images of a surgical site with an endoscopic camera device;

storing one or more frames of the digital video images as source pixels;

displaying images on a first display device, the first display device including a first plurality of pixels to display the images;

accepting a first user input through a first user interface displayed on the first display device, the first user input to selectively display a fovea on the first display device, the first input including a selection of a first source pixel array of source pixels within frames of the digital video images and a selection of a first target pixel array of target pixels within the first display device to display images;

selectively mapping and filtering source pixels in the first source rectangle from the image buffer into the target pixels in the first target pixel array within the first display device.

10. The method of claim 9, further comprising:

accepting a second user input through a second user interface displayed on the first display device; and

digitally panning the fovea displayed on the first display device based on the second user input.

11. The method of claim 10, wherein

when the fovea is digitally panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.

12. The method of claim 9, further comprising:

mechanically panning the endoscopic camera with the first source pixel array within the first frame; and

continuing to selectively map and filter source pixels in the first source pixel array from the image buffer into target pixels in the first target pixel array to continue to display the fovea on the first display device.

13. The method of claim 12, wherein

when the endoscopic camera is mechanically panned, one or more source pixels outside the first source pixel array are brought within the first source pixel array and one or more source pixels within the first source pixel array are moved outside the first source pixel array.