Apparatus for automated panning and digital zooming in robotic surgical systems

• US 9,699,445 B2
• Filed: 02/29/2012
• Issued: 07/04/2017
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Hasser, Christopher J., Nowlin, William C., Hoffman, Brian D.
• Primary Examiner(s): Koval, Melissa
• Assistant Examiner(s): Nguyen, Trung
• Application Number:
  US13/408,236
• Publication Number:
  US 20120154564A1
• Time in Patent Office: 1,952 Days
• Field of Search: 324309, 324310, 324317, 324300, 324522, 324 72, 324133, 324457, 382276, 382293-299
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

In one embodiment of the invention, a digital zoom and panning system for digital video includes an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

Claims

1. A digital zoom and panning system for digital video, the system comprising:

an image acquisition device to capture digital video images;

an image buffer coupled to the image acquisition device, the image buffer to store one or more frames of digital video images as source pixels;

a first display device having first pixels to display images;

a first digital mapping and filtering device coupled to the image buffer, the first digital mapping and filtering device to selectively map and filter source pixels in a first source rectangle from the image buffer into target pixels in a first destination rectangle of the first display device; and

a first user interface displayed on the first display device to accept a first user input, including the first source rectangle to select first source pixels within frames of the digital video images and the first destination rectangle to select target pixels within the first display device to display images.

2. The digital zoom and panning system of claim 1, wherein:

the first digital mapping and filtering device to automatically select a first region of interest within the digital video images to display in the first destination rectangle; and

the first digital mapping and filtering device selectively maps and filters the source pixels in the first region of interest into target pixels of the first display device.

3. The digital zoom and panning system of claim 2, wherein

the automatic selection of the first region of interest is in response to a tool centroid, a depth of focus, a user focal-point, an average working envelope, or a combination thereof.

4. The digital zoom and panning system of claim 1, further comprising:

a first user interface to accept a first user input including the first source rectangle to select first source pixels within frames of the digital video images, the first destination rectangle to select target pixels within the first display device to display images, and the first region of interest within the digital video images to display in the first destination rectangle; and

wherein the first digital mapping and filtering device selectively maps and filters the source pixels in the first region of interest into target pixels of the first display device in response to the first user input.

5. The digital zoom and panning system of claim 1, further comprising:

a first display buffer coupled to and between the first display device and the digital mapping and filtering device, the first display buffer to store one or more frames of target pixels received from the digital mapping and filtering device and write one or more frames of target pixels to the first display device to display images.

6. The digital zoom and panning system of claim 1, further comprising:

a second display device having target pixels to display images; and

a second digital mapping and filtering device coupled to the image buffer and the user interface, the second digital mapping and filtering device to selectively map and filter source pixels in a second region of interest from the image buffer to digitally zoom into target pixels in a second destination rectangle of the second display device.

7. The digital zoom and panning system of claim 6, further comprising:

a second user interface to accept second user input including the second source rectangle to select second source pixels within frames of the digital video images, the second destination rectangle to select target pixels within the second display device to display images, and the second region of interest within the digital video images to display in the second destination rectangle.

8. The digital zoom and panning system of claim 1, further comprising:

a gaze detection system to detect the first region of interest within the digital video images to digitally pan and display into the first destination rectangle in response to a gaze detected from a user'"'"'s left and right eyes.

9. The digital zoom and panning system of claim 1, further comprising:

a tool tracking system to detect the first region of interest within the digital video images to digitally pan and display in the first destination rectangle in response to one or more tracked positions of one or more tools by the tool tracking system.

10. The digital zoom and panning system of claim 1, wherein

the image acquisition device is a robotic endoscopic camera to capture digital video images of a surgical site and to be mechanically panned within the surgical site;

and

in response to the detection of the first region of interest being outside the first source rectangle, the robotic endoscopic camera to mechanically pan within the surgical site and the first digital mapping and filtering device to digitally pan and display images in the first region of interest into the first destination rectangle.

11. A digital zoom and panning method for digital video, the method comprising:

capturing digital video images;

storing one or more frames of digital video images as source pixels;

displaying images including a first plurality of pixels on a first display device;

selectively mapping and filtering source pixels in a first source rectangle from the image buffer into target pixels in a first destination rectangle of the first display device; and

accepting a first user input through a first user interface displayed on the first display device, the first user input including a first source rectangle to select first source pixels within frames of the digital video images and the first destination rectangle to select target pixels within the first display device to display images.

12. The digital zoom and panning system method of claim 11, further comprising:

automatically selecting a first region of interest within the digital video images to display in the first destination rectangle; and

selectively mapping and filtering the source pixels in the first region of interest into target pixels of the first display device.

13. The digital zoom and panning method of claim 12, wherein

the automatic selecting of the first region of interest is in response to a tool centroid, a depth of focus, a user focal-point, an average working envelope, or a combination thereof.

14. The digital zoom and panning method of claim 11, wherein the first user input includes a first region of interest within the digital video images to display in the first destination rectangle;

and

the method further comprises

selectively mapping and filtering the source pixels in the first region of interest into target pixels of the first display device in response to the first user input.

15. The digital zoom and panning method of claim 11, further comprising:

storing one or more frames of target pixels to a first display buffer coupled to and between the first display device and a digital mapping filtering device; and

writing one or more frames of target pixels to the first display device.

16. The digital zoom and panning system method of claim 11, further comprising:

displaying images on a second display device; and

selectively mapping and filter source pixels in a second region of interest from the image buffer to digitally zoom into target pixels in a second destination rectangle of the second display device.

17. The digital zoom and panning method of claim 16, further comprising:

accepting second user input including the second source rectangle to select second source pixels within frames of the digital video images, the second destination rectangle to select target pixels within the second display device to display images, and the second region of interest within the digital video images to display in the second destination rectangle.

18. The digital zoom and panning method of claim 11, further comprising:

detecting a gaze from a user'"'"'s left and right eyes in the first region of interest within the digital video images; and

digitally panning and zooming into the first destination rectangle in response to the detected gaze.

19. The digital zoom and panning method of claim 11, further comprising:

tool tracking one or more tools; and

detecting the first region of interest within the digital video images to digitally pan and display in the first destination rectangle in response to one or more tracked positions of one or more tools.

20. The digital zoom and panning method of claim 11, wherein

the capturing of digital video images is carried out by a robotic endoscopic camera to capture digital video images of a surgical site and to be mechanically panned within the surgical site;

and

in response to the detection of the first region of interest being outside the first source rectangle, the robotic endoscopic camera to mechanically pan within the surgical site and the selective mapping and filtering to digitally pan and display images in the first region of interest into the first destination rectangle.