AUTOMATED PANNING AND DIGITAL ZOOMING FOR ROBOTIC SURGICAL SYSTEMS

• US 20090245600A1
• Filed: 03/28/2008
• Published: 10/01/2009
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Nowlin, William C., Hoffman, Brian David, Hasser, Christopher J.
• Granted Patent: US 8,155,479 B2
• Time in Patent Office: Days
• US Class Current: 382/128
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

In one embodiment of the invention, a digital zoom and panning system for digital video is disclosed including an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

Claims

1. A method comprising:

capturing video images of a site, each image including an array of source pixels having pixel information of a first resolution;

selecting first inner and outer source windows in the video images for a first array of source pixels in the first inner source window and a first set of source pixels between the first inner and outer source windows;

selecting first inner and outer target windows in target pixels of a first display device for a first array of target pixels in the first inner target window and a first set of target pixels between the first inner and outer target windows;

mapping the first array of source pixels into the first array of target pixels;

mapping the first set of source pixels into the first set of target pixels; and

displaying on the first display device the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows around the first array of target pixels.

2. The method of claim 1, wherein

the mapping of the first array of source pixels into the first array of target pixels is a linear mapping;

and

the mapping of the first set of source pixels into the first set of target pixels is a non-linear mapping.

3. The method of claim 2, wherein

the linear mapping is a linear scaling factor function.

the non-linear mapping is a bilinear interpolation function.

4. The method of claim 3, further comprising:

subdividing the first set of target pixels into a plurality of rectangular arrays of target pixels with four vertices of each being assigned a predetermined mapping from source pixels to target pixels, and

for the first set of target pixels in an interior of each of the plurality of rectangular arrays of target pixels, bilinearly interpolating between the first set of source pixels and the first set of target pixels.

5. The method of claim 2, further comprising:

adjusting the selection of the first inner source window to digitally zoom a first portion of the video images mapped into the first inner target window.

6. The method of claim 5, wherein

the size of the first inner source window is decreased to magnify the first portion of the video images mapped in the first inner target window.

7. The method of claim 5, wherein

the size of the first inner source window is increased to demagnify the first portion of the video images mapped in the first inner target window.

8. The method of claim 2, further comprising:

adjusting a position of the first inner source window with respect to the first outer source window to digitally pan over source pixels in the video images.

9. The method of claim 7, wherein

if the first inner source window becomes near the outer source window, the method further includes

adjusting a position of the first outer source window to center the first inner source window within the first outer source window.

10. The method of claim 2, wherein

the first inner and outer source windows are concentric, and

the method further includes

adjusting positions of the first inner and outer source windows together to digitally pan over source pixels in the video images mapped into the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows.

11. The method of claim 2, wherein

the first inner and outer target windows are concentric, and

the method further includes

adjusting positions of the first inner and outer target windows together to reposition the first inner and outer target windows in the first display.

12. The method of claim 2, further comprising:

adjusting a position of the first inner target window to digitally pan over target pixels in the first display device and reposition the first inner target window within the first outer target window.

13. The method of claim 1, wherein

the mapping of the first array of source pixels into the first array of target pixels is a first linear mapping;

and

the mapping of the first set of source pixels into the first set of target pixels is a second linear mapping.

14. The method of claim 13, wherein

the first linear mapping is a linear scaling factor function, and

the second linear mapping is a piece-wise linear mapping.

15. The method of claim 1, further comprising:

selecting second inner and outer source windows in the video images for a second array of source pixels in the second inner source window and a second set of source pixels between the second inner and outer source windows;

selecting second inner and outer target windows in target pixels of a second display device for a second array of target pixels in the second inner target window and a second set of target pixels between the second inner and outer target windows;

mapping the second array of source pixels into the second array of target pixels;

mapping the second set of source pixels into the second set of target pixels; and

displaying on the second display device the second array of target pixels in the second inner target window and the second set of target pixels between the second inner and outer target windows around the second array of target pixels.

16. The method of claim 1, further comprising:

displaying on a second display device the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows around the first array of target pixels.

17. The method of claim 1, further comprising:

prior to displaying the first array of target pixels and the first set of target pixels, digitally filtering the first array of target pixels and the first set of target pixels.

18. The method of claim 1, wherein

the site is a minimally invasive surgical site and the video images are captured by a robotic endoscopic camera.

19. The method of claim 9, further comprising

detecting motion of a user to automatically adjust the positions of the first inner and outer source windows.

20. The method of claim 19, wherein

the motion of the user to detect is one of a finger by a touch sensitive panel, a pair of hands by a motion sensing display device, a pair of eyes by a gaze detector, and a foot by a pedal.

21. A digital zoom and panning system for digital video, the system comprising:

an image acquisition device to capture digital video images;

an image buffer coupled to the image acquisition device, the image buffer to store one or more frames of digital video images as source pixels;

a first display device having first pixels to display images; and

a first digital mapping and filtering device coupled to the image buffer, the first digital mapping and filtering device to selectively map and filter source pixels in a first region of interest from the image buffer into target pixels in a first destination rectangle of the first display device.

22. (canceled)

23. (canceled)

24. (canceled)

25. (canceled)

26. (canceled)

27. (canceled)

28. (canceled)

29. (canceled)

30. (canceled)

31. A method for a digital video system, the method comprising:

selecting an outer target window on a display device within which to display a fovea image and a surround image around the fovea image;

selecting an inner target window on the display device within the outer target window to display the fovea image;

linearly mapping source pixels of a digital video into target pixels in the inner target window to display the fovea image and non-linearly mapping source pixels of the digital video into target pixels between the inner target window and the outer target window to display the surround image around the fovea image.

32. The method of claim 31, wherein

the display device includes a touch sensitive display panel;

and

the outer target window and the inner target window are selected by a touching a finger on the touch sensitive display panel.

33. The method of claim 32, further comprising:

touching the touch sensitive display panel with the finger to pan over the source pixels of the digital video for mapping into the inner target window of the display.

34. The method of claim 31, further comprising:

touching one or more buttons to pan over the source pixels of the digital video for mapping into the inner target window of the display.

35. The method of claim 31, further comprising:

adjusting the position of the display device to pan over the source pixels of the digital video for mapping into the inner target window of the display.

36. The method of claim 35, further comprising:

sensing the motion of the display device with an inertia sensor to digitally pan the source pixels of the digital video.

37. The method of claim 35, further comprising:

sensing the motion of the display device with one or more encoders at one or more joints of set-up arms supporting the display device.