Image processing system and method with high resolution view-point conversion images

• US 8,134,595 B2
• Filed: 05/30/2007
• Issued: 03/13/2012
• Est. Priority Date: 06/02/2006
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Takano, Teruhisa
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US11/755,368
• Publication Number:
  US 20070285549A1
• Time in Patent Office: 1,749 Days
• Field of Search: 348 36- 39, 348 47- 48, 348/153, 348/159, 348/218.1, 348/333.05, 348/148, 348/147, 382/154, 382/284, 382/294
• US Class Current: 348/153
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An image processing system so disclosed has respective image pickup parts and respective viewpoint conversion parts configured to perform viewpoint conversion of original images captured by the respective image pickup parts to generate converted images. The image processing system also includes a display device controllable to display the converted images.

Claims

1. An image processing system, comprising:

plural image pickup devices, each including an image pickup part configured to capture an original image and each including an onboard processor, each of the onboard processors including a viewpoint conversion part configured to perform viewpoint conversion for the original image to generate a converted image, wherein the converted image has a virtual viewpoint position different from an intrinsic viewpoint position of the image pickup part as shown in the original image;

an image processor coupled to each of the onboard processors and including an image synthesis part configured to synthesize converted images output from the onboard processors of the plural image pickup devices to generate a synthesized image; and

a display device controllable by the image processor to selectively display the synthesized image and at least one original image output from the plural image pickup devices.

2. The system according to claim 1 wherein the plural image pickup devices are configured to output less than an entirety of the converted images generated by respective viewpoint conversion parts to the image processor.

3. The system according to claim 1 wherein each of the onboard processors further comprises:

a rotating part configured to rotate the converted image generated by the viewpoint conversion part to match an orientation in the synthesized image.

4. The system according to claim 1 wherein each of the onboard processors is configured to alternately output the original image and a converted image to the image processor.

5. The system according to claim 4 wherein each of the plural image pickup devices further comprises:

a storage part with at least a 4-image storage capacity; and

wherein a ratio of write speed to read speed relative to the storage part is 1;

2.

6. The system according to claim 1 wherein each of the onboard processors further comprises:

a second viewpoint conversion part, different from the first viewpoint conversion part and configured to perform viewpoint conversion of the original image to generate a second converted image, wherein the second converted image has a virtual viewpoint position different from the intrinsic viewpoint position of the image pickup part as shown in the original image; and

wherein each of the onboard processors is configured to sequentially output the original image, the converted image and the second converted image to the image processor.

7. The system according to claim 6 wherein each of the image pickup devices further comprises:

a storage part with a 6-image storage capacity; and

wherein a ratio of write speed to read speed relative to the storage part is 1;

3.

8. An image processing system, comprising:

plural image pickup devices, each including an image pickup part configured to capture an original image and a viewpoint conversion part configured to perform viewpoint conversion for the original image to generate a converted image, wherein the converted image has a virtual viewpoint position different from an intrinsic viewpoint position of the image pickup part as shown in the original image;

an image processor operable to synthesize converted images output from the plural image pickup devices to generate a synthesized image; and

a display device controllable to selectively display the synthesized image and at least one original image output from the plural image pickup devices;

wherein each of the image pickup devices further comprises;

an image synthesis part configured to synthesize the original image and the converted image to generate a second synthesized image.

9. An image processing system, comprising:

image pickup devices having respective image pickup parts configured to capture original images having intrinsic viewpoint positions based on respective positions of the image pickup devices and having respective onboard processors, each onboard processor having a respective viewpoint conversion part configured to perform viewpoint conversion of the original images to generate converted images having virtual viewpoint positions different from the intrinsic viewpoint positions of the original images;

an image processor configured to receive the original images and the converted images from each of the onboard processors; and

a display device controllable by the image processor to display the original images and the converted images.

10. An image processing system, comprising:

plural image pickup devices, each including means for capturing an original image and each including onboard processing means for processing the original image, each onboard processing means including viewpoint conversion means for performing viewpoint conversion for the original image to generate a converted image, wherein the converted image has a virtual viewpoint position different from an intrinsic viewpoint position of the capture means as shown in the original image;

image processing means coupled to each onboard processing means for synthesizing converted images output from each onboard processing means of the plural image pickup devices to generate a synthesized image; and

display means for selectively displaying the synthesized image and at least one original image output from the plural image pickup devices responsive to the image processing means.

11. The system according to claim 10 wherein each of the onboard processing means further comprises:

means for rotating the converted image generated by the viewpoint conversion means to match an orientation in the synthesized image.

12. The system according to claim 10 wherein each of the onboard processing means further comprises:

second viewpoint conversion means, different from the first viewpoint conversion means, for performing viewpoint conversion of the original image to generate a second converted image, wherein the second converted image has a virtual viewpoint position different from the intrinsic viewpoint position of the capture means as shown in the original image; and

means for sequentially outputting respective original images, converted images and second converted images to the image processing means.

13. The system according to claim 12 wherein the sequentially outputting means comprises:

means for storing six images; and

wherein a ratio of write speed to read speed relative to the storing means is 1;

3.

14. An image processing method, comprising:

capturing original images using respective image pickup devices, each image pickup device including a respective onboard processor;

subjecting the original images to viewpoint conversion to form respective converted images in the respective image pickup devices using the respective onboard processors;

outputting the original images and the respective converted images from the respective onboard processors of the respective image pickup devices to an image processor;

synthesizing the respective converted images output from the respective image pickup devices to form a synthesized image using the image processor; and

selectively displaying the synthesized image and at least one of the original images.

15. The method according to claim 14, further comprising:

rotating at least one of the respective converted images using at least one of the respective image pickup devices to match an orientation in the synthesized image.

16. The method according to claim 14 wherein outputting the original images and the respective converted images from the respective onboard processors of the respective image pickup devices to the image processor further comprises:

alternately outputting the original images and the respective converted images.

17. The method according to claim 16, further comprising:

alternately storing the original images and the respective converted images in a storage part with at least a 4-image storage capacity; and

wherein a ratio of write speed to read speed relative to the storage part is 1;

2.

18. The method according to claim 16, further comprising:

performing viewpoint conversion of the original images to generate respective second converted images different from the respective converted images using the respective onboard processors; and

wherein outputting the original images and the respective converted images from the respective onboard processors of the respective image pickup devices to the image processor further comprises;

sequentially outputting the original images, the respective converted images and the respective second converted images to the image processor.

19. An image processing method, comprising:

capturing a first original image using an image pickup part of a first image pickup device, the first image pickup device including an onboard processor having a viewpoint conversion part;

subjecting the first original image to viewpoint conversion using the viewpoint conversion part of the first image pickup device to produce a first visible image having a virtual viewpoint position different from an intrinsic viewpoint position of the first image pickup device;

capturing a second original image using an image pickup part of a second image pickup device, the second image pickup device including an onboard processor having a viewpoint conversion part;

subjecting the second original image to viewpoint conversion using the viewpoint conversion part of the second image pickup device to produce a second visible image having a virtual viewpoint position different from an intrinsic viewpoint position of the second image pickup device;

transmitting the first and second visible images from each of the onboard processors to a common image processor; and

displaying the first and the second visible images on a display device.