IMAGE PROCESSING APPARATUS AND METHOD

• US 20140118341A1
• Filed: 09/10/2013
• Published: 05/01/2014
• Est. Priority Date: 10/25/2012
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: SHIMIZU, Seiya
• Granted Patent: US 9,478,061 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An image processing apparatus synthesize a synthesized image by taking pixel values of pixels in a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image based on the first correspondence relationship, and taking pixel values of each pixel in the synthesized image corresponding to pixels identified in the first image as being pixels representing a solid object, as pixel values of corresponding pixels of the second image based on the second correspondence relationship.

Claims

1. An image processing apparatus, comprising:

a processor; and

a memory storing instructions, which, when executed by the processor, perform a procedure, the procedure including;

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(d) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time and positions of each pixel of the first image, and, based on the movement amount calculated by (b), creating, by virtual viewpoint projection, a second correspondence relationship indicating correspondence between each point on the three dimensional projection plane and positions of each pixel in a second image captured at the second time by expressing the position of the moving body at the second time in the moving body coordinate system at the first time; and

(e) synthesizing a synthesized image by

taking pixel values of pixels in a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image based on the first correspondence relationship, and

taking pixel values of each pixel in the synthesized image corresponding to pixels identified by (c) in the first image as being pixels representing a solid object, as pixel values of corresponding pixels of the second image based on the second correspondence relationship.

2. The image processing apparatus of claim 1, further comprising:

(i) managing the second image based on the movement amount that the moving body has moved during the interval between the first time and the second time as calculated by (b) such that a difference between a position of the moving body at the first time and a position of the moving body at the second time is within a predetermined specific range.

3. The image processing apparatus of claim 1, wherein (c) results in generating a solid object mask image that identifies pixels in the first image that represent a solid object.

4. The image processing apparatus of claim 1, wherein (d) or (g) includes projecting a captured image onto a virtual viewpoint projection plane that has a shape configured by a combination of a hemisphere of infinite radius with the position of the moving body as its origin, and a horizontal plane in which the moving body is present, and employing a virtual viewpoint projection that re-projects points projected onto the virtual viewpoint projection plane onto the three dimensional projection plane with reference to a virtual viewpoint position.

5. The image processing apparatus of claim 1, wherein (e) or (h) includes synthesizing, onto the synthesized image, a solid object based on the 3D position data acquired by (a) and a pre-recorded shape of the moving body.

6. An image processing apparatus, comprising:

a processor; and

a memory storing instructions, which, when executed by the processor, perform a procedure, the procedure including;

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(f) expressing a position of the moving body at the second time based on the movement amount calculated by (b) in the moving body coordinate system at the first time, and infilling pixel values of pixels in the first image identified by (c) as being pixels representing a solid object with corresponding pixel values of pixels of a second image captured at the second time;

(g) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time, and positions of each pixel of the first image; and

(h) synthesizing a synthesized image by taking pixel values of each pixel on a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image after infilling in (f) based on the first correspondence relationship.

7. An image processing method to be executed by a processor, comprising:

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(d) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time and positions of each pixel of the first image, and, based on the movement amount calculated by (b), creating, by virtual viewpoint projection, a second correspondence relationship indicating correspondence between each point on the three dimensional projection plane and positions of each pixel in a second image captured at the second time by expressing the position of the moving body at the second time in the moving body coordinate system at the first time; and

(e) synthesizing a synthesized image by

taking pixel values of pixels on a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image based on the first correspondence relationship, and

taking pixel values of each pixel on the synthesized image corresponding to pixels identified by (c) in the first image as being pixels representing a solid object, as pixel values of corresponding pixels of the second image based on the second correspondence relationship.

8. The image processing method to be executed by a processor of claim 7, further comprising:

(i) managing the second image based on the movement amount that the moving body has moved in the interval between the first time and the second time, as calculated by (b), such that a difference between a position of the moving body at the first time and a position of the moving body at the second time is within a predetermined specific range.

9. The image processing method to be executed by a processor of claim 7, wherein (c) results in generating a solid object mask image that identifies pixels in the first image that represent a solid object.

10. The image processing method to be executed by a processor of claim 7, wherein (d) or (g) includes projecting a captured image onto a virtual viewpoint projection plane that has a shape configured by a combination of a hemisphere of infinite radius with the position of the moving body as its origin, and a horizontal plane in which the moving body is present, and employing a virtual viewpoint projection that re-projects points projected onto the virtual viewpoint projection plane onto the three dimensional projection plane with reference to a virtual viewpoint position.

11. The image processing method to be executed by a processor of claim 7, wherein (e) or (h) includes synthesizing, onto the synthesized image, a solid object based on the 3D position data acquired by (a) and a pre-recorded shape of the moving body.

12. An image processing method to be executed by a processor, comprising:

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(f) expressing a position of the moving body at the second time based on the movement amount calculated by (b) in the moving body coordinate system at the first time, and infilling pixel values of pixels in the first image identified by (c) as being pixels representing a solid object with corresponding pixel values of pixels of a second image captured at the second time;

(g) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time and positions of each pixel of the first image; and

(h) synthesizing a synthesized image by taking pixel values of each pixel on a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image after in-filling in (f) based on the first correspondence relationship.

13. A computer-readable recording medium having stored therein a program for causing a computer to execute an image processing process comprising:

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(d) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time and positions of each pixel of the first image, and, based on the movement amount calculated by (b), creating, by virtual viewpoint projection, a second correspondence relationship indicating correspondence between each point on the three dimensional projection plane and positions of each pixel in a second image captured at the second time by expressing the position of the moving body at the second time in the moving body coordinate system at the first time; and

(e) synthesizing a synthesized image by

taking pixel values of each pixel on a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image based on the first correspondence relationship, and

taking pixel values of each pixel on the synthesized image corresponding to pixels identified by (c) in the first image as being pixels representing a solid object, as pixel values of corresponding pixels of the second image based on the second correspondence relationship.

14. The computer-readable recording medium of claim 13, wherein the process further comprises:

(i) managing the second image based on the movement amount that the moving body has moved in the interval between the first time and the second time as calculated by (b) such that a difference between a position of the moving body at the first time and a position of the moving body at the second time is within a predetermined specific range.

15. The computer-readable recording medium of claim 13, wherein (c) results in generating a solid object mask image that identifies pixels in the first image that represent a solid object.

16. The computer-readable recording medium of claim 13, wherein (d) or (g) includes projecting a captured image onto a virtual viewpoint projection plane that has a shape configured by a combination of a hemisphere of infinite radius with the position of the moving body as its origin, and a horizontal plane in which the moving body is present, and employing a virtual viewpoint projection that re-projects points projected onto the virtual viewpoint projection plane onto the three dimensional projection plane with reference to a virtual viewpoint position.

17. The computer-readable recording medium of claim 13, wherein (e) or (h) includes synthesizing, onto the synthesized image, a solid object based on the 3D position data acquired by (a) and a pre-recorded shape of the moving body.

18. A computer-readable recording medium having stored therein a program for causing a computer to execute an image processing process comprising:

(a) acquiring image data representing successively captured images of a moving body'"'"'s surroundings, and acquiring 3D position data representing 3D positions of a plurality of points on a solid object present in the moving body'"'"'s surroundings as expressed in a moving body coordinate system set for the moving body;

(b) calculating a movement amount that the moving body has moved during an interval from a first time to a second time;

(c) identifying pixels representing a solid object in a first image captured at the first time based on the 3D position data acquired by (a);

(f) expressing a position of the moving body at the second time based on the movement amount calculated by (b) in the moving body coordinate system at the first time, and infilling pixel values of pixels in the first image identified by (c) as being pixels representing a solid object with corresponding pixel values of pixels of a second image captured at the second time;

(g) creating, by virtual viewpoint projection, a first correspondence relationship indicating correspondence between each point on a three dimensional projection plane determined so as to contain the moving body therein at the first time and positions of each pixel of the first image; and

(h) synthesizing a synthesized image by taking pixel values of each pixel on a synthesized image corresponding to each point on the three dimensional projection plane as viewed from a specific viewpoint position, as pixel values of corresponding pixels of the first image after infilling in (f) based on the first correspondence relationship.