METHOD AND APPARATUS FOR GENERATING PANORAMIC IMAGE WITH TEXTURE MAPPING

• US 20180018807A1
• Filed: 07/15/2016
• Published: 01/18/2018
• Est. Priority Date: 07/15/2016
• Status: Abandoned Application

• Current Assignee: Aspen Technology (Telefonaktiebolaget LM Ericsson)
• Original Assignee: Aspen Technology (Telefonaktiebolaget LM Ericsson)
• Inventors: LU, Chung-Yen, HUNG, Pei-Hen, HUANG, HUNG-JU, LIN, HUNG-MING
• Application Number:
  US15/211,732
• Publication Number:
  US 20180018807A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An image processing apparatus is disclosed, which comprises a rasterization engine, a texture mapping module and a destination buffer. The rasterization engine receives a group of vertices from a vertex list and performs polygon rasterization operations for a point within the group of vertices forming a polygon to generate texture coordinates for each camera image. The vertex list comprises a plurality of vertices with their data structures. The texture mapping module texture maps texture data from each camera image according to its texture coordinates to generate a sample value for each camera image. The destination buffer is coupled to the texture mapping module and stores the panoramic image. Here, the data structures define a mapping between the panoramic image and the camera images.

Claims

1. An image processing apparatus for receiving a plurality of camera images and generating a panoramic image, comprising:

a rasterization engine for receiving a group of vertices from a vertex list and performing polygon rasterization operations for a point within the group of vertices forming a polygon to generate texture coordinates for each camera image, wherein the vertex list comprises a plurality of vertices with their data structures;

a texture mapping module for generating a sample value for each camera image corresponding to the point by texture mapping texture data from each camera image according to its texture coordinates; and

a destination buffer coupled to the texture mapping module for storing the panoramic image;

wherein the data structures define a vertex mapping between the panoramic image and the camera images.

2. The apparatus according to claim 1, wherein the panoramic image is a 360 degree panoramic image, and wherein the rasterization engine receives four vertices from the vertex list and performs polygon rasterization operations for the point within the four vertices forming a quadrilateral to generate the texture coordinates for each camera image.

3. The apparatus according to claim 1, wherein the panoramic image is a default equirectangular panoramic image and the data structures further define a vertex mapping between the default equirectangular panoramic image and the camera images, and wherein the default equirectangular panoramic image is a fully-filled rectangular image without any blank regions.

4. The apparatus according to claim 2, wherein the polygon rasterization operations are quadrilateral rasterization operations and the quadrilateral is located in the top row or the bottom row of a polygon mesh modeling the 360 degree panoramic image.

5. The apparatus according to claim 2, wherein in quadrilateral mode, the rasterization engine performs quadrilateral rasterization operations for the point within the quadrilateral to generate the texture coordinates for each camera image.

6. The apparatus according to claim 2, wherein the rasterization engine further determines whether to divide the quadrilateral into two triangles according to either the data structures or destination coordinates of the four vertices, and wherein each data structure indicates whether its vertex is a pole point.

7. The apparatus according to claim 6, wherein when any of the four vertices is a pole point, the rasterization engine performs quadrilateral rasterization operations for the point within the quadrilateral, otherwise performs triangle rasterization operations for the point within any of the two triangles.

8. The apparatus according to claim 1, wherein the texture mapping module comprises P texture mapping engines that operate in parallel, wherein each texture mapping engine texture maps texture data from one camera image according to its texture coordinates to generate the sample value for the camera image, and wherein the rasterization engine supplies the texture coordinates for the camera images to the P texture mapping engines in parallel.

9. The apparatus according to claim 8, further comprising:

a blending unit coupled between the texture mapping module and the destination buffer for blending the sample value for each camera image at a time to store a blended value in the destination buffer.

10. The apparatus according to claim 1, wherein the texture mapping module sequentially texture maps texture data from each camera image according to its sequentially-received texture coordinates to generate the sample value for the camera image.

11. The apparatus according to claim 10, further comprising:

a blending unit coupled between the texture mapping module and the destination buffer for sequentially blending the sample value for each camera image to generate the blended value.

12. The apparatus according to claim 1, further comprising:

a blending unit coupled between the texture mapping module and the destination buffer for blends the sample value for each camera image according to a corresponding face blending weight to generate the blended value;

wherein the rasterization engine further generates the corresponding face blending weight for each camera image according to the data structures of the four vertices and destination coordinates of the point.

13. The apparatus according to claim 1, further comprising:

an up-scaling unit for generating a reconstructed equirectangular panoramic image by sequentially re-sampling a modified equirectanglar panoramic image from the destination buffer on a line-by-line base;

wherein the data structures further define a vertex mapping between the modified equirectangular panoramic image and the camera images,

wherein the modified equirectangular panoramic image forms a shape of a polygon or a closed curve with at least one blank region, and wherein the reconstructed equirectangular panoramic image is a fully-filled rectangular image without any blank regions.

14. The apparatus according to claim 1, further comprising:

a blending unit coupled between the texture mapping module and the destination buffer for sequentially blending the sample value for a corresponding camera image to generate the blended value;

wherein the vertex list is divided into a plurality of surface vertex lists and a number of surface vertex lists is equal to a number of camera images, wherein the rasterization engine receives one surface vertex list at a time, and wherein the texture mapping module sequentially texture maps texture data from the corresponding camera image according to its sequentially-received texture coordinates to generate the sample value for the corresponding camera image.

15. An image processing method applied in an image processing apparatus, the method comprising:

receiving a group of vertices from a vertex list;

performing polygon rasterization operations for a point within the group of vertices forming a polygon to obtain texture coordinates for each of a plurality of camera images, wherein the vertex list comprises a plurality of vertices with their data structures;

texture mapping texture data from each camera image according to its texture coordinates to obtain a sample value for each camera image corresponding to the point; and

repeating the above steps until all the points within the polygon are processed;

wherein the data structures define a vertex mapping between a panoramic image and the camera images.

16. The method according to claim 15, further comprising:

storing the sample value for each camera image corresponding to the point in a destination buffer after the step of texture mapping and before the step of repeating;

repeating the above steps until all the points within all the polygons from the vertex list are processed; and

outputting a contain of the destination buffer as the panoramic image.

17. The method according to claim 16, further comprising:

repeating the above steps until all surface vertex lists are processed before the step of outputting;

wherein the step of receiving further comprises;

receiving the group of vertices from one surface vertex list of the vertex list, wherein the vertex list is divided into a plurality of surface vertex lists and a number of surface vertex lists is equal to a number of camera images.

18. The method according to claim 15, wherein the step of performing the polygon rasterization operations further comprises:

receiving four vertices from the vertex list; and

performing the polygon rasterization operations for the point within the four vertices forming a quadrilateral to generate the texture coordinates for each camera image;

wherein the panoramic image is a 360 degree panoramic image.

19. The method according to claim 18, wherein the polygon rasterization operations are quadrilateral rasterization operations and the quadrilateral is located in the top row or the bottom row of a polygon mesh modeling the panoramic image.

20. The method according to claim 18, wherein the step of performing the polygon rasterization operations further comprises:

performing quadrilateral rasterization operations for the point within the quadrilateral to generate the texture coordinates for each camera image.

21. The method according to claim 18, wherein the step of performing the polygon rasterization operations further comprises:

determining whether to divide the quadrilateral into two triangles according to either the data structures or destination coordinates of the four vertices;

wherein each data structure indicates whether its vertex is a pole point.

22. The method according to claim 21, wherein the step of performing the polygon rasterization operations further comprises:

when any of the four vertices is a pole point, performing quadrilateral rasterization operations for the point within the quadrilateral, otherwise performing triangle rasterization operations for the point within any of the two triangles.

23. The method according to claim 15, wherein the step of texture mapping further comprises:

receiving the texture coordinates for the camera images in parallel;

texture mapping the texture data from the camera images according to their texture coordinates to generate the sample value for each camera image in parallel.

24. The method according to claim 23, further comprising:

blending the sample value for each camera image at a time to generate a blended value; and

storing the blended value in a destination buffer.

25. The method according to claim 15, wherein the step of texture mapping further comprises:

sequentially texture mapping texture data from each camera image according to its sequentially-received texture coordinates for each camera image to generate the sample value for each camera image.

26. The method according to claim 25, further comprising:

sequentially blends the sample value for each camera image to generate a blended value; and

storing the blended value in a destination buffer.

27. The method according to claim 15, further comprising:

obtaining a face blending weight for each camera image according to the data structures of the four vertices and destination coordinates of the point;

blending the sample value for each camera image according to its face blending weight to generate a blended value; and

storing the blended value in a destination buffer.

28. The method according to claim 16, further comprising:

sequentially re-sampling a modified equirectanglar panoramic image from the destination buffer on a line-by-line base to obtain a reconstructed equirectangular panoramic image;

wherein the data structures further define a vertex mapping between the modified equirectangular panoramic image and the camera images,

wherein the modified equirectangular panoramic image forms a shape of a polygon or a closed curve with at least one blank region, and wherein the reconstructed equirectangular panoramic image is a fully-filled rectangular image without any blank regions.

29. The method according to claim 15, wherein the panoramic image is a default equirectangular panoramic image and the data structures further define a vertex mapping between the default equirectangular panoramic image and the camera images, and wherein the default equirectangular panoramic image is a fully-filled rectangular image without any blank regions.