Method and apparatus for generating panoramic image with rotation, translation and warping process

• US 10,186,067 B2
• Filed: 10/25/2016
• Issued: 01/22/2019
• Est. Priority Date: 10/25/2016
• Status: Active Grant

• Current Assignee: Aspen Technology (Telefonaktiebolaget LM Ericsson)
• Original Assignee: Aspen Technology (Telefonaktiebolaget LM Ericsson)
• Inventors: Lu, Chung-Yen, Hung, Pei-Hen, Chen, Jing-Chuan
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/333,765
• Publication Number:
  US 20180114348A1
• Time in Patent Office: 819 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A transform apparatus applied in an image processing system with an image capture module is disclosed. The transform apparatus comprises a geometry information detector, a parameter setup unit, a primitive assembly unit and a geometry processing circuit. The geometry information detector measures geometry information of the image capture module. The parameter setup unit calculates geometry parameters and determines whether to assert an enable signal according to the geometry information. The primitive assembly unit receives an original vertex list and the enable signal to generate a vertex/ID flow. The geometry processing circuit receives the geometry parameters and performs geometry transform operations over the vertex/ID flow in response to whether the first enable signal is asserted to generate a modified vertex list. The geometry information comprises at least one of a displacement and rotation angles of the image capture module.

Claims

1. A transform apparatus applied in an image processing system having an image capture module and an image processing apparatus, the image capture module capturing a 360-degree horizontal field of view and 180-degree vertical field of view to generate a plurality of camera images, the apparatus comprising:

a geometry information detector for measuring geometry information of the image capture module to determine whether the image capture module is moved;

a parameter setup unit coupled to the geometry information detector for calculating geometry parameters according to the geometry information;

a primitive assembly unit for receiving an original vertex list to generate a vertex/ID flow; and

a geometry processing circuit for performing geometry transform operations over the vertex/ID flow according to the geometry parameters to generate a modified vertex list if the image capture module is moved;

wherein the geometry processing circuit comprises;

a geometry transform unit for sequentially performing the geometry transform operations over a current group of three first vertices with their data structures from the vertex/ID flow according to the geometry parameters to correct equirectangular coordinates of the three first vertices as three temporary vertices if the image capture module is moved; and

a triangle assembly unit for measuring shortest distances along θ

axis among the three temporary vertices and their imaginary points to determine a transformed triangle with three second vertices as a part of the modified vertex list, wherein a difference between each temporary vertex and its corresponding imaginary point is 2π

radians along θ

axis;

wherein the geometry information comprises at least one of a displacement and rotation angles associated with the movement of the image capture module;

wherein the vertex/ID flow comprises a plurality of groups of three first vertices with their data structures, and each group of three first vertices form a corresponding triangle in a polygon mesh modeling a panoramic image from the image processing apparatus;

wherein the modified vertex list comprises a plurality of groups of second vertices with their data structures, and each group of three second vertices form a corresponding triangle in the polygon mesh;

wherein the original vertex list comprises a plurality of third vertices with their data structures, and the original vertex list is obtained by conducting calibration between the plurality of camera images and the panoramic image for the plurality of third vertices;

wherein each of the data structures defines a corresponding vertex mapping between the plurality of camera images and the panoramic image; and

wherein the modified vertex list is inputted to the image processing apparatus and causes the image processing apparatus to generate a modified panoramic image.

2. The apparatus according to claim 1, wherein the geometry information detector comprises:

a rotation detector for measuring the rotation angles of the image capture module;

wherein the geometry parameters comprise a rotation matrix and an inverse rotation matrix.

3. The apparatus according to claim 1, wherein the geometry information detector comprises:

a displacement detector for measuring the displacement of the image capture module;

wherein the geometry parameters comprise the displacement.

4. The apparatus according to claim 1, wherein the geometry processing circuit further comprises:

a pole detection unit for detecting whether the three temporary vertices surround a pole point to generate an indication signal.

5. The apparatus according to claim 4, wherein the pole detection unit detects whether the three temporary vertices surround the pole point according to relative distances of the three temporary vertices along θ

axis.

6. The apparatus according to claim 4, wherein the vertex/ID flow further comprises IDs for the groups of three first vertices, wherein the pole detection unit detects whether the three temporary vertices surround the pole point by comparing a plurality of pre-defined IDs with an ID associated with the current group of three first vertices.

7. The apparatus according to claim 6, wherein the parameter setup unit further performs inverse geometry transform operations over a north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points, and calculates which polygons with the pre-defined IDs in the polygon mesh respectively contain the two inverse pole points according to a polygon size and a width of the panoramic image to output the geometry parameters further comprising the equirectangular coordinates of the two inverse pole points and the pre-defined IDs.

8. The apparatus according to claim 4, wherein the geometry processing circuit further comprises:

a pole parameter generation unit for performing interpolation operations over the three input vertices according to equirectangular coordinates of a desirous inverse pole point to generate a data structure of the pole point if the pole point is detected, wherein the desirous inverse pole point is a point that becomes the pole point surrounded by the three temporary vertices after the geometry transform operations are performed; and

a triangle tessellation unit for dividing the transformed triangle including the three second vertices and the pole point into multiple regrouped triangles to output regrouped vertices with their data structures for the regrouped triangles as a part of the modified vertex list if the pole point is detected;

wherein the three second vertices with their data structures are directly outputted as a part of the modified vertex list if no pole point is detected.

9. The apparatus according to claim 8, wherein the parameter setup unit further performs inverse geometry transform operations over a the north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points, wherein the pole parameter generation unit further selects one from the two inverse pole points as the desirous inverse pole point according to the indication signal.

10. The apparatus according to claim 9, wherein the pole parameter generation unit is coupled between the pole detection unit and the triangle tessellation unit.

11. The apparatus according to claim 9, wherein the geometry transform unit and the triangle assembly unit are coupled between the pole parameter generation unit and triangle tessellation unit.

12. The apparatus according to claim 8, wherein the geometry processing circuit further comprises:

an inverse geometry transform unit coupled between the pole detection unit and the pole parameter generation unit for performing inverse geometry transform operations over one of a north pole and a south pole to generate the equirectangular coordinates of the desirous inverse pole point according to the indication signal if the pole point is detected.

13. The apparatus according to claim 1, wherein the original vertex list is divided into a plurality of original surface vertex lists and the modified vertex list is divided into a plurality of modified surface vertex lists, and numbers of original surface vertex lists and modified surface vertex lists are equal to a number of camera images.

14. A transform method applied in an image processing system having an image capture module and an image processing apparatus, the image capture module capturing a 360-degree horizontal field of view and 180-degree vertical field of view to generate a plurality of camera images, the method comprising:

when the image capture module is moved, performing transform operations over a vertex/ID flow until all first vertices from the vertex/ID flow are processed to generate a modified vertex list, wherein the modified vertex list is inputted to the image processing apparatus and causes the image processing apparatus to generate a modified panoramic image;

wherein the step of performing the transform operations comprises;

first, sequentially performing geometry transform operations over a current group of three first vertices with their data structures from the vertex/ID flow according to geometry parameters to correct equirectangular coordinates of the three first vertices as three temporary vertices; and

then, measuring shortest distances along θ

axis among the three temporary vertices and their imaginary points to determine a transformed triangle with three second vertices as a part of the modified vertex list, wherein a difference between each temporary vertex and its corresponding imaginary point is 2π

radians along θ

axis; and

repeating the steps of sequentially performing the geometry transform operations and measuring until all the first vertices from the vertex/ID flow are processed;

wherein the geometry parameters comprise at least one of a rotation matrix, an inverse rotation matrix and a displacement associated with the movement of the image capture module; and

wherein the vertex/ID flow comprises a plurality of groups of three first vertices with their data structures, and each group of three first vertices form a corresponding triangle in a polygon mesh modeling a panoramic image from the image processing apparatus;

wherein the modified vertex list comprises a plurality of groups of second vertices with their data structures, and each group of three second vertices form a corresponding triangle in the polygon mesh; and

wherein each of the data structures defines a corresponding vertex mapping between the plurality of camera images and the panoramic image.

15. The method according to claim 14, further comprising:

first, measuring geometry information of the image capture module prior to the step of performing the transform operations;

then, calculating the geometry parameters according to the geometry information; and

then, generating the vertex/ID flow according to an original vertex list from a calibration system and the geometry parameters;

wherein the original vertex list comprises a plurality of second third vertices with their data structures and the original vertex list is obtained by conducting calibration between the plurality of camera images and the panoramic image for the plurality of third vertices.

16. The method according to claim 15, wherein the step of measuring the geometry information comprises:

measuring rotation angles of the image capture module to determine whether the image capture module is moved;

wherein the geometry parameters comprise the rotation matrix and the inverse rotation matrix.

17. The method according to claim 15, wherein the step of measuring the geometry information comprises:

measuring a displacement of the image capture module to determine whether the image capture module is moved;

wherein the geometry parameters comprise the displacement.

18. The method according to claim 15, wherein the original vertex list is divided into a plurality of original surface vertex lists and the modified vertex list is divided into a plurality of modified surface vertex lists, and numbers of original surface vertex lists and modified surface vertex lists are equal to a number of camera images.

19. The method according to claim 14, wherein the step of performing the transform operations further comprises:

detecting whether the three temporary vertices surround a pole point to generate a detecting result; and

outputting the three second vertices with their data structures of the transformed triangle as a part of the modified vertex list if no pole point is detected.

20. The method according to claim 19, wherein the step of detecting comprises:

detecting whether the three temporary vertices surround the pole point according to relative distances of the three temporary vertices along θ

axis.

21. The method according to claim 19, wherein the step of detecting comprises:

comparing a plurality of pre-defined IDs with an ID associated with the current group of three first vertices from the vertex/ID list to detect whether the three temporary vertices surround the pole point.

22. The method according to claim 21, wherein the step of calculating further comprises:

performing inverse geometry transform operations over a north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points; and

calculating which polygons with the pre-defined IDs in the polygon mesh respectively contain the two inverse pole points according to a polygon size and a width of the panoramic image.

23. The method according to claim 19, wherein the step of performing the transform operations further comprises:

if the pole point is detected, determining equirectangular coordinates of a desirous inverse pole point, wherein the desirous inverse pole point is a point that becomes the pole point surrounded by the three temporary vertices after the geometry transform operations are performed;

then, performing interpolation operations over the three first vertices according to equirectangular coordinates of the desirous inverse pole point to generate a data structure of the pole point;

then, performing triangle division operations to divide the transformed triangle including the three second vertices and the pole point into multiple regrouped triangles to output regrouped vertices with their data structures for the regrouped triangles as a part of the modified vertex list; and

repeating the steps of performing the geometry transform operations, measuring shortest distances, detecting, outputting, determining the equirectangular coordinates, performing the interpolation operations and performing the triangle division operations until all of the first vertices from the vertex/ID flow are processed.

24. The method according to claim 23, wherein the step of determining the equirectangular coordinates further comprises:

performing inverse geometry transform operations over the pole point according to the geometry parameters to generate the equirectangular coordinates of the desirous inverse pole point.

25. The method according to claim 23, wherein the step of calculating the geometry parameters further comprises:

performing inverse geometry transform operations over a north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points;

wherein the geometry parameters further comprises the equirectangular coordinates of the two inverse pole points.

26. The method according to claim 25, wherein the step of determining the coordinates comprises:

selecting one from the two inverse pole points as the desirous pole point according to the detecting result.

27. An image processing system for receiving an original vertex list from a calibration system and generating a panoramic image, comprising:

an image capture module for capturing a 360-degree horizontal field of view and 180-degree vertical field of view to generate a plurality of camera images;

a transform apparatus for generating a modified vertex list according to the original vertex list if the image capture module is moved; and

an image processing apparatus for performing rasterization, texture mapping and blending operations to form a modified panoramic image according to the modified vertex list if the image capture module is moved;

wherein the transform apparatus comprises;

a geometry information detector for measuring geometry information of the image capture module to determine whether the image capture module is moved;

a parameter setup unit coupled to the geometry information detector for calculating geometry parameters according to the geometry information;

a primitive assembly unit for receiving the original vertex list to generate a vertex/ID flow; and

a geometry processing circuit for performing geometry transform operations over the vertex/ID flow according to the geometry parameters to generate the modified vertex list if the image capture module is moved;

wherein the geometry processing circuit comprises;

a geometry transform unit for sequentially performing the geometry transform operations over a current group of three first vertices with their data structures from the vertex/ID flow according to the geometry parameters to correct equirectangular coordinates of the three first vertices as three temporary vertices if the image capture module is moved; and

a triangle assembly unit for measuring shortest distances along θ

axis among the three temporary vertices and their imaginary points to determine a transformed triangle with three second vertices as a part of the modified vertex list, wherein a difference between each temporary vertex and its corresponding imaginary point is 2π

radians along θ

axis;

wherein the geometry information comprises at least one of a displacement and rotation angles associated with the movement of the image capture module;

wherein the vertex/ID flow comprises a plurality of groups of three first vertices with their data structures, and each group of three first vertices form a corresponding triangle in a polygon mesh modeling the panoramic image;

wherein the modified vertex list comprises a plurality of groups of second vertices with their data structures, and each group of three second vertices form a corresponding triangle in the polygon mesh;

wherein the original vertex list comprises a plurality of third vertices with their data structures, and the calibration system conducts calibration between the plurality of camera images and the panoramic image for the plurality of third vertices to generate the original vertex list; and

wherein each of the data structures defines a corresponding vertex mapping between the plurality of camera images and the panoramic image.

28. The system according to claim 27, wherein the geometry information detector comprises:

a rotation detector for measuring the rotation angles of the image capture module;

wherein the geometry parameters comprise a rotation matrix and an inverse rotation matrix.

29. The system according to claim 27, wherein the geometry information detector comprises:

a displacement detector for measuring the displacement of the image capture module;

wherein the geometry parameters comprise the displacement.

30. The system according to claim 27, wherein the geometry processing circuit further comprises:

a pole detection unit for detecting whether the three temporary vertices surround a pole point to generate an indication signal.

31. The system according to claim 30, wherein the pole detection unit detects whether the three temporary vertices surround the pole point according to relative distances of the three temporary vertices along θ

axis.

32. The system according to claim 30, wherein the vertex/ID flow further comprises IDs for the groups of three first vertices, wherein the pole detection unit detects whether the three temporary vertices surround the pole point by comparing a plurality of pre-defined IDs with an ID associated with the current group of three first vertices.

33. The system according to claim 32, wherein the parameter setup unit further performs inverse geometry transform operations over a north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points, and calculates which polygons with the pre-defined IDs in the polygon mesh respectively contain the two inverse pole points according to a polygon size and a width of the panoramic image to output the geometry parameters further comprising the equirectangular coordinates of the two inverse pole points and the pre-defined IDs.

34. The system according to claim 30, wherein the geometry processing circuit further comprises:

a pole parameter generation unit for performing interpolation operations over the three input vertices according to equirectangular coordinates of a desirous inverse pole point to generate a data structure of the pole point if the pole point is detected, wherein the desirous inverse pole point is a point that becomes the pole point surrounded by the three temporary vertices after the geometry transform operations are performed; and

a triangle tessellation unit for dividing the transformed triangle including the three second vertices and the pole point into multiple regrouped triangles to output regrouped vertices with their data structures for the regrouped triangles as a part of the modified vertex list if the pole point is detected;

wherein the three second vertices with their data structures are directly outputted as a part of the modified vertex list if no pole point is detected.

35. The system according to claim 34, wherein the parameter setup unit further performs inverse geometry transform operations over a north pole and a south pole according to the geometry parameters to generate equirectangular coordinates of two inverse pole points, wherein the pole parameter generation unit further selects one from the two inverse pole points as the desirous inverse pole point according to the indication signal.

36. The system according to claim 35, wherein the pole parameter generation unit is coupled between the pole detection unit and the triangle tessellation unit.

37. The system according to claim 35, wherein the geometry transform unit and the triangle assembly unit are coupled between the pole parameter generation unit and triangle tessellation unit.

38. The system according to claim 34, wherein the geometry processing circuit further comprises:

an inverse geometry transform unit coupled between the pole detection unit and the pole parameter generation unit for performing inverse geometry transform operations over one of a north pole and a south pole to generate the equirectangular coordinates of the desirous inverse pole point according to the indication signal if the pole point is detected.

39. The system according to claim 27, wherein the original vertex list is divided into a plurality of original surface vertex lists and the modified vertex list is divided into a plurality of modified surface vertex lists, and numbers of original surface vertex lists and modified surface vertex lists are equal to a number of camera images.

40. A geometry processing circuit applied in an image processing system having an image capture module and an image processing apparatus, the image capture module capturing a field of view up to 360 degrees horizontally and 180 degrees vertically to generate a plurality of camera images, the geometry processing circuit comprising:

a geometry transform unit for sequentially performing geometry transform operations over a current group of three first vertices with their data structures from a vertex/ID flow according to geometry parameters to correct equirectangular coordinates of the three first vertices as three temporary vertices when the image capture module is moved; and

a triangle assembly unit for measuring shortest distances along θ

axis among the three temporary vertices and their imaginary points to determine a transformed triangle with three second vertices as a part of the modified vertex list, wherein a difference between each temporary vertex and its corresponding imaginary point is 2π

radians along θ

axis;

wherein the vertex/ID flow comprises a plurality of groups of three first vertices with their data structures, and each group of three first vertices form a corresponding triangle in a polygon mesh modeling the panoramic image;

wherein the modified vertex list comprises a plurality of groups of second vertices with their data structures, and each group of three second vertices form a corresponding triangle in the polygon mesh;

wherein each of the data structures defines a vertex mapping between a plurality of camera images from the image capture module and the panoramic image;

wherein the geometry parameters comprise at least one of a rotation matrix, an inverse rotation matrix and a displacement associated with the movement of the image capture module; and

wherein the modified vertex list is inputted to the image processing apparatus and causes the image processing apparatus to generate a modified panoramic image.

41. The circuit according to claim 40, further comprising:

a pole detection unit for detecting whether the three temporary vertices surround a pole point to generate an indication signal.

42. The circuit according to claim 41, wherein the pole detection unit detects whether the three temporary vertices surround the pole point according to relative distances of the three temporary vertices along θ

axis.

43. The circuit according to claim 41, wherein the pole detection unit detects whether the three temporary vertices surround the pole point by comparing a plurality of pre-defined IDs with an ID associated with the current group of three input first vertices.

44. The circuit according to claim 43, wherein two polygons with the pre-defined IDs in the polygon mesh respectively contain two inverse pole points.

45. The circuit according to claim 41, further comprising:

a pole parameter generation unit for performing interpolation operations over the three input vertices according to equirectangular coordinates of a desirous inverse pole point to generate a data structure of the pole point if the pole point is detected, wherein the desirous inverse pole point is a point that becomes the pole point surrounded by the three temporary vertices after the geometry transform operations are performed; and

a triangle tessellation unit for dividing the transformed triangle including the three second vertices and the pole point into multiple regrouped triangles to output regrouped vertices for the regrouped triangles with their data structures if the pole point is detected;

wherein the three second vertices with their data structures are directly outputted if no pole point is detected.

46. The circuit according to claim 45, wherein the pole parameter generation unit further selects one from two inverse pole points contained in the geometry parameters as the desirous inverse pole point according to the indication signal.

47. The circuit according to claim 46, wherein the pole parameter generation unit is coupled between the pole detection unit and the triangle tessellation unit.

48. The circuit according to claim 46, wherein the geometry transform unit and the triangle assembly unit are coupled between the pole parameter generation unit and triangle tessellation unit.

49. The circuit according to claim 45, further comprising:

an inverse geometry transform unit coupled between the pole detection unit and the pole parameter generation unit for performing inverse geometry transform operations over one of a north pole and a south pole to generate the equirectangular coordinates of the desirous inverse pole point according to the indication signal if the pole point is detected.