Electronic apparatus and method for incremental pose estimation and photographing thereof

• US 9,443,349 B2
• Filed: 12/18/2014
• Issued: 09/13/2016
• Est. Priority Date: 12/09/2014
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Wei, Chen-Hao, Wang, Te-Mei, Chen, Hsiao-Wei
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/576,163
• Publication Number:
  US 20160163091A1
• Time in Patent Office: 635 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

An electronic apparatus and a method for incremental pose estimation and photographing are provided. In the method, at least two images of a 3D object are captured at different positions encircling the 3D object. Displacements and angular displacements are detected when capturing the images. Features of the 3D object in the images, the displacements and the angular displacements are used to estimate a central position of the 3D object and a distance between the electronic apparatus and the 3D object. A circumference suitable for capturing the images of the 3D object is estimated based on the distance and is divided into several segments, and a timing interval of a timer is adjusted based on a length of the segments. A camera of the electronic apparatus is triggered at intervals set by the timer to capture the images of the 3D object.

Claims

1. A method for incremental pose estimation and photographing, adapted for an electronic apparatus having an image capturing device comprising a lens and a photosensitive element, a sensor, a timer and a calculation device, comprising:

capturing at least two images of a three-dimensional (3D) object at different positions encircling the 3D object by the image capturing device, and detecting displacements and angular displacements of the electronic apparatus by the sensor;

calculating a circumference suitable for capturing the images of the 3D object according to a central position of the 3D object and a distance between the 3D object and the electronic apparatus estimated from a plurality of features of the 3D object in the images, the displacements and the angular displacements by the calculation device which is coupled to the sensor;

dividing the circumference into a plurality of segments, and adjusting a timing interval of the timer according to a length of the segments by the calculation device; and

triggering the image capturing device at the timing intervals by the timer to capture the images of the 3D object.

2. The method for incremental pose estimation and photographing as claimed in claim 1, wherein after the step of triggering the image capturing device at the timing intervals by the timer, the method further comprises:

re-estimate the central position of the 3D object whenever a new image is captured by the image capturing device;

examining whether a difference between the re-estimated central position and the originally estimated central position exceeds a predetermined ratio; and

adjusting the circumference and the timing interval of the timer according to the re-estimated central position if the difference exceeds the predetermined ratio.

3. The method for incremental pose estimation and photographing as claimed in claim 1, wherein after the step of triggering the image capturing device at the timing intervals by the timer, the method further comprises:

re-estimate the circumference of the 3D object whenever a new image is captured by the image capturing device;

examining whether a difference between the re-estimated circumference and the originally estimated circumference exceeds a predetermined ratio; and

adjusting the timing interval of the timer according to the re-estimated circumference if the difference exceeds the predetermined ratio.

4. The method for incremental pose estimation and photographing as claimed in claim 1, wherein after the step of capturing the images of the 3D object at different positions encircling the 3D object by the image capturing device, and detecting the displacements and the angular displacements of the electronic apparatus by the sensor, the method further comprises:

estimating a first photographing pose of the electronic apparatus according to the displacement and the angular displacement of the electronic apparatus detected by the sensor when capturing each image;

estimating a second photographing pose of the electronic apparatus and calculating an overlap between the images by matching a plurality of features between images; and

respectively recording the images and corresponding pose tracking information into a plurality of nodes in a linked list, wherein the pose tracking information comprises the first photographing pose, the second photographing pose, a tolerance of the photographing poses and the overlap between the images.

5. The method for incremental pose estimation and photographing as claimed in claim 4, wherein the step of estimating the photographing pose of the electronic apparatus comprises:

dividing each move of the electronic apparatus into a plurality of small moves, and retrieving a small displacement and a small angular displacement detected by the sensor in each small move of the electronic apparatus; and

integrating the small displacement and the small angular displacement detected by the sensor in each small move to estimate a moving path of the electronic apparatus in a 3D space and the photographing pose of the electronic apparatus when capturing each image.

6. The method for incremental pose estimation and photographing as claimed in claim 5, wherein after the step of respectively recording the images and corresponding pose tracking information into the nodes in the linked list, the method further comprises:

finding a plurality of images with the closest pose tracking information from the linked list whenever the electronic apparatus captures a new image of the 3D object by the image capturing device, and adding a node to the linked list to insert the new image and corresponding pose tracking information between the images or in front of or behind one of the images.

7. The method for incremental pose estimation and photographing as claimed in claim 5, wherein after the step of estimating the circumference suitable for the electronic apparatus to capture the images of the 3D object, the method further comprises:

creating a surface enclosing the 3D object according to the circumference;

respectively projecting the images captured by the electronic apparatus onto the surface to identify a plurality of covered regions and a plurality of uncovered regions on the surface;

searching the uncovered region with a shortest distance to a present position of the electronic apparatus, and estimating a photographing position for capturing a new image of the uncovered region; and

calculating a moving direction and a moving distance of the electronic apparatus moving from the present position to the photographing position, and prompting the moving direction and the moving distance on a user interface of the electronic apparatus.

8. The method for incremental pose estimation and photographing as claimed in claim 7, wherein the step of identifying the covered regions and the uncovered regions on the surface comprises:

respectively projecting four corners of each image captured by the electronic apparatus onto the surface to identify the covered regions and the uncovered regions on the surface.

9. The method for incremental pose estimation and photographing as claimed in claim 7, wherein after the step of prompting the moving direction and the moving distance on the user interface of the electronic apparatus, the method further comprises:

capturing the new image of the uncovered region by the image capturing device;

re-estimating a new central position of the 3D object and a new distance between the 3D object and the electronic apparatus by re-analyzing the images and the new image;

adjusting the surface enclosing the 3D object to a new surface according to the new central position and the new distance; and

re-projecting the images captured by the electronic apparatus onto the new surface to identify the covered regions and the uncovered regions on the new surface.

10. The method for incremental pose estimation and photographing as claimed in claim 9, wherein the step of estimating the photographing position for capturing the new image of the uncovered region comprises:

estimating the photographing position for capturing the new image of the uncovered region to have an overlap of a predetermined ratio between the captured new image and the previously captured image.

11. The method for incremental pose estimation and photographing as claimed in claim 1, wherein the step of estimating the circumference suitable for the electronic apparatus to capture the images of the 3D object comprises:

estimating a plurality of circumferences suitable for the electronic apparatus to capture the images of the 3D object to form a spherical surface enclosing the 3D object, and dividing the circumferences into the segments for adjusting the timing interval of the timer.

12. The method for incremental pose estimation and photographing as claimed in claim 1, wherein the step of dividing the estimated circumference into the segments and adjusting the timing interval of the timer according to the length of the segments comprises:

dividing the estimated circumference into the segments according to a displacement of the electronic apparatus when capturing the images.

13. An electronic apparatus, comprising:

an image capturing device comprising a lens and a photosensitive element, capturing images of a 3D object at different positions encircling the 3D object;

a sensor, detecting displacements and angular displacements of the electronic apparatus when the image capturing device captures the images;

a timer, triggering the image capturing device to capture the images of the 3D object at timing intervals; and

a calculation device, coupled to the image capturing device, the sensor and the timer, executing a plurality of modules, wherein the modules comprise;

a circumference estimation module, calculating a circumference suitable for capturing the images of the 3D object according to a central position of the 3D object and a distance between the 3D object and the electronic apparatus estimated from a plurality of features of the 3D object in the images, the displacements and the angular displacements detected by the sensor; and

a timer setting module, dividing the circumference estimated by the circumference estimating module into a plurality of segments, and adjusting the timing interval of the timer according to a length of the segments.

14. The electronic apparatus as claimed in claim 13, wherein

after the timer triggers the image capturing device to capture a new image of the 3D object according to the adjusted timing interval, the circumference estimating module re-estimates the central position of the 3D object;

and

the timer setting module adjusts the timing interval of the timer according to the re-estimated the central position if a difference between the re-estimated central position and the originally estimated central position exceeds a predetermined ratio.

15. The electronic apparatus as claimed in claim 13, wherein

after the timer triggers the image capturing device to capture a new image of the 3D object according to the adjusted timing interval, the circumference estimating module further re-estimates the circumference of the 3D object;

and

the timer setting module adjusts the timing interval of the timer according to the re-estimated circumference if a difference between the re-estimated circumference and the originally estimated circumference exceeds a predetermined ratio.

16. The electronic apparatus as claimed in claim 13, wherein the modules further comprise:

a feature matching module, extracting and matching a plurality of features in the images to calculate overlaps between the images;

a pose estimation module, estimating a first photographing pose of the electronic apparatus according to the displacement and the angular displacement of the electronic apparatus detected by the sensor when capturing each image, and estimating a second photographing pose of the electronic apparatus according to the features matched by the feature matching module;

a pose tracking module, respectively recording the images and corresponding pose tracking information into a plurality of nodes in a linked list, wherein the pose tracking information comprises the first photographing pose, the second photographing pose, a tolerance of the photographing poses and the overlap between the images.

17. The electronic apparatus as claimed in claim 16, wherein the modules further comprise:

a path refinement module, dividing each move of the electronic apparatus into a plurality of small moves, retrieving a small displacement and a small angular displacement detected by the sensor in each small move, and integrating the small displacement and the small angular displacement detected by the sensor in each small move to estimate a moving path of the electronic apparatus in a 3D space and the photographing pose of the electronic apparatus when the image capturing device captures each image.

18. The electronic apparatus as claimed in claim 17, wherein the modules further comprise:

an image sorting module, finding a plurality of images with the closest pose tracking information from the linked list whenever the electronic apparatus captures a new image of the 3D object by the image capturing device, and adding a node to the linked list to insert the new image and corresponding pose tracking information between the images or in front of or behind one of the images.

19. The electronic apparatus as claimed in claim 17, wherein the modules further comprise:

a surface creating module, creating a surface enclosing the 3D object according to the circumference;

a projection module, respectively projecting the images captured by the electronic apparatus onto the surface to identify a plurality of covered regions and a plurality of uncovered regions on the surface;

a searching module, searching the uncovered region with a shortest distance to a present position of the electronic apparatus, and estimating a photographing position for capturing a new image of the uncovered region; and

a directing module, calculating a moving direction and a moving distance of the electronic apparatus moving from the present position to the photographing position, and prompting the moving direction and the moving distance on a user interface of the electronic apparatus.

20. The electronic apparatus as claimed in claim 19, wherein the circumference estimating module further estimates a plurality of circumferences suitable for the electronic apparatus to capture the images of the 3D object to form a spherical surface enclosing the 3D object, and dividing the circumferences into a plurality of the segments and providing the divided segments to the timer setting module for adjusting the timing interval of the timer.