METHOD AND SYSTEM FOR FRAME STITCHING BASED IMAGE CONSTRUCTION IN AN INDOOR ENVIRONMENT

• US 20190333187A1
• Filed: 02/04/2019
• Published: 10/31/2019
• Est. Priority Date: 04/30/2018
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: GUBBI LAKSHMINARASIMHA, Jayavardhana Rama, RAMASWAMY, Akshaya, RAJ, Rishin, PURUSHOTHAMAN, Balamuralidhar
• Granted Patent: US 10,803,551 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

This disclosure relates generally to a method and a system for frame stitching based image construction for an indoor environment. The method enables constructing an image of a scene by stitching a plurality of key frames identified from a plurality of image frames captured by a mobile imaging device. The method overcomes multiple challenges posed by the indoor environment, effectively providing clean stitching of the key frames to construct the image of the scene. The method provides image stitching approach that combines visual data from the mobile imaging device and an inertial sensor from an Inertial Measurement Unit (IMU) mounted on the mobile imaging device, with a feedback for error correction to generate robust stitching outputs in indoor scenario.

Claims

1. A processor implemented frame stitching based image construction method, the method comprising:

receiving (302),

a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment;

a positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames; and

a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV, and frame dimensions of the plurality image frames;

selecting (304) a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame, wherein each key frame from the set of key frames provides a maximum average of luminance and variance;

stitching (306) the set of key frames to construct an image of the scene of interest, wherein the stitching comprises;

selecting one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames;

determining a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the plurality of device parameters;

generating a refined warping matrix from the warping matrix to perform an image intensity based registration refinement;

estimating a constraint feature based warping matrix to stitch the selected key frame and the corresponding consecutive key frame, wherein the constraint feature based warping matrix is estimated from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame, wherein the extracted plurality of features provide key point correspondence between the selected key frame and the corresponding consecutive key frame; and

constructing (308) an image of the scene of interest by iterating the stitching of each the key frame of the set of keyframes and the corresponding consecutive key frame of each key frame, in sequence.

2. The processor implemented method of claim 1, wherein the method further comprises applying a selective approach with feedback for the frame stitching based image construction method, where the selective approach with feedback comprises:

choosing (402) one of;

the refined warping matrix as a final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame are below a key point threshold; and

the constrained feature based warping matrix as the final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features providing key point correspondence between the key frame and the consecutive key frame and are above the key point threshold;

modifying (404) the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix, wherein the previous warping matrix is the warping matrix created during frame stitching of preceding frame;

stitching (406) one key frame and the corresponding consecutive keyframe based on the modified warping matrix; and

constructing (408) the image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key frame, in sequence.

3. The processor implemented method of claim 1, wherein generating the refined warping matrix from the warping matrix comprises applying a gradient decent optimizer and minimization of a mean square error (MSE) metric with transformation set to translation.

4. The processor implemented method of claim 1, wherein the constraint feature based warping matrix is defined by a rotation constraint, a refined warping matrix based horizontal translation constraint, and a refined warping matrix based vertical translation constraint set using the refined warping matrix, wherein the constraint feature based warped matrix provides refinement over the refined warping matrix.

5. An image construction system (102) for frame stitching based image construction, the image construction system (102) comprises:

a processor (202);

an Input/Output (I/O) interface (206);

a memory (204), the memory comprising a frame stitching module (212);

wherein the frame stitching module (212) is configured to;

receive,

a sequence of a plurality of image frames from a mobile imaging device (104), wherein the plurality of image frames correspond to a scene of interest in an indoor environment;

a positional data of the mobile imaging device (104) from an Inertial Measurement Unit (IMU) of the mobile imaging device (104), wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames; and

a plurality of device parameters of the mobile imaging device (104), wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV and frame dimensions of the plurality image frames;

select a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame, wherein each key frame from the set of key frames provides a maximum average of luminance and variance over a predefined interval defined by number of image frames;

stitch the set of key frames to construct an image of the scene of interest, wherein to stitch the key frames the frame stitching module (212) is configured to;

select one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames;

determine a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the plurality of device parameters;

generate a refined warping matrix from the warping matrix to perform an image intensity based registration refinement;

estimate a constraint feature based warping matrix to stitch the selected keyframe and the corresponding consecutive key frame, wherein the constraint feature based warping matrix is estimated from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame, wherein the extracted plurality of features provide key point correspondence between the key frame and the consecutive key frame; and

construct an image of the scene of interest by iterating the stitching of each key frame from the set of key frames and the corresponding consecutive key frame of each key frame, in sequence.

6. The image construction system (102) of claim 5, wherein the frame stitching module (212) is configured to apply a selective approach with feedback for the frame stitching based image construction method, wherein to apply the selective approach with feedback the frame stitching module (212) is configured to:

choose one of;

the refined warping matrix as a final warping matrix for stitching the one key frame and the corresponding consecutive keyframes, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame, are below a key point threshold; and

the constrained feature based warping matrix as the final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features providing key point correspondence between the key frame and the consecutive key frame are above the key point threshold;

modify the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix, wherein the previous warping matrix is the warping matrix created during frame stitching of preceding frame;

stitch one key frame and the corresponding consecutive keyframe based on the modified warping matrix; and

construct the image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key frame, in sequence.

7. The image construction system (102) of claim 5, wherein the frame stitching module (212) is configured to generate the refined warping matrix from the warping matrix comprises applying a gradient decent optimizer and minimization of a mean square error (MSE) metric with transformation set to translation.

8. The image construction system (102) of claim 5, wherein the constraint feature based warping matrix is defined by a rotation constraint, a refined warping matrix based horizontal translation constraint and a refined warping matrix based vertical translation constraint set using the refined warping matrix, wherein the constraint feature based warped matrix provides refinement over the refined warping matrix.

9. One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors causes the one or more hardware processor to perform a method for frame stitching based image construction, said method comprising:

receiving (302),

a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment;

a positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames; and

a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV, and frame dimensions of the plurality image frames;

selecting (304) a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame, wherein each key frame from the set of key frames provides a maximum average of luminance and variance;

stitching (306) the set of key frames to construct an image of the scene of interest, wherein the stitching comprises;

selecting one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames;

determining a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the plurality of device parameters;

generating a refined warping matrix from the warping matrix to perform an image intensity based registration refinement;

estimating a constraint feature based warping matrix to stitch the selected key frame and the corresponding consecutive key frame, wherein the constraint feature based warping matrix is estimated from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame, wherein the extracted plurality of features provide key point correspondence between the selected key frame and the corresponding consecutive key frame; and

constructing (308) an image of the scene of interest by iterating the stitching of each the key frame of the set of keyframes and the corresponding consecutive key frame of each key frame, in sequence.

10. The one or more non-transitory machine readable information storage mediums of claim 9, further comprising:

choosing (402) one of;

the refined warping matrix as a final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame are below a key point threshold; and

the constrained feature based warping matrix as the final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features providing key point correspondence between the key frame and the consecutive key frame and are above the key point threshold;

modifying (404) the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix, wherein the previous warping matrix is the warping matrix created during frame stitching of preceding frame;

stitching (406) one key frame and the corresponding consecutive keyframe based on the modified warping matrix; and

constructing (408) the image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key frame, in sequence.

11. The one or more non-transitory machine readable information storage mediums of claim 9, wherein generating the refined warping matrix from the warping matrix comprises applying a gradient decent optimizer and minimization of a mean square error (MSE) metric with transformation set to translation.

12. The one or more non-transitory machine readable information storage mediums of claim 9, wherein the constraint feature based warping matrix is defined by a rotation constraint, a refined warping matrix based horizontal translation constraint, and a refined warping matrix based vertical translation constraint set using the refined warping matrix, wherein the constraint feature based warped matrix provides refinement over the refined warping matrix.