HANDHELD PORTABLE OPTICAL SCANNER AND METHOD OF USING

• US 20140225985A1
• Filed: 04/16/2014
• Published: 08/14/2014
• Est. Priority Date: 10/17/2012
• Status: Active Grant

• Current Assignee: DotProduct LLC
• Original Assignee: DotProduct LLC
• Inventors: Klusza, Mark, Spring, Rafael
• Granted Patent: US 10,674,135 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A system and method for real-time or near-real time processing and post-processing of RGB-D image data using a handheld portable device and using the results for a variety of applications. The disclosure is based on the combination of off-the-shelf equipment (e.g. an RGB-D camera and a smartphone/tablet computer) in a self-contained unit capable of performing complex spatial reasoning tasks using highly optimized computer vision algorithms. New applications are disclosed using the instantaneous results obtained and the wireless connectivity of the host device for remote collaboration. One method includes steps of projecting a dot pattern from a light source onto a plurality of points on a scene, measuring distances to the points, and digitally reconstructing an image or images of the scene, such as a 3D view of the scene. A plurality of images may also be stitched together to re-position an orientation of the view of the scene.

Claims

1. A method comprising:

capturing an image in digital data with a camera;

storing digital data of the image on a non-transitory computer readable medium;

extracting gray values from image color channels of the image digital data;

creating image pyramids from the grey values and from depth data of the digital data;

computing a scene fitness value using the image pyramids;

predicting a camera pose;

aligning the image with each element of a first subset of selected keyframes, yielding an aligned pose and also yielding a quantity of poses and a corresponding quantity of overlap values with respect to the selected keyframes;

computing a new camera pose estimate using the aligned image and the quantity of poses and the quantity of overlap values;

creating a keyframe from the digital data of the image using the new camera pose estimate when desired;

after the keyframe is created, selecting a second subset of keyframes different from the first subset of keyframes and repeating the step of aligning with each keyframe of the second subset of keyframes to yield a plurality of pose values and overlap values;

deciding for each element of the selected first subset of keyframes and the selected second subset of keyframes whether new links are required to the keyframe in a keyframe pose graph; and

linking the keyframes.

2. The method of claim 1 wherein the keyframe is created upon user command from the digital image data and the camera pose equal to an identity matrix if a set of existing keyframes is empty or a camera pose equal to the estimated camera pose if the set of existing keyframes is non-empty.

3. The method of claim 2 wherein the digital image data used to create a keyframe comprises a red-green-blue- and depth-channel (RGB-D) image.

4. The method of claim 1, further comprising requesting and loading a new digital image from the camera or from the non-transitory computer readable medium and using the new digital image to repeat the steps of claim 1.

5. The method of claim 4, further comprising representing a 3D model through a collection of spatially positioned and oriented keyframes.

6. The method of claim 1, further comprising correcting the digital data of the stored image for characteristics of the camera according to a camera calibration.

7. The method of claim 1, wherein the camera pose is predicted using rotational data from the group consisting of:

a visual predictor based on image optical flow;

data from a digital gyroscope;

a linear motion model; and

a combination of these data sources.

8. The method of claim 5, further comprising sending data for visualization of the digital images and of the 3D model to a graphics processing unit.

9. The method of claim 1, further comprising, after the step of computing the new aligned pose, checking for a failure in alignment and if there is a failure in alignment, performing a re-localization procedure.

10. The method of claim 9, if there is no failure in alignment, further comprising preparing a real-time correctly oriented visualization in space of a saved 3D model for a user.

11. The method claim 1, further comprising computing estimates of overlap between the aligned image with a selection of keyframes to determine whether to create a new keyframe.

12. The method of claim 11, wherein a decision to create a new keyframe is determined from data comprising overlap values between the aligned image and a selection of keyframes and comprising digital depth data from the selection of keyframes.

13. The method of claim 1, further comprising reprojecting depth data from the digital image into a plurality of keyframes, each reprojection using an estimate of a relative pose resulting from an alignment step of the depth data with the keyframe and a calibration model of the camera.

14. The method of claim 13, further comprising recomputing depth data for a keyframe using a combination of reprojected depth data and the existing depth data of the keyframe.

15. The method of claim 5, further comprising transforming the 3D model from a local coordinate system to another coordinate system using digital images of physical targets acquired with the image, wherein the target is selected from the group consisting of:

a checkerboard target;

a QR code or QR-like code; and

a user-selected point.

16. The method of claim 15, further comprising adding, updating, differencing or refining geometry of a pre-existing 3D model with data acquired from the camera using a spatial pose of a visible target for positioning and orientating the camera in space, the visible target being also present in the 3D model, the 3D model selected from the group consisting of a plurality of digital images or keyframes, a spatial dataset and a CAD file.

17. The method of claim 15 wherein a position and an orientation of the camera for the existing 3D model is determined based on a localization procedure using an alignment algorithm that minimizes geometric and photometric alignment error.

18. The method of claim 16 wherein the pre-existing 3D model is loaded either from a digital storage medium or from another computer accessible through a computer network.

19. The method of claim 1, further comprising appending an additional image to a set of linked keyframes, the additional image selected from the group consisting of 3D data captured with the camera;

3-D data captured by a laser scanner; and

data from a 3D model or CAD software.

20. The method of claim 5, further comprising representing the 3D model in a compressed manner by compressing digital image data of keyframes into a lossless format.

21. An apparatus for performing the method of claim 1 and for collecting a plurality of spatially positioned and oriented keyframes and representing the collection of spatially positioned and oriented keyframes as a 3D model, the apparatus comprising:

an RGB-D camera for capturing and storing a plurality of digital images, the RGB-D camera including a depth computation system for deriving depth information for a plurality of arbitrary points in the digital images in real time;

a handheld computer system having a processor, a display with user interface controls, and a communications module, wherein the processor is in communication with the camera and the depth computation system; and

a non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to cause the RGB-D camera, the depth computation system, the handheld computer system and the communications module to perform one or more operations to gather the plurality of digital images and the depth information to form the 3D model, the 3D model suitable for presentation to a user on a user interface, the 3D model also suitable for appending additional data to update the 3D model.