Scanning of a handheld object for 3-dimensional reconstruction

• US 9,639,943 B1
• Filed: 12/21/2015
• Issued: 05/02/2017
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Kutliroff, Gershom, Madmony, Maoz
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/976,021
• Time in Patent Office: 498 Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20156 Automatic seed setting
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Techniques are provided for generating a 3-Dimensional (3D) reconstruction of a handheld object. An example method may include receiving 3D image frames of the object from a static depth camera, each frame including a color image and a depth map. Each of the frames is associated with an updated pose of the object during scanning. The method may also include extracting a segment of each frame corresponding to the object and the hand; isolating the hand from each extracted segment; and filtering each frame by removing regions of the frame outside of the extracted segment and removing regions of the frame corresponding to the isolated hand. The method may further include calculating the updated object pose in each filtered frame; and calculating a 3D position for each depth pixel from the depth map of each filtered frame based on the updated object pose associated with that filtered frame.

Claims

1. A processor-implemented method for generating a 3-Dimensional (3D) reconstruction of an object, the method comprising:

receiving, by a processor, a plurality of 3D image frames of a handheld object from a static depth camera, each frame comprising a color image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with an updated pose of the object;

extracting, by the processor, a segment of each 3D image frame corresponding to the object and a hand in which the object is being held;

isolating, by the processor, the hand from each extracted segment;

filtering, by the processor, each 3D image frame by removing regions of the frame outside of the extracted segment and removing regions of the frame corresponding to the isolated hand;

calculating, by the processor, the updated object pose in each filtered 3D image frame; and

calculating, by the processor, a 3D position for each depth pixel from the depth map frame of each filtered 3D image frame, the calculated position based on the updated object pose associated with that filtered 3D image frame.

2. The method of claim 1, further comprising rectifying the color image frame and the depth map frame to compensate for offset between sensors of the depth camera.

3. The method of claim 1, wherein the extracting further comprises determining the segment of each 3D image frame corresponding to the object and the hand based on distance to the camera and based on a connected components analysis.

4. The method of claim 1, wherein the isolating further comprises:

over-segmenting the extracted segment into sub-regions;

initializing a hand region seed comprising one or more sub-regions of a first of the image frames that border the image frame;

classifying sub-regions as being associated with the hand based on a comparison of color pixels and depth pixels of the sub-region to color pixels and depth pixels of the hand region seed; and

updating the hand region seed based on the classification.

5. The method of claim 4, wherein over-segmenting further comprises performing k-means clustering on the extracted segment.

6. The method of claim 1, wherein calculating the updated object pose further comprises estimating a rigid transformation of the object based on an Iterative Closest Point (ICP) matching operation performed on the depth pixels or an RGB feature matching operation performed on the color pixels.

7. The method of claim 1, further comprising accumulating the calculated 3D positions of each depth pixel into a global coordinate system based volumetric representation of the object.

8. The method of claim 7, further comprising providing feedback to a user holding the object, the feedback to guide the future movement of the object, the feedback comprising a raycast rendering of the volumetric representation of the object at a given point in time.

9. A system for generating a 3-Dimensional (3D) reconstruction of an object, the system comprising:

a static depth camera to provide a plurality of 3D image frames of a handheld object, each frame comprising a color image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with an updated pose of the object;

a hand and object segment extraction circuit to extract a segment of each 3D image frame corresponding to the object and a hand in which the object is being held;

a hand isolation circuit to isolate the hand from each extracted segment;

a object filter circuit to filter each 3D image frame by removing regions of the frame outside of the extracted segment and removing regions of the frame corresponding to the isolated hand;

an object pose update calculation circuit to calculate the updated object pose in each filtered 3D image frame; and

an object depth pixel accumulation circuit to calculate a 3D position for each depth pixel from the depth map frame of each filtered 3D image frame, the calculated position based on the updated object pose associated with that filtered 3D image frame.

10. The system of claim 9, further comprising a rectification circuit to rectify the color image frame and the depth map frame to compensate for offset between sensors of the depth camera.

11. The system of claim 9, wherein the hand and object segment extraction circuit is further to determine the segment of each 3D image frame corresponding to the object and the hand based on distance to the camera and based on a connected components analysis.

12. The system of claim 9, wherein the hand isolation circuit is further to:

over-segment the extracted segment into sub-regions;

initialize a hand region seed comprising one or more sub-regions of a first of the image frames that border the image frame;

classify sub-regions as being associated with the hand based on a comparison of color pixels and depth pixels of the sub-region to color pixels and depth pixels of the hand region seed; and

update the hand region seed based on the classification.

13. The system of claim 12, wherein over-segmenting further comprises performing k-means clustering on the extracted segment.

14. The system of claim 9, wherein the object pose update calculation circuit is further to estimate a rigid transformation of the object based on an Iterative Closest Point (ICP) matching operation performed on the depth pixels or an RGB feature matching operation performed on the color pixels.

15. The system of claim 9, wherein the object depth pixel accumulation circuit is further to accumulate the calculated 3D positions of each depth pixel into a global coordinate system based volumetric representation of the object.

16. The system of claim 15, further comprising a user feedback circuit to provide feedback to a user holding the object, the feedback to guide the future movement of the object, the feedback comprising a raycast rendering of the volumetric representation of the object at a given point in time.

17. At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for generating a 3-Dimensional (3D) reconstruction of an object, the operations comprising:

receiving a plurality of 3D image frames of the object from a static depth camera, each frame comprising a color image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with an updated pose of the object;

extracting a segment of each 3D image frame corresponding to the object and a hand in which the object is being held;

isolating the hand from each extracted segment;

filtering each 3D image frame by removing regions of the frame outside of the extracted segment and removing regions of the frame corresponding to the isolated hand;

calculating the updated object pose in each filtered 3D image frame; and

calculating a 3D position for each depth pixel from the depth map frame of each filtered 3D image frame, the calculated position based on the updated object pose associated with that filtered 3D image frame.

18. The computer readable storage medium of claim 17, further comprising rectifying the color image frame and the depth map frame to compensate for offset between sensors of the depth camera.

19. The computer readable storage medium of claim 17, wherein the extracting further comprises determining the segment of each 3D image frame corresponding to the object and the hand based on distance to the camera and based on a connected components analysis.

20. The computer readable storage medium of claim 17, wherein the isolating further comprises:

over-segmenting the extracted segment into sub-regions;

initializing a hand region seed comprising one or more sub-regions of a first of the image frames that border the image frame;

classifying sub-regions as being associated with the hand based on a comparison of color pixels and depth pixels of the sub-region to color pixels and depth pixels of the hand region seed; and

updating the hand region seed based on the classification.

21. The computer readable storage medium of claim 20, wherein over-segmenting further comprises performing k-means clustering on the extracted segment.

22. The computer readable storage medium of claim 17, wherein calculating the updated object pose further comprises estimating a rigid transformation of the object based on an Iterative Closest Point (ICP) matching operation performed on the depth pixels or an RGB feature matching operation performed on the color pixels.

23. The computer readable storage medium of claim 17, further comprising accumulating the calculated 3D positions of each depth pixel into a global coordinate system based volumetric representation of the object.

24. The computer readable storage medium of claim 23, further comprising providing feedback to a user holding the object, the feedback to guide the future movement of the object, the feedback comprising a raycast rendering of the volumetric representation of the object at a given point in time.