TECHNOLOGIES FOR AUTONOMOUS THREE-DIMENSIONAL MODELING

• US 20180286119A1
• Filed: 03/30/2017
• Published: 10/04/2018
• Est. Priority Date: 03/30/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Felip Leon, Javier, Gonzalez Aguirre, David I., Alvarez, Ignacio J.
• Granted Patent: US 10,455,222 B2
• Time in Patent Office: Days
• CPC Class Codes: B25J 13/085 Force or torque sensors B25J13/082, B25J13/084 take precedence
• CPC Class Codes: B25J 15/103 for gripping the object in three contact points
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/0096 co-operating with a working support, e.g. work-table
• CPC Class Codes: B25J 9/046 Revolute coordinate type
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G01G 19/52 Weighing apparatus combined with other objects, e.g. furniture with walking sticks A45B3/08
• CPC Class Codes: G01L 5/00 Apparatus for, or methods of, measuring force, work, mechanical power, or torque, specially adapted for specific purposes
• CPC Class Codes: G01L 5/009 Force sensors associated with material gripping devices
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

An autonomous object modeler includes a modeling table, a controllable arm, a depth camera attached to the controllable arm, and a controller to control operation of the modeling table, the controllable arm, and the depth camera. The modeling table may be movable and includes a mass sensor to produce mass sensor data indicative of a mass of an object positioned on the modeling table. The controllable arm includes a force-torque sensor to produce force-torque sensor data indicative of an inertia of the object while the object is moved by the controllable arm. The controller is configured to control operation of the controllable arm to reposition the object on the modeling table to generate three-dimensional models of the object. The three-dimensional models include the mass data and the inertia data.

Claims

1. An object modeler for autonomously modeling an object, the object modeler comprising:

a modeling table having a mass sensor to produce mass sensor data indicative of a mass of an object positioned on the modeling table;

a controllable arm having a controllable manipulator to grasp and move the object, wherein the controllable arm includes a force-torque sensor to produce force-torque sensor data indicative of an inertia of the object while the object is moved by the controllable arm;

a depth camera attached to the controller arm to capture depth images of the object; and

a controller to control operation of the modeling table, the controllable arm, and the depth camera.

2. The object modeler of claim 1, wherein the controller is to generate a three-dimensional model of the object based on the depth images captured by the depth camera, wherein the three-dimensional model includes (i) mass data indicative of the mass of the object, (ii) an estimated center of mass of the object, and (iii) an inertia matrix to the three-dimensional model.

3. The object modeler of claim 1, wherein the controller is to:

determine mass data indicative of the mass of the object based on the mass sensor data received from the mass sensor;

control the depth camera to capture a depth image of the object while the object is positioned on the modeling table;

determine whether a surface of the object is occluded from the depth camera;

control the controllable arm to reposition the object on the modeling table to a new position at which the occluded surface is viewable by the depth camera;

determine inertia data of the object based on the force-torque sensor data while the object is being repositioned by the controllable arm;

control the depth camera to capture a subsequent depth image of the object while in the new position;

generate a three-dimensional model of the object based on the captured depth images of the object; and

append the mass data and inertia data to the three-dimensional model.

4. The object modeler of claim 3, wherein to control the controllable arm to reposition the object on the modeling table comprises to:

determine a grasp plan for the controllable arm based on a present point cloud model of the object, wherein the grasp plan defines movements of the controller arm required for the controller arm to grasp and move the object to the new position;

control the controllable manipulator to grasp the object based on the grasp plan; and

control the controllable arm to move the object to the new position based on the grasp plan.

5. The object modeler of claim 3, wherein to determine the inertia data comprises to (i) determine an estimated center of mass of the object based on the force-torque sensor data and the movements of the object and (ii) determine an inertia matrix based on the force-torque sensor data and the movements of the object, wherein the inertia matrix defines an inertia of the object.

6. The object modeler of claim 3, wherein to generate the three-dimensional model comprises to convert a point cloud model generated based on the depth images of the object to a three-dimensional mesh model.

7. The object modeler of claim 3, wherein to append the mass data and inertia data comprises to append mass data indicative of the mass of the object, an estimated center of mass of the object, and an inertia matrix that defines an inertia of the object to the three-dimensional model.

8. The object modeler of claim 3, wherein the controller is further to:

segment the subsequent depth image to remove background structures form the subsequent depth image;

align the segmented subsequent depth image with the present point cloud model; and

aggregate the aligned subsequent depth image with the point cloud model to generate an updated point cloud model.

9. The object modeler of claim 3, wherein the controller is further to:

determine whether determination of the inertia data is completed;

control the controllable arm to move the object in response to a determination that the determination of the inertia data is not completed;

determine additional inertia data of the object while the object is being moved by the controllable arm; and

update the inertia data with the additional inertia data.

10. A method for autonomously modeling an object, the method comprising:

determining, by an object modeler, mass data indicative of a mass of the object while the object is positioned on a modeling table of the object modeler;

capturing, by a depth camera of the object modeler, a depth image of the object;

determining, by the object modeler, whether a surface of the object is occluded from the depth camera;

repositioning, by a controllable arm of the object modeler, the object on the modeling table to a new position at which the occluded surface is viewable by the depth camera;

determining, by the object modeler, inertia data of the object while the object is being repositioned by the controllable arm;

capturing, by the depth camera the object modeler, a subsequent depth image of the object while in the new position;

generating, by the object modeler, a three-dimensional model of the object based on the captured depth images of the object; and

appending, by the object modeler, the mass data and inertia data to the three-dimensional model.

11. The method of claim 10, wherein determining the mass data comprises:

receiving, by a controller of the object modeler and from a weight sensor of the modeling table, sensor data indicative of the mass of the object while the object is positioned on the modeling table; and

determining, by the controller, the mass data based on the sensor data.

12. The method of claim 10, wherein repositioning the object on the modeling table comprises:

determining, by a controller of the object modeler, a grasp plan for the controllable arm based on a present point cloud model of the object, wherein the grasp plan defines movements of the controller arm required for the controller arm to grasp and move the object to the new position;

grasping, by a manipulator of the controllable arm, the object based on the grasp plan; and

moving, by the controller arm, the object to the new position based on the grasp plan.

13. The method of claim 10, wherein determining inertia data comprises:

receiving, by a controller of the object modeler and from a force-torque sensor of the controllable arm, force-torque sensor data indicative of a torque of the controller arm while performing movements of the object with the controllable arm;

determining, by the controller, an estimated center of mass of the object based on the force-torque sensor data and the movements of the object; and

determining, by the controller, an inertia matrix based on the force-torque sensor data and the movements of the object, wherein the inertia matrix defines an inertia of the object; and

14. The method of claim 10, wherein appending the mass data and inertia data comprises appending mass data indicative of the mass of the object, an estimated center of mass of the object, and an inertia matrix that defines an inertia of the object to the three-dimensional model.

15. The method of claim 10, further comprising updating a present point cloud model of the object based on the subsequent depth image of the object.

16. The method of claim 15, wherein updating the present point cloud module comprises:

segmenting the subsequent depth image to remove background structures form the subsequent depth image;

aligning the segmented subsequent depth image with the present point cloud model; and

aggregating the aligned subsequent depth image with the point cloud model to generate an updated point cloud model.

17. The method of claim 10, further comprising:

determining, by a controller, whether determination of the inertia data is completed;

moving the object by the controllable arm of the object modeler in response to a determination that the determination of the inertia data is not completed;

determining, by the controller, additional inertia data of the object while the object is being moved by the controllable arm; and

updating, by the controller, the inertia data with the additional inertia data.

18. One or more machine-readable storage media comprising a plurality of instructions stored thereon that, when executed, causes a controller of an object modeler to:

determine mass data indicative of a mass of the object while the object is positioned on a modeling table of the object modeler;

capture a depth image of the object;

determine whether a surface of the object is occluded from the depth camera;

reposition the object on the modeling table to a new position at which the occluded surface is viewable by the depth camera;

determine inertia data of the object while the object is being repositioned by the controllable arm;

capture a subsequent depth image of the object while in the new position;

generate a three-dimensional model of the object based on the captured depth images of the object; and

append the mass data and inertia data to the three-dimensional model.

19. The one or more machine-readable storage media of claim 18, wherein to determine the mass data comprises to:

receive, from a weight sensor of the modeling table, sensor data indicative of the mass of the object while the object is positioned on the modeling table; and

determine the mass data based on the sensor data.

20. The one or more machine-readable storage media of claim 18, wherein to reposition the object on the modeling table comprises to:

determine a grasp plan for the controllable arm based on a present point cloud model of the object, wherein the grasp plan defines movements of the controller arm required for the controller arm to grasp and move the object to the new position;

grasp the object based on the grasp plan; and

move the object to the new position based on the grasp plan.

21. The met one or more machine-readable storage media hod of claim 18, wherein to determine inertia data comprises to:

receive, from a force-torque sensor of the controllable arm, force-torque sensor data indicative of a torque of the controller arm while performing movements of the object with the controllable arm;

determine an estimated center of mass of the object based on the force-torque sensor data and the movements of the object; and

determine an inertia matrix based on the force-torque sensor data and the movements of the object, wherein the inertia matrix defines an inertia of the object; and

22. The one or more machine-readable storage media of claim 18, wherein to append the mass data and inertia data comprises to append mass data indicative of the mass of the object, an estimated center of mass of the object, and an inertia matrix that defines an inertia of the object to the three-dimensional model.

23. The one or more machine-readable storage media of claim 18, wherein the plurality of instructions, when executed, further cause the object modeler to update a present point cloud model of the object based on the subsequent depth image of the object.

24. The one or more machine-readable storage media of claim 23, wherein to update the present point cloud module comprises to:

segment the subsequent depth image to remove background structures form the subsequent depth image;

align the segmented subsequent depth image with the present point cloud model; and

aggregate the aligned subsequent depth image with the point cloud model to generate an updated point cloud model.

25. The one or more machine-readable storage media of claim 18, wherein the plurality of instructions, when executed, further cause the object modeler to:

determine whether determination of the inertia data is completed;

move, in response to a determination that the determination of the inertia data is not completed;

determine additional inertia data of the object while the object is being moved by the controllable arm; and

update the inertia data with the additional inertia data.