Detection and reconstruction of an environment to facilitate robotic interaction with the environment
First Claim
1. A method comprising:
- determining a three-dimensional (3D) virtual environment based on data received from one or more sensors, the 3D virtual environment being representative of a physical environment of a robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment;
determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors;
determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects;
based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images;
based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and
providing an instruction to the robotic manipulator to move the particular physical object.
5 Assignments
0 Petitions
Accused Products
Abstract
Methods and systems for detecting and reconstructing environments to facilitate robotic interaction with such environments are described. An example method may involve determining a three-dimensional (3D) virtual environment representative of a physical environment of the robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment. The method may then involve determining two-dimensional (2D) images of the virtual environment including 2D depth maps. The method may then involve determining portions of the 2D images that correspond to a given one or more physical objects. The method may then involve determining, based on the portions and the 2D depth maps, 3D models corresponding to the portions. The method may then involve, based on the 3D models, selecting a physical object from the given one or more physical objects. The method may then involve providing an instruction to the robotic manipulator to move that object.
311 Citations
22 Claims
-
1. A method comprising:
-
determining a three-dimensional (3D) virtual environment based on data received from one or more sensors, the 3D virtual environment being representative of a physical environment of a robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment; determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors; determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects; based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images; based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and providing an instruction to the robotic manipulator to move the particular physical object. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:
-
determining a three-dimensional (3D) virtual environment based on data received from one or more sensors, the 3D virtual environment being representative of a physical environment of a robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment; determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors; determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects; based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images; based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and providing an instruction to the robotic manipulator to move the particular physical object. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A system comprising:
-
a robotic manipulator; one or more sensors; at least one processor; and data storage comprising instructions executable by the at least one processor to cause the system to perform functions comprising; determining a three-dimensional (3D) virtual environment based on data received from the one or more sensors, the 3D virtual environment being representative of a physical environment of the robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment; determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors; determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects; based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images; based on the determined 3D models, selecting a particular physical object from the given one or more physical objects; and providing an instruction to the robotic manipulator to move the particular physical object. - View Dependent Claims (16, 17, 18, 19, 20, 21, 22)
-
Specification