Integration of auxiliary sensors with point cloud-based haptic rendering and virtual fixtures

• US 10,394,327 B2
• Filed: 09/11/2015
• Issued: 08/27/2019
• Est. Priority Date: 09/12/2014
• Status: Active Grant

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Chizeck, Howard Jay, Huang, Kevin, Ryden, Fredrik, Stewart, Andrew
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/510,190
• Publication Number:
  US 20180232052A1
• Time in Patent Office: 1,446 Days
• Field of Search: 345619
• CPC Class Codes: B23K 9/0953 using computing means
• CPC Class Codes: G05B 2219/40 Robotics, robotics mapping to robotics vision
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

Apparatus and methods for generating virtual environment displays based on a group of sensors are provided. A computing device can receive first data about an environment from a first group of one or more sensors. The computing device can model the environment as a virtual environment based on the first data. The computing device can determine whether to obtain additional data to model the environment. After determining to obtain additional data to model the environment, the computing device can: receive second data about the environment, and model the environment as the virtual environment based on at least the second data. The computing device can generate a display of the virtual environment.

Claims

1. A method, comprising:

receiving first data about an environment at a computing device from a first group of one or more sensors, wherein the environment comprises a first object and a second object, wherein the first object comprises a robotic device, and wherein the first data comprises a plurality of data points related to the environment;

modeling the environment as a virtual environment using the computing device, wherein the virtual environment is based on the first data, and wherein the robotic device is controlled using the virtual environment of the computing device;

determining whether to obtain one or more additional data points for the plurality of data points to model the environment using the computing device by;

determining whether an object distance in the environment between the first object and the second object is within a predetermined distance, and

after determining that the object distance is within the predetermined distance, determining whether to obtain one or more additional data points for the plurality of data points to model a surface of the second object in the environment so that the first object avoids contacting the second object;

after determining to obtain additional data points for the plurality of data points to model the surface of the second object in the environment, using the computing device to;

receive second data about the environment, the second data comprising the one or more additional data points, and

model the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment using the computing device.

2. The method of claim 1, wherein receiving the second data about the environment comprises:

determining whether a first sensor of the first group of one or more sensors is configured to obtain at least one data point of the one or more additional data points; and

after determining that the first sensor is configured to provide at least one data point of the additional data points, obtaining at least part of the second data using the first sensor.

3. The method of claim 2, wherein determining whether the first sensor of the first group of one or more sensors is configured to obtain at least one data point of the one or more additional data points comprises determining whether the first sensor is configured to measure a particular property of the environment.

4. The method of claim 2, wherein determining whether the first sensor of the first group of one or more sensors is configured to obtain at least one data point of the one or more additional data points comprises determining whether the first sensor is configured to obtain the at least one data point of the one or more additional data points based on a range of the first sensor.

5. The method of claim 1, wherein receiving the second data comprises:

determining whether a first sensor of the first group of one or more sensors interferes with a second sensor of the first group of one or more sensors; and

after determining that the first sensor interferes with the second sensor;

deactivating the first sensor and/or the second sensor, and

after deactivating the first sensor and/or the second sensor, obtaining the second data from activated sensors of the first group of one or more sensors.

6. The method of claim 1, wherein the first data comprises first depth data,

wherein modeling the environment as the virtual environment comprises modeling the environment using a first plurality of points that are obtained from the first depth data,

wherein determining whether to obtain additional data to model the environment comprises determining whether the first plurality of points is sufficient to model the environment,

wherein the second data comprises second depth data, and

wherein modeling the environment as the virtual environment based on at least the second data comprises modeling the environment using at least a second plurality of points that are obtained from the second depth data.

7. The method of claim 6, wherein the plurality of data points comprises the first plurality of points, and wherein receiving the second data comprises:

requesting the second depth data from a second group of one or more sensors, wherein the second group of one or more sensors differs from the first group of one or more sensors;

generating the second plurality of points from the second depth data; and

merging the plurality of data points and the second plurality of points into the plurality of data points.

8. The method of claim 7, wherein the predetermined distance is based on a sensing distance that a first sensor of the second group of one or more sensors is configured to sense the surface of the object.

9. The method of claim 7, wherein requesting second depth data from the second group of one or more sensors comprises using the first sensor of the second group of one or more sensors to obtain the one or more additional data points.

10. The method of claim 7, further comprising:

after merging the plurality of data points with the second plurality of points, determining that the first object moved in the environment with respect to the second object;

determining whether a new object distance between the first object and the second object is within the predetermined distance; and

after determining that the new object distance is within the predetermined distance, determining whether to obtain at least one additional data point for the plurality of data points to model the surface of the second object in the environment so that the first object avoids contacting the second object.

11. The method of claim 1, wherein the virtual environment comprises a first virtual object corresponding to the first object and a second virtual object corresponding to the second object, and wherein determining whether to obtain one or more additional data points of the plurality of data points to model the surface of the second object in the environment so that the first object avoids contacting the second object comprises:

determining a virtual-object distance in the virtual environment between the first virtual object and the second virtual object; and

determining whether to obtain the one or more additional data points of the plurality of data points to model the surface of the second object based on the virtual-object distance.

12. The method of claim 11, wherein the first virtual object comprises a virtual haptic fixture.

13. The method of claim 11, wherein the first virtual object is associated with a plurality of sensors, and wherein receiving second data about the environment comprises selecting a group of one or more sensors from the plurality of sensors to obtain the second data based on the virtual-object distance.

14. The method of claim 11, further comprising:

determining that the first virtual object moved in the environment with respect to the second virtual object;

determining whether a new virtual-object distance between the first virtual object and the second virtual object is within the predetermined distance; and

after determining that the new virtual-object distance is within the predetermined distance, determining whether to obtain at least one additional data point for the plurality of data points to model the surface of the second object in the environment so that the first object avoids contacting the second object.

15. The method of claim 11, wherein the first virtual object represents a robotic device controlled by the computing device.

16. The method of claim 1, further comprising:

performing a task in the environment using one or more commands provided by the computing device, wherein the one or more commands are generated based on haptic feedback provided for at least one virtual haptic fixture in the virtual environment.

17. The method of claim 16, wherein the at least one virtual haptic feature comprises a multi-sensor dynamic virtual fixture (MSDVF) that comprises at least one virtual fixture region that is updated using data obtained from a first plurality of sensors.

18. The method of claim 17, wherein the task comprises a first portion of the task and a second portion of the task, and wherein performing the task comprises:

performing the first portion of the task utilizing the MSDVF;

at least after the first portion of the task was performed, obtaining first-portion data from the first plurality of sensors;

updating the MSDVF based on the first-portion data; and

after updating the MSDVF, performing the second portion of the task utilizing the updated MSDVF.

19. The method of claim 18, wherein updating the MSDVF comprises one or more of:

modifying a geometry of the MSDVF based on the first-portion data;

changing at least a portion of the MSDVF from being a guidance fixture to a forbidden-region fixture based on the first-portion data; and

changing at least a portion of the MSDVF from being a forbidden-region fixture to a guidance fixture based on the first-portion data.

20. A computing device, comprising:

a processor; and

data storage, storing instructions that, upon execution by the processor, cause the computing device to perform functions comprising;

receiving first data about an environment from a first group of one or more sensors, wherein the environment comprises a first object and a second object, wherein the first object comprises a robotic device, and wherein the first data comprises a plurality of data points related to the environment;

modeling the environment as a virtual environment based on the first data, wherein the virtual environment is based on the first data, and wherein the robotic device is controlled using the virtual environment of the computing device;

determining whether to obtain one or more additional data points for the plurality of data points to model the environment by;

determining whether an object distance in the environment between the first object and the second object is within a predetermined distance, and

after determining that the object distance is within the predetermined distance, determining whether to obtain one or more additional data points for the plurality of data points to model a surface of the second object in the environment so that the first object avoids contacting the second object;

after determining to obtain additional data points for the plurality of data points to model the surface of the second object in the environment;

receiving second data about the environment, the second data comprising the one or more additional data points, and

modeling the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment.

21. The computing device of claim 20, wherein the first group of one or more sensors comprises at least one of:

a red-green-blue-depth (RGB-D) camera, a video and/or still camera, a seashell sensor, a proximity sensor, a radiation sensor, an infra-red sensor, a thermometer, a pH sensor, a chemical sensor, a neural signal sensor, a SONAR device, a RADAR device, a LIDAR device, a laser, a position sensor, an electrical sensor, and a biological sensor.

22. An article of manufacture, comprising a tangible computer-readable storage medium storing instructions that, upon execution by a processor of the article of manufacture, cause the article of manufacture to perform functions comprising:

receiving first data about an environment from a first group of one or more sensors, wherein the environment comprises a first object and a second object, wherein the first object comprises a robotic device, and wherein the first data comprises a plurality of data points related to the environment;

modeling the environment as a virtual environment based on the first data, wherein the virtual environment is based on the first data, and wherein the robotic device is controlled using the virtual environment of the computing device;

determining whether to obtain one or more additional data points for the plurality of data points to model the environment by;

determining whether an object distance in the environment between the first object and the second object is within a predetermined distance, and

after determining that the object distance is within the predetermined distance, determining whether to obtain one or more additional data points for the plurality of data points to model a surface of the second object in the environment so that the first object avoids contacting the second object;

after determining to obtain additional data points for the plurality of data points to model the surface of the second object in the environment;

receiving second data about the environment, the second data comprising the one or more additional data points, and

modeling the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment.