Integration of Auxiliary Sensors with Point Cloud-Based Haptic Rendering and Virtual Fixtures

• US 20180232052A1
• Filed: 09/11/2015
• Published: 08/16/2018
• Est. Priority Date: 09/12/2014
• Status: Active Grant

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: CHIZECK, Howard Jay, HUANG, Kevin, RYDEN, Fredrik, STEWART, Andrew
• Granted Patent: US 10,394,327 B2
• Time in Patent Office: Days
• CPC Class Codes: B23K 9/0953 using computing means
• CPC Class Codes: G05B 2219/40 Robotics, robotics mapping to robotics vision
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

Apparatus and methods for generating virtual environment displays based on a group of sensors are provided. A computing device can receive first data about an environment from a first group of one or more sensors. The computing device can model the environment as a virtual environment based on the first data. The computing device can determine whether to obtain additional data to model the environment. After determining to obtain additional data to model the environment, the computing device can: receive second data about the environment, and model the environment as the virtual environment based on at least the second data. The computing device can generate a display of the virtual environment.

Claims

1. A method, comprising:

receiving first data about an environment at a computing device from a first group of one or more sensors;

modeling the environment as a virtual environment using the computing device, wherein the virtual environment is based on the first data;

determining whether to obtain additional data to model the environment using the computing device;

after determining to obtain additional data to model the environment, using the computing device to;

receive second data about the environment, and

model the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment using the computing device.

2. The method of claim 1, wherein determining second data about the environment comprises:

determining whether a particular sensor of the first group of one or more sensors is configured to provide at least part of the additional data; and

after determining that the particular sensor is configured to provide the at least part of the additional data, obtaining at least part of the second data using the particular sensor.

3. The method of claim 2, wherein determining whether the particular sensor of the first group of one or more sensors is configured to provide at least part of the additional data comprises determining whether the particular sensor is configured to measure a particular property of the environment.

4. The method of claim 2, wherein determining whether the particular sensor of the first group of one or more sensors is configured to provide at least part of the additional data comprises determining whether the particular sensor is configured to provide at least part of the additional data based on a range of the particular sensor.

5. The method of claim 1, wherein receiving the second data comprises:

determining whether a first sensor of the first group of one or more sensors interferes with a second sensor of the first group of one or more sensors; and

after determining that the first sensor interferes with the second sensor;

deactivating the first sensor and/or the second sensor, and

after deactivating the first sensor and/or the second sensor, obtaining the second data from activated sensors of the first group of one or more sensors.

6. The method of claim 1, wherein the first data comprises first depth data,

wherein modeling the environment as the virtual environment comprises modeling the environment using a first plurality of points that are obtained from the first depth data,

wherein determining whether to obtain additional data to model the environment comprises determining whether the first plurality of points is sufficient to model the environment,

wherein the second data comprises second depth data, and

wherein modeling the environment as the virtual environment based on at least the second data comprises modeling the environment using at least a second plurality of points that are obtained from the second depth data.

7. The method of claim 6, wherein receiving the second data comprises:

requesting the second depth data from a second group of one or more sensors, wherein the second group of one or more sensors differs from the first group of one or more sensors;

generating an additional plurality of points from the second depth data; and

merging the first plurality of points and the additional plurality of points into the second plurality of points.

8. The method of claim 6, wherein the environment comprises a first object and a second object, and wherein determining whether the first plurality of points is sufficient to model the environment comprises:

determining an object distance in the environment between the first object and the second object; and

determining whether the first plurality of points is sufficient to model the environment based on the object distance.

9. The method of claim 8, wherein the first object is associated with a plurality of sensors, and wherein requesting second depth data from the second group of one or more sensors comprises selecting the second group of one or more sensors from the plurality of sensors based on the object distance.

10. The method of claim 8, further comprising:

determining that the first object moved in the environment with respect to the second object;

determining a new object distance between the first object and the second object; and

determining whether the second plurality of points is sufficient to model the environment based on the new object distance.

11. The method of claim 8, wherein the first object comprises a robotic device controlled by the computing device.

12. The method of claim 8, wherein the virtual environment comprises a first virtual object corresponding to the first object and a second virtual object corresponding to the second object, and wherein determining whether the first plurality of points is sufficient to model the environment comprises:

determining a virtual-object distance in the virtual environment between the first virtual object and the second; and

determining whether the first plurality of points is sufficient to model the environment based on the virtual-object distance.

13. The method of claim 12, wherein the first virtual object comprises a virtual haptic fixture.

14. The method of claim 12, wherein the first virtual object is associated with a plurality of sensors, and wherein requesting second depth data from the second group of one or more sensors comprises selecting the second group of one or more sensors from the plurality of sensors based on the virtual-object distance.

15. The method of claim 12, further comprising:

determining that the first virtual object moved in the environment with respect to the second virtual object;

determining a new virtual-object distance between the first virtual object and the second virtual object; and

determining whether the second plurality of points is sufficient to model the environment based on the new virtual-object distance.

16. The method of claim 12, wherein the first virtual object represents a robotic device controlled by the computing device.

17. The method of claim 1, further comprising:

performing a task in the environment using one or more commands provided by the computing device, wherein the one or more commands are generated based on haptic feedback provided for at least one virtual haptic fixture in the virtual environment.

18. The method of claim 17, wherein the at least one virtual haptic feature comprises a multi-sensor dynamic virtual fixture (MSDVF).

19. The method of claim 18, wherein the task comprises a first portion of the task and a second portion of the task, and wherein performing the task comprises:

performing the first portion of the task utilizing the MSDVF;

at least after the first portion of the task was performed, obtaining first-portion data from a first plurality of sensors;

updating the MSDVF based on the first-portion data; and

after updating the MSDVF, performing the second portion of the task utilizing the updated MSDVF.

20. The method of claim 19, wherein updating the MSDVF comprises one or more of:

modifying a geometry of the MSDVF based on the first-portion data;

changing at least a portion of the MSDVF from being a guidance fixture to a forbidden-region fixture based on the first-portion data; and

changing at least a portion of the MSDVF from being a forbidden-region fixture to a guidance fixture based on the first-portion data.

21. A computing device, comprising:

a processor; and

data storage, storing instructions that, upon execution by the processor, cause the computing device to perform functions comprising;

receiving first data about an environment from a first group of one or more sensors;

modeling the environment as a virtual environment based on the first data;

determining whether to obtain additional data to model the environment;

after determining to obtain additional data to model the environment;

receiving second data about the environment, and

modeling the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment.

22. The computing device of claim 21, wherein the first group of sensors comprises at least one of:

a red-green-blue-depth (RGB-D) camera, a video and/or still camera, a seashell sensor, a proximity sensor, a radiation sensor, an infra-red sensor, a thermometer, a pH sensor, a chemical sensor, a neural signal sensor, a SONAR device, a RADAR device, a LIDAR device, a laser, a position sensor, an electrical sensor, and a biological sensor.

23. An article of manufacture, comprising a tangible computer-readable storage medium storing instructions that, upon execution by a processor of the article of manufacture, cause the article of manufacture to perform functions comprising:

receiving first data about an environment from a first group of one or more sensors;

modeling the environment as a virtual environment based on the first data;

determining whether to obtain additional data to model the environment;

after determining to obtain additional data to model the environment;

receiving second data about the environment, and

modeling the environment as the virtual environment based on at least the second data; and

generating a display of the virtual environment.