Enhancing sensor data by coordinating and/or correlating data attributes

• US 8,874,266 B1
• Filed: 08/01/2012
• Issued: 10/28/2014
• Est. Priority Date: 01/19/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Francis, Anthony G. Jr., Hickman, Ryan, Poursohi, Arshan
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): Nguyen, Bao Long T
• Application Number:
  US13/564,313
• Time in Patent Office: 818 Days
• US Class Current: 700/252
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: G05B 23/0221 Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A method includes receiving first sensor data acquired by a first sensor in communication with a cloud computing system. The first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired. The method also includes receiving second sensor data acquired by a second sensor in communication with the cloud computing system. The second data has a second set of associated attributes including a time and a location at which the second sensor data was acquire. Further, the method includes generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes and instructing a robot in communication with the cloud computing system to perform a task based at least in part on the data processing result.

Claims

1. A method, comprising:

receiving first sensor data acquired by a first sensor that is in communication with the cloud computing system, wherein the first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object;

receiving second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a synchronization of the first sensor data and the second sensor data and further includes a localization of the first sensor data and the second sensor data, wherein the synchronization is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time;

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localization being inconclusive, receiving third sensor data by at least one third sensor that is in communication with the cloud computing system, wherein the third sensor data has a third set of associated attributes including a time and a location at which the third sensor data is acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object;

generating an updated data processing result based at least in part on the first sensor data and associated attributes, the second sensor data and associated attributes, and the third sensor data and associated attributes; and

in response to the updated data processing result being conclusive of the identification of the at least one object, providing instructions to one or more robots that are in communication with the cloud computing system to perform a task associated with the identified at least one object.

2. The method of claim 1, wherein the first image, the second image, and the third image include images of the at least one object from different perspectives.

3. The method of claim 1, wherein the task includes interacting with the identified at least one object.

4. The method of claim 1, wherein one or more of the first audio signal, the second audio signal, and the third audio signal include acoustic signals that represent an audible command, and wherein the instructions are related to executing the audible command.

5. The method of claim 1, further comprising removing noise from at least one of the first sensor data, the second sensor data, and the third sensor data based at least in part on one or more of the first set of associated attributes, the second set of associated attributes, and the third set of associated attributes.

6. The method of claim 1, wherein the one or more robots include a mechanical actuator.

7. The method of claim 1, wherein one or more of the first sensor, the second sensor, and the at least one third sensor are one and only one sensor.

8. The method of claim 1, wherein the one or more robots include at least a first robot and a second robot, wherein the first sensor is associated with the first robot, and wherein the second sensor is associated with the second robot.

9. The method of claim 8, wherein at least one of the first sensor and the second sensor is in wireless communication with the first and second robots, respectively.

10. The method of claim 1, wherein receiving the third sensor data based on the data processing result being inconclusive of the identification of the at least one object comprises receiving the third sensor data based on the data processing result being indicative of a threshold low level of certainty of the identification.

11. The method of claim 1, wherein the localization is based on an identification of a location of the first sensor relative to the at least one object and an identification of a location of the second sensor relative to the at least one object.

12. A cloud computing system, comprising:

a computing device configured to receive a request to perform a task, wherein the computing device is configured to communicate with a cloud computing system to;

receive first sensor data associated with first attributes and acquired by a first sensor, wherein the first attributes include a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object,

receive second sensor data associated with second attributes and acquired by a second sensor, wherein the second attributes include a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals,

process the first attributes and the second attributes to localize and synchronize the first sensor data and the second sensor data and to generate a data processing result representative of the localized and synchronized first sensor data and second sensor data, wherein synchronizing the first sensor data and the second sensor data is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time,

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localized first sensor data and second sensor data being inconclusive, receive third sensor data associated with third attributes and acquired by at least one third sensor, wherein the third attributes include a time and a location at which the third sensor data was acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object,

process the first attributes, the second attributes, and the third attributes to localize and synchronize the first sensor data, the second sensor data, and the third sensor data and to generate an updated data processing result, and

in response to the updated data processing result being conclusive of the identification of the at least one object, responsively provide instructions to perform the task, wherein the task is associated with the identified at least one object.

13. The system of claim 12, wherein the first sensor data, the second sensor data, and the third sensor data include images of the at least one object from different perspectives.

14. The system of claim 12, wherein the first and second sensor data include acoustic signals that represent an audible command, and wherein the computing device is configured to communicate with the cloud computing system to provide instructions to execute the audible command.

15. The system of claim 12, wherein the computing device is configured to communicate with the cloud computing system to process at least one of the first attributes, the second attributes, and the third attributes to remove noise from at least one of the first sensor data, the second sensor data, and the third sensor data.

16. The system of claim 12, wherein the computing device is a server device and the instructions are for a client device to perform the task.

17. The system of claim 16, wherein the client device includes a mechanical actuator.

18. A non-transitory computer readable medium having stored therein instructions executable by at least one processor to cause the at least one processor to perform functions comprising:

accessing first sensor data acquired by a first sensor that is in communication with a cloud computing system, wherein the first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object;

accessing second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a synchronization of the first sensor data and the second sensor data and further includes a localization of the first sensor data and the second sensor data, wherein the synchronization is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time;

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localization being inconclusive, accessing third sensor data acquired by at least one third sensor that is in communication with the cloud computing system, wherein the third sensor data has a third set of associated attributes including a time and a location at which the third sensor data is acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object;

generating an updated data processing result based at least in part on the first sensor data and associated attributes, the second sensor data and associated attributes, and the third sensor data and associated attributes; and

in response to the updated data processing result being conclusive of the identification of the at least one object, providing instructions to one or more robots that are in communication with the cloud computing system to perform a task associated with the identified at least one object.