Audio based robot control and navigation

• US 8,532,863 B2
• Filed: 09/28/2010
• Issued: 09/10/2013
• Est. Priority Date: 09/28/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Hadsell, Raia, Samarasekera, Supun, Divakaran, Ajay
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): Sample, Jonathan L
• Application Number:
  US12/892,048
• Publication Number:
  US 20110077813A1
• Time in Patent Office: 1,078 Days
• Field of Search: None
• US Class Current: 701/26
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593

Abstract

A computer implemented method for unattended detection of a current terrain to be traversed by a mobile device is disclosed. Visual input of the current terrain is received for a plurality of positions. Audio input corresponding to the current terrain is received for the plurality of positions. The video input is fused with the audio input using a classifier. The type of the current terrain is classified with the classifier. The classifier may also be employed to predict the type of terrain proximal to the current terrain. The classifier is constructed using an expectation-maximization (EM) method.

Claims

1. A method for unattended detection of a current terrain to be traversed by a mobile device, comprising the steps of:

receiving, from a visual sensor, visual input of the current terrain for a plurality of positions;

receiving audio input corresponding to the current terrain for the plurality of positions;

fusing the video input with the audio input using a classifier; and

classifying the type of the terrain with the classifier.

2. The method of claim 1, wherein the classifier is constructed using an expectation maximization (EM) method.

3. The method of claim 2, wherein the expectation-maximization (EM) method comprises the steps of:

constructing a parameterized video processing function based on the visual input received over the plurality of positions;

constructing a parameterized audio processing function based on the audio input received over the plurality of positions;

holding the video processing function fixed and optimizing the parameterized audio processing function using the EM method to produce an optimized audio classification of the terrain;

holding the audio processing function fixed and optimizing the parameterized video processing function using the EM method to produce an optimized video classification of the terrain; and

associating the optimized audio classification with the optimized video classification to produce ,an optimized fused classification of the current terrain.

4. The method of claim 3, wherein parameters of at least one of the video and audio processing function are obtained using a regression.

5. The method of claim 1, wherein the classifier is employed to predict the type of terrain proximal to the current terrain.

6. The method of claim 5, further comprising the step of the mobile device changing course based on the prediction.

7. The method of claim 1, wherein the classifier is one of a fixed classifier and adaptively trained classifier.

8. The method of claim 1, wherein the classifier is based on one of a Gaussian mixture model and a hidden Markov model.

9. The method of claim 1, wherein the visual input is represented by scale-invariant feature transformation (SIFT) features.

10. The method of claim 1, wherein the video input is represented by a scale-invariant pyramid.

11. The method of claim 1, wherein the audio input is represented by one of a Gaussian mixture model (GMM), a hidden Markov model, and a time-delay neural network (TDNN).

12. A system for unattended detection unattended detection of a current terrain to be traversed by a mobile device, comprising:

at least one image capturing device for receiving visual input of the current terrain for a plurality of positions;

at least one audio capturing device for receiving audio input corresponding to the current terrain for the plurality of positions;

at least one processor communicatively connected to the at least one image capturing device and the at least one audio capturing device for;

fusing the video input with the audio input using a classifier; and

classifying the type of the current terrain with the classifier.

13. The system of claim 12, wherein the classifier is constructed using an expectation-maximization (EM) method.

14. The system of claim 13, wherein the expectation-maximization (EM) method comprises the steps of:

constructing a parameterized video processing function based on the visual input received over the plurality of positions;

constructing a parameterized audio processing function based on the audio input received over the plurality of positions;

holding the video processing function fixed and optimizing the parameterized audio processing function using the EM method to produce an optimized audio classification of the terrain;

holding the audio processing function fixed and optimizing the parameterized video processing function using the EM method to produce an optimized video classification of the terrain; and

associating the optimized audio classification with the optimized video classification to produce an optimized fused classification of the current terrain.

15. The system of claim 14, wherein parameters of at least one of the video and audio processing function are obtained using a regression.

16. The system of claim 12, wherein the classifier is employed to predict the type of terrain proximal to the current terrain.

17. The system of claim 16, further comprising the step of the mobile device changing course based on the prediction.

18. The system of claim 12, wherein the at least one audio capturing device is communicatively connected to at least one wheel of the mobile device.

19. The system of claim 12, wherein the audio capturing device is one of an audio microphone and a:

contact microphone.

20. A non-transitory computer-readable medium comprising computer-executable instructions for unattended detection of a current terrain to be traversed by a mobile device, the instructions configured to:

receive visual input, from a visual sensor, of the current terrain for a plurality of positions;

receive audio input corresponding to the current terrain for the plurality of positions;

fuse the video input with the audio input using a classifier; and

classify the type of the current terrain with the classifier.

21. The computer-readable medium of claim 20, wherein the classifier is constructed using an expectation-maximization (EM) method.

22. The computer-readable medium of claim 21, wherein the expectation maximization (EM) method comprises the steps of:

constructing a parameterized video processing function based on the visual input received over the plurality of positions;

constructing a parameterized audio processing function based on the audio input received over the plurality of positions;

holding the video processing function fixed and optimizing the parameterized audio processing function using the EM method to produce an optimized audio classification of the terrain;

holding the audio processing function fixed and optimizing the parameterized video processing function using the EM method to produce an optimized video classification of the terrain; and

associating the optimized audio classification with the optimized video classification to produce an optimized fused classification of the current terrain.

23. The computer-readable medium of claim 22, wherein parameters of at least one of the video and audio processing function are obtained using a regression.

24. The computer-readable medium of claim 22, wherein the classifier is employed to predict the type of terrain proximal to the current terrain.

25. The computer-readable medium of claim 22, wherein the instructions are further configured cause the mobile device to change course based on the prediction.