Context aware localization, mapping, and tracking

• US 9,367,811 B2
• Filed: 03/15/2013
• Issued: 06/14/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ramachandran, Mahesh, Swaminathan, Ashwin, Sadasivam, Shankar
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US13/842,928
• Publication Number:
  US 20140279790A1
• Time in Patent Office: 1,187 Days
• Field of Search: 706/46
• US Class Current: 1/1
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

Exemplary methods, apparatuses, and systems infer a context of a user or device. A computer vision parameter is configured according to the inferred context. Performing a computer vision task, in accordance with the configured computer vision parameter. The computer vision task may by at least one of: a visual mapping of an environment of the device, a visual localization of the device or an object within the environment of the device, or a visual tracking of the device within the environment of the device.

Claims

1. A machine-implemented method for performing a computer vision task, the method comprising:

inferring a context of a user or device, wherein the context comprises one or more of motion, activity, environment, or location, or any combination thereof;

configuring a computer vision task according to the inferred context; and

performing the computer vision task, wherein the computer vision task comprises at least one of;

mapping of an environment of the device,

localizing of the device or an object within the environment of the device,

tracking of the device within the environment of the device, or

any combination thereof.

2. The machine-implemented method of claim 1, wherein the inferring a context comprises reading sensor data to infer the context of the user or the device, wherein the sensor data is from one or more of:

Bluetooth sensor, microphone, camera(s), global positioning sensor (GPS), WiFi, proximity sensor, temperature sensor, barometric (i.e., pressure) sensor, ambient light sensor (ALS), red-green-blue (RGB) color sensor, ultra-violet (UV) sensor, UV-A sensor, UV-B sensor, fingerprint sensor, touch sensor, accelerometer, gyro, compass, magnetometer, or any combination thereof.

3. The machine-implemented method of claim 1, wherein the inferring a context comprises reading application data to infer the context of the user or the device, wherein the application data is from one or more of:

calendar, geo tagging, social media data, battery, time of day, or any combination thereof.

4. The machine-implemented method of claim 1, wherein the configuring the computer vision task includes one or more of:

selecting parameters of an algorithm associated with the task;

selecting a prior model as inputs to the algorithm associated with the task;

selecting an approach to solve the problem based on environment constants;

or any combination thereof.

5. The machine-implemented method of claim 4, wherein the configuring the computer vision task includes one or more of:

using edge based methods;

using point based methods;

using rectangle based methods;

or

any combination thereof.

6. The machine-implemented method of claim 1, wherein the context is an indoor context and the computer vision task uses information associated with the indoor context to modify the computer vision task configuration for use with one or more of:

an aligned rectangle coordinate system, bounded interior scene dimensions, predetermined objects, indoor structural features, or any combination thereof.

7. The machine-implemented method of claim 1, wherein the context is an outdoor context and the computer vision task uses information associated with the outdoor context to modify the computer vision task configuration for use with one or more of:

dynamic world modeling, predetermined outdoor structural features, distant features, a panoramic model or tracker, images from a ground facing camera, or any combination thereof.

8. The machine-implemented method of claim 1, wherein the context is a cluttered context and the computer vision task uses information associated with the cluttered context to modify the computer vision task configuration for use with one or more of:

feature point calculations, increasing a keyframe rate, or any combination thereof.

9. The machine-implemented method of claim 1, wherein the context is an uncluttered context and the computer vision task uses information associated with the uncluttered context to modify the computer vision task for use with one or more of:

decreasing a keyframe rate,

using one or more features of;

lines, vanishing points, rectangles, or any combination thereof, or

any combination thereof.

10. The machine-implemented method of claim 1, wherein when the context is an excited context, the computer vision task increases a number of tracked features, and wherein when the context is an unexcited context the computer vision task decreases the number of tracked features.

11. The machine-implemented method of claim 1, wherein the context is a dynamic context and the computer vision task uses information associated with the dynamic context to modify the computer vision task configuration for use with one or more of:

delaying performing the computer vision task until the context is determined to be static,

selecting one of a plurality of camera sensors to capture a static portion of the dynamic environment and performing the computer vision task using the selected camera sensor, or

any combination thereof.

12. The machine-implemented method of claim 1, wherein the context is a combination of two or more of:

a location context, an environment context, an activity context, a motion context, or any combination thereof.

13. The machine-implemented method of claim 1, further comprising:

providing, for the computer vision task, a suggested camera direction or viewpoint based on the inferred context.

14. A machine readable non-transitory storage medium containing executable program instructions which cause a data processing device to perform a method for performing a computer vision task, the method comprising:

inferring a context of a user or device, wherein the context comprises one or more of;

motion, activity, environment, location, or any combination thereof;

configuring a computer vision task according to the inferred context; and

performing the computer vision task, wherein the computer vision task comprises at least one of;

mapping of an environment of the device,

localizing of the device or an object within the environment of the device,

tracking of the device within the environment of the device, or any combination thereof.

15. The medium of claim 14, wherein the inferring a context comprises reading sensor data to infer the context of the user or the device, wherein the sensor data is from one or more of:

Bluetooth sensor, microphone, camera(s), global positioning sensor (GPS), WiFi, proximity sensor, temperature sensor, barometric (i.e., pressure) sensor, ambient light sensor (ALS), red-green-blue (RGB) color sensor, ultra-violet (UV) sensor, UV-A sensor, UV-B sensor, fingerprint sensor, touch sensor, accelerometer, gyro, compass, magnetometer, or any combination thereof.

16. The medium of claim 14, wherein the inferring a context comprises reading application data to infer the context of the user or the device, wherein the application data is from one or more of:

calendar, geo tagging, social media data, battery, time of day, or any combination thereof.

17. The medium of claim 14, wherein the context comprises one or more of:

motion, activity, environment, location, or any combination thereof.

18. The medium of claim 14, wherein the configuring the computer vision task includes one or more of:

selecting parameters of an algorithm associated with the task;

selecting a prior model as inputs to the algorithm associated with the task;

selecting an approach to solve the problem based on environment constants;

or

any combination thereof.

19. The medium of claim 18, wherein the configuring the computer vision task includes one or more of:

using edge based methods;

using point based methods;

using rectangle based methods;

or

any combination thereof.

20. The medium of claim 14, wherein the context is an indoor context and the computer vision task uses information associated with the indoor context to modify the computer vision task configuration for use with one or more of:

an aligned rectangle coordinate system, bounded interior scene dimensions, predetermined objects, indoor structural features, or any combination thereof.

21. The medium of claim 14, wherein the context is an outdoor context and the computer vision task uses information associated with the outdoor context to modify the computer vision task configuration for use with one or more of:

dynamic world modeling, predetermined outdoor structural features, distant features, a panoramic model or tracker, images from a ground facing camera, or any combination thereof.

22. The medium of claim 14, wherein the context is a cluttered context and the computer vision task uses information associated with the cluttered context to modify the computer vision task configuration for use with one or more of:

feature point calculations, increasing a keyframe rate, or any combination thereof.

23. The medium of claim 14, wherein the context is an uncluttered context and the computer vision task uses information associated with the uncluttered context to modify the computer vision task configuration for use with one or more of:

decreasing a keyframe rate,

using one or more features of;

lines, vanishing points, rectangles, or any combination thereof, or

any combination thereof.

24. The medium of claim 14, wherein when the context is an excited context the computer vision task increases a number of tracked features, and wherein when the context is an unexcited context the computer vision task decreases the number of tracked features.

25. The medium of claim 14, wherein the context is a dynamic context and the computer vision task uses information associated with the dynamic context to modify the computer vision task configuration for use with one or more of:

delaying performing computer the vision task until the context is determined to be static,

selecting one of a plurality of camera sensors to capture a static portion of the dynamic environment and performing the computer vision task using the selected camera sensor, or

any combination thereof.

26. The medium of claim 14, wherein the context is a combination of two or more of:

a location context, an environment context, an activity context, a motion context, or any combination thereof.

27. The medium of claim 14, further comprising:

providing, for the computer vision task, a suggested camera direction or viewpoint based on the inferred context.

28. A system for performing a computer vision task, the system comprising:

a processor; and

a storage device coupled to the processor and configurable for storing instructions, which, when executed by the processor cause the processor to;

infer a context of a user or device, wherein the context comprises one or more of;

motion, activity, environment, location, or any combination thereof;

configure a computer vision task according to the inferred context; and

perform the computer vision task, wherein the computer vision task is configured to;

map an environment of the device,

localize the device or an object within the environment of the device, or

track the device within the environment of the device.

29. The system of claim 28, wherein the inferring a context comprises reading sensor data to infer the context of the user or the device, wherein the sensor data is from one or more of:

Bluetooth sensor, microphone, camera(s), global positioning sensor (GPS), WiFi, proximity sensor, temperature sensor, barometric (i.e., pressure) sensor, ambient light sensor (ALS), red-green-blue (RGB) color sensor, ultra-violet (UV) sensor, UV-A sensor, UV-B sensor, fingerprint sensor, touch sensor, accelerometer, gyro, compass, magnetometer, or any combination thereof.

30. The system of claim 28, wherein the inferring a context comprises reading application data to infer the context of the user or the device, wherein the application data is from one or more of:

calendar, geo tagging, social media data, battery, time of day, or any combination thereof.

31. The system of claim 28, wherein the configuring the computer vision task comprises the processor executing one or more of instructions to:

select parameters of an algorithm associated with the task;

select a prior model as inputs to the algorithm associated with the task;

select an approach to solve the problem based on environment constants;

or

any combination thereof.

32. The system of claim 31, wherein the configuring the computer vision task comprises the processor executing one or more of instructions to:

use edge based methods;

use point based methods;

use rectangle based methods;

or

any combination thereof.

33. The system of claim 28, wherein the context is an indoor context and the computer vision task uses information associated with the indoor context to modify the computer vision task configuration for use with one or more of:

an aligned rectangle coordinate system, bounded interior scene dimensions, predetermined objects, indoor structural features, or any combination thereof.

34. The system of claim 28, wherein the context is an outdoor context and the computer vision task uses information associated with the outdoor context to modify the computer vision task configuration for use with one or more of:

dynamic world modeling, predetermined outdoor structural features, distant features, a panoramic model or tracker, images from a ground facing camera, or any combination thereof.

35. The system of claim 28, wherein the context is a cluttered context and the computer vision task uses information associated with the cluttered context to modify the computer vision task configuration for use with one or more of:

feature point calculations, increasing a keyframe rate, or any combination thereof.

36. The system of claim 28, wherein the context is an uncluttered context and the computer vision task uses information associated with the uncluttered context to modify the computer vision task configuration for use with one or more of instructions to:

decrease a keyframe rate,

use one or more features of;

lines, vanishing points, rectangles, or any combination thereof, or

any combination thereof.

37. The system of claim 28, wherein when the context is an excited context the computer vision task increases a number of tracked features, and wherein when the context is an unexcited context the computer vision task decreases the number of tracked features.

38. The system of claim 28, wherein the context is a dynamic context and the computer vision task uses information associated with the dynamic context to modify the computer vision task configuration for use with one or more of instructions to:

delay performing the computer vision task until the context is determined to be static,

select one of a plurality of camera sensors to capture a static portion of the dynamic environment and performing the computer vision task using the selected camera sensor, or

any combination thereof.

39. The system of claim 28, wherein the context is a combination of two or more of:

a location context, an environment context, an activity context, a motion context, or any combination thereof.

40. The system of claim 28, further comprising instructions to:

provide, for the computer vision task, a suggested camera direction or viewpoint based on the inferred context.

41. An apparatus for performing a computer vision task, the apparatus comprising:

means for inferring a context of a user or device, wherein the context comprises one or more of;

motion, activity, environment, location, or any combination thereof;

means for configuring a computer vision task according to the inferred context; and

means for performing the computer vision task, wherein the computer vision task comprises at least one of;

a mapping of an environment of the device,

a localizing the device or an object within the environment of the device,

a tracking of the device within the environment of the device, or

any combination thereof.

42. The apparatus of claim 41, wherein the means for inferring a context comprises means for reading sensor data to infer the context of the user or the device, wherein the sensor data is from one or more of:

Bluetooth sensor, microphone, camera(s), global positioning sensor (GPS), WiFi, proximity sensor, temperature sensor, barometric (i.e., pressure) sensor, ambient light sensor (ALS), red-green-blue (RGB) color sensor, ultra-violet (UV) sensor, UV-A sensor, UV-B sensor, fingerprint sensor, touch sensor, accelerometer, gyro, compass, magnetometer, or any combination thereof.

43. The apparatus of claim 41, wherein the means for inferring a context comprises means for reading application data to infer the context of the user or the device, wherein the application data is from one or more of:

calendar, geo tagging, social media data, battery, time of day, or any combination thereof.

44. The apparatus of claim 41, wherein the means for configuring the computer vision task includes one or more of:

means for selecting parameters of an algorithm associated with the task;

means for selecting a prior model as inputs to the algorithm associated with the task;

means for selecting an approach to solve the problem based on environment constants, or

any combination thereof.

45. The apparatus of claim 44, wherein the means for configuring the computer vision task further includes one or more of:

means for using edge based methods;

means for using point based methods;

means for using rectangle based methods;

or

any combination thereof.

46. The apparatus of claim 41, wherein the context is an indoor context and the computer vision task uses information associated with the indoor context to modify the computer vision task configuration for use with one or more of:

an aligned rectangle coordinate system, bounded interior scene dimensions, predetermined objects, indoor structural features, or any combination thereof.

47. The apparatus of claim 41, wherein the context is an outdoor context and the computer vision task uses information associated with the outdoor context to modify the computer vision task configuration for use with one or more of:

dynamic world modeling, predetermined outdoor structural features, distant features, a panoramic model or tracker, images from a ground facing camera, or any combination thereof.

48. The apparatus of claim 41, wherein the context is a cluttered context and the computer vision task uses information associated with the cluttered context to modify the computer vision task configuration for use with one or more of:

feature point calculations, increasing a keyframe rate, or any combination thereof.

49. The apparatus of claim 41, wherein the context is an uncluttered context and the computer vision task uses information associated with the uncluttered context to modify the computer vision task configuration for use with one or more of:

means for decreasing a keyframe rate, or

means for using one or more features of;

lines, vanishing points, rectangles, or any combination thereof, or

any combination thereof.

50. The apparatus of claim 41, wherein when the context is an excited context the computer vision task increases a number of tracked features, and wherein when the context is an unexcited context the computer vision task decreases the number of tracked features.

51. The apparatus of claim 41, wherein the context is a dynamic context and the computer vision task uses information associated with the dynamic context to modify the computer vision task configuration for use with one or more of:

means for delaying performing the computer vision task until the context is determined to be static,

means for selecting one of a plurality of camera sensors to capture a static portion of the dynamic environment and performing the computer vision task using the selected camera sensor, or

any combination thereof.

52. The apparatus of claim 41, wherein the context is a combination of two or more of:

a location context, an environment context, an activity context, a motion context, or any combination thereof.

53. The apparatus of claim 41, further comprising:

means for providing, for the computer vision task, a suggested camera direction or viewpoint based on the inferred context.

54. A machine-implemented method for performing a computer vision task, the method comprising:

inferring a context of a user or device, wherein the inferring a context comprises reading sensor data to infer the context of the user or the device, wherein the sensor data is from one or more of;

Bluetooth sensor, microphone, camera(s), global positioning sensor (GPS), WiFi, proximity sensor, temperature sensor, barometric (i.e., pressure) sensor, ambient light sensor (ALS), red-green-blue (RGB) color sensor, ultra-violet (UV) sensor, UV-A sensor, UV-B sensor, fingerprint sensor, touch sensor, accelerometer, gyro, compass, magnetometer, or any combination thereof;

configuring a computer vision task according to the inferred context; and

performing the computer vision task, wherein the computer vision task further comprises at least one of;

mapping of an environment of the device,

localizing of the device or an object within the environment of the device,

tracking of the device within the environment of the device, or

any combination thereof.

55. A machine-implemented method for performing a computer vision task, the method comprising:

inferring a context of a user or device, wherein the inferring a context comprises reading application data to infer the context of the user or the device, wherein the application data is from one or more of;

calendar, geo tagging, social media data, battery, time of day, or any combination thereof;

configuring a computer vision task according to the inferred context; and

performing the computer vision task, wherein the computer vision task further comprises at least one of;

mapping of an environment of the device,

localizing of the device or an object within the environment of the device,

tracking of the device within the environment of the device, or

any combination thereof.