Systems and Methods of Merging Multiple Maps for Computer Vision Based Tracking

• US 20140119598A1
• Filed: 03/14/2013
• Published: 05/01/2014
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ramachandran, Mahesh, Tyagi, Abhishek, Diaz Spindola, Serafin
• Granted Patent: US 9,177,404 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3833 characterised by the source of data
• CPC Class Codes: G01C 21/3863 Structures of map data
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

Method, apparatus, and computer program product for merging multiple maps for computer vision based tracking are disclosed. In one embodiment, a method of merging multiple maps for computer vision based tracking comprises receiving a plurality of maps of a scene in a venue from at least one mobile device, identifying multiple keyframes of the plurality of maps of the scene, and merging the multiple keyframes to generate a global map of the scene.

Claims

1. A method of merging multiple maps for computer vision based tracking, comprising:

receiving a plurality of maps of a scene in a venue from at least one mobile device;

identifying multiple keyframes of the plurality of maps of the scene; and

merging the multiple keyframes to generate a global map of the scene.

2. The method of claim 1, wherein receiving a plurality of maps of a scene in a venue comprises:

receiving the plurality of maps of the scene from multiple mobile devices.

3. The method of claim 2 further comprises:

evaluating the multiple keyframes from the plurality of maps of the scene from multiple mobile devices for redundancy; and

eliminating redundant keyframes.

4. The method of claim 2, further comprising:

generating a simultaneous localization and mapping (SLAM) map using the plurality of maps of the scene from multiple mobile devices; and

sharing the SLAM map among the multiple mobile devices.

5. The method of claim 1, wherein identifying the multiple keyframes of the plurality of maps of the scene comprises at least one of:

identifying the multiple keyframes representing multiple viewpoints of the scene from multiple angles; and

identifying the multiple keyframes representing multiple viewpoints of the scene with multiple scales;

wherein the multiple keyframes include common features, location coordinates, and appearances of the plurality of maps.

6. The method of claim 5, wherein multiple scales comprise at least one of:

multiple distances between the at least one mobile device and the scene; and

multiple aspect ratios of the scene from the at least one mobile device.

7. The method of claim 1, wherein merging the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

determining a contribution score based at least in part on angular diversity of the keyframe and scale diversity of the keyframe, and

eliminating the keyframe in response to the contribution score being below a predetermined threshold.

8. The method of claim 7, wherein determining a contribution score based at least in part on angular diversity and scale diversity of the keyframe comprises:

computing an angular diversity scoring metric by applying a first weight factor to the angular diversity of the keyframe;

computing a scale diversity scoring metric by applying a second weight factor to the scale diversity of the keyframe; and

combining the angular diversity scoring metric and the scale diversity scoring metric to generate the contribution score.

9. The method of claim 1, wherein merging the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

determining a contribution score based at least in part on angular diversity of the keyframe; and

eliminating the keyframe in response to the contribution score being below a predetermined threshold.

10. The method of claim 1, wherein merging the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

determining a contribution score based at least in part on scale diversity of the keyframe; and

eliminating the keyframe in response to the contribution score being below a predetermined threshold.

11. The method of claim 1, wherein merging the multiple keyframes to generate a global map of the scene comprises:

tallying the multiple keyframes in an angular histogram, wherein the angular histogram includes a plurality of bins representing different viewing angles of the scene, and each bin in the plurality of bins includes a plurality of sections representing different viewing scales of the scene;

for each keyframe of the multiple keyframes in the angular histogram,

determining a contribution score based at least in part on angular diversity of the keyframe and scale diversity of the keyframe, and

eliminating the keyframe in response to the contribution score being below a predetermined threshold.

12. The method of claim 11, wherein determining a contribution score based at least in part on angular diversity and scale diversity of the keyframe comprises:

computing an angular diversity scoring metric by applying a first weight factor to the angular diversity of the keyframe, wherein the first weight factor is based at least in part on number of keyframes in a bin the keyframe is in;

computing a scale diversity scoring metric by applying a second weight factor to the scale diversity of the keyframe, wherein the second weight factor is based at least in part on number of keyframes in a section the keyframe is in; and

combining the angular diversity scoring metric and the scale diversity scoring metric to generate the contribution score.

13. The method of claim 1 further comprises:

tracking location of the at least one mobile device in the venue.

14. An apparatus, comprising:

logic configured to receive a plurality of maps of a scene in a venue from at least one mobile device;

logic configured to identify multiple keyframes of the plurality of maps of the scene; and

logic configured to merge the multiple keyframes to generate a global map of the scene.

15. The apparatus of claim 14, wherein logic configured to receive a plurality of maps of a scene in a venue comprises:

logic configured to receive the plurality of maps of the scene from multiple mobile devices.

16. The apparatus of claim 15 further comprises:

logic configured to evaluate the multiple keyframes from the plurality of maps of the scene from multiple mobile devices for redundancy; and

logic configured to eliminate redundant keyframes.

17. The apparatus of claim 15, further comprising:

logic configured to generate a simultaneous localization and mapping (SLAM) map using the plurality of maps of the scene from multiple mobile devices; and

logic configured to share the SLAM map among the multiple mobile devices.

18. The apparatus of claim 14, wherein logic configured to identify the multiple keyframes of the plurality of maps of the scene comprises at least one of:

logic configured to identify the multiple keyframes representing multiple viewpoints of the scene from multiple angles; and

logic configured to identify the multiple keyframes representing multiple viewpoints of the scene with multiple scales;

wherein the multiple keyframes include common features, location coordinates, and appearances of the plurality of maps.

19. The apparatus of claim 18, wherein multiple scales comprise at least one of:

multiple distances between the at least one mobile device and the scene; and

multiple aspect ratios of the scene from the at least one mobile device.

20. The apparatus of claim 14, wherein logic configured to merge the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

logic configured to determine a contribution score based at least in part on angular diversity of the keyframe and scale diversity of the keyframe, and

logic configured to eliminate the keyframe in response to the contribution score being below a predetermined threshold.

21. The apparatus of claim 20, wherein logic configured to determine a contribution score based at least in part on angular diversity and scale diversity of the keyframe comprises:

logic configured to compute an angular diversity scoring metric by applying a first weight factor to the angular diversity of the keyframe;

logic configured to compute a scale diversity scoring metric by applying a second weight factor to the scale diversity of the keyframe; and

logic configured to combine the angular diversity scoring metric and the scale diversity scoring metric to generate the contribution score.

22. The apparatus of claim 14, wherein logic configured to merge the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

logic configured to determine a contribution score based at least in part on angular diversity of the keyframe; and

logic configured to eliminate the keyframe in response to the contribution score being below a predetermined threshold.

23. The apparatus of claim 14, wherein logic configured to merge the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

logic configured to determine a contribution score based at least in part on scale diversity of the keyframe; and

logic configured to eliminate the keyframe in response to the contribution score being below a predetermined threshold.

24. The apparatus of claim 14, wherein logic configured to merge the multiple keyframes to generate a global map of the scene comprises:

logic configured to tally the multiple keyframes in an angular histogram, wherein the angular histogram includes a plurality of bins representing different viewing angles of the scene, and each bin in the plurality of bins includes a plurality of sections representing different viewing scales of the scene;

for each keyframe of the multiple keyframes in the angular histogram,

logic configured to determine a contribution score based at least in part on angular diversity of the keyframe and scale diversity of the keyframe, and

logic configured to eliminate the keyframe in response to the contribution score being below a predetermined threshold.

25. The apparatus of claim 24, wherein logic configured to determine a contribution score based at least in part on angular diversity and scale diversity of the keyframe comprises:

logic configured to compute an angular diversity scoring metric by applying a first weight factor to the angular diversity of the keyframe, wherein the first weight factor is based at least in part on number of keyframes in a bin the keyframe is in;

logic configured to compute a scale diversity scoring metric by applying a second weight factor to the scale diversity of the keyframe, wherein the second weight factor is based at least in part on number of keyframes in a section the keyframe is in; and

logic configured to combine the angular diversity scoring metric and the scale diversity scoring metric to generate the contribution score.

26. The apparatus of claim 14 further comprises:

logic configured to track location of the at least one mobile device in the venue.

27. A computer program product comprising a non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for receiving a plurality of maps of a scene in a venue from at least one mobile device;

instructions for identifying multiple keyframes of the plurality of maps of the scene; and

instructions for merging the multiple keyframes to generate a global map of the scene.

28. The computer program product of claim 27, wherein instructions for receiving a plurality of maps of a scene in a venue comprises:

instructions for receiving the plurality of maps of the scene from multiple mobile devices.

29. The computer program product of claim 28 further comprises:

instructions for evaluating the multiple keyframes from the plurality of maps of the scene from multiple mobile devices for redundancy; and

instructions for eliminating redundant keyframes.

30. The computer program product of claim 28 further comprising:

instructions for generating a simultaneous localization and mapping (SLAM) map using the plurality of maps of the scene from multiple mobile devices; and

instructions for sharing the SLAM map among the multiple mobile devices.

31. The computer program product of claim 27, wherein instructions for identifying the multiple keyframes of the plurality of maps of the scene comprises at least one of:

instructions for identifying the multiple keyframes representing multiple viewpoints of the scene from multiple angles; and

instructions for identifying the multiple keyframes representing multiple viewpoints of the scene with multiple scales;

wherein the multiple keyframes include common features, location coordinates, and appearances of the plurality of maps.

32. The computer program product of claim 27, wherein multiple scales comprise at least one of:

multiple distances between the at least one mobile device and the scene; and

multiple aspect ratios of the scene from the at least one mobile device.

33. The computer program product of claim 27, wherein instructions for merging the multiple keyframes to generate a global map of the scene comprises:

for each keyframe of the multiple keyframes,

instructions for determining a contribution score based at least in part on angular diversity of the keyframe and scale diversity of the keyframe, and

instructions for eliminating the keyframe in response to the contribution score being below a predetermined threshold.

34. The computer program product of claim 33, wherein instructions for determining a contribution score based at least in part on angular diversity and scale diversity of the keyframe comprises:

instructions for computing an angular diversity scoring metric by applying a first weight factor to the angular diversity of the keyframe;

instructions for computing a scale diversity scoring metric by applying a second weight factor to the scale diversity of the keyframe; and

instructions for combining the angular diversity scoring metric and the scale diversity scoring metric to generate the contribution score.

35. The computer program product of claim 27 further comprises:

instructions for tracking location of the at least one mobile device in the venue.

36. A system, comprising:

means for receiving a plurality of maps of a scene in a venue from at least one mobile device;

means for identifying multiple keyframes of the plurality of maps of the scene; and

means for merging the multiple keyframes to generate a global map of the scene.