Environment mapping with automatic motion model selection

• US 9,495,761 B2
• Filed: 11/03/2014
• Issued: 11/15/2016
• Est. Priority Date: 11/04/2013
• Status: Expired due to Fees

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Gauglitz, Steffen, Sweeney, Christopher Michael, Ventura, Jonathan Daniel, Turk, Matthew Alan, Hllerer, Tobias
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US14/531,608
• Publication Number:
  US 20150125045A1
• Time in Patent Office: 743 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion

Abstract

Various embodiments each include at least one of systems, methods, devices, and software for environment mapping with automatic motion model selection. One embodiment in the form of a method includes receiving a video frame captured by a camera device into memory and estimating a type of motion from a previously received video frame held in memory to the received video frame. When the type of motion is the same as motion type of a current keyframe group held in memory, the method includes adding the received video frame to the current keyframe group. Conversely, when the type of motion is not the same motion type of the current keyframe group held in memory, the method includes creating a new keyframe group in memory and adding the received video frame to the new keyframe group.

Claims

1. A method comprising:

receiving a video frame captured by a camera device into memory;

estimating a type of motion relative to a previously received video frame held in memory to the received video frame, the estimating of the type of motion including;

for each of at least two motion types, estimating the respective motion type;

applying a metric to each of the at least two estimated motion types; and

selecting a motion type that represents motion from the previously received video frame to the received video frame;

when the type of motion is the same as motion type of a current keyframe group held in memory, adding the received video frame to the current keyframe group; and

when the type of motion is not the same motion type of the current keyframe group held in memory, creating a new keyframe group in memory and adding the received video frame to the new keyframe group.

2. The method of claim 1, wherein after receipt of the video frame and prior to estimating the type of motion, the method further comprising:

identifying features in the received video frame.

3. The method of claim 2, wherein features are identified according to a corner detector algorithm.

4. The method of claim 1, wherein the at least two motion types are parallax-inducing motion and rotation-only motion.

5. The method of claim 1, wherein the metric applied to each of the at least two estimated motion types is a modified Geometric Robust Information Criterion metric.

6. The method of claim 1, further comprising:

executing a modeling process against keyframe group data of each of one or more keyframe groups held in memory to generate and maintain keyframe group models held in memory, the modeling process including;

triangulating new features of a video frame of a particular keyframe group for which the video frame has not yet been added to a keyframe group model for the particular keyframe group and the new features are not yet present in the keyframe group model of the particular keyframe group; and

processing keyframe group models of disjoint keyframe groups to determine whether two or more keyframe group models can be joined and joining disjoint keyframe group models that can be joined.

7. The method of claim 6, wherein the video frame captured by the camera device into memory is captured by a camera and into a memory of a first computing device, the method further comprising:

transmitting at least a portion of at least one keyframe group model held in memory via a network interface device over a network to at least one second computing device.

8. The method of claim 7, further comprising:

receiving, via the network interface device from the network, annotation data including data identifying a location within an identified keyframe group model held in memory; and

adding the annotation data to the identified keyframe group model, the annotation data presentable within a view presented on a display of the first computing device when the view includes the location identified in the received annotation data.

9. A non-transitory computer readable medium, with instructions stored thereon, which when executed by the at least one processor cause a computing device to perform data processing activities, the data processing activities comprising:

receiving a video frame captured by a camera device into memory;

identifying features in the received video frame;

estimating a type of motion from a previously received video frame held in memory to the received video frame;

when the type of motion is the same as motion type of a current keyframe group held in memory, adding the received video frame to the current keyframe group;

when the type of motion is not the same motion type of the current keyframe group held in memory, creating a new keyframe group in memory and adding the received video frame to the new keyframe group; and

executing a modeling process against keyframe group data of each of one or more keyframe groups held in memory to generate and maintain keyframe group models held in memory, the modeling process including;

triangulating new features of a video frame of a particular keyframe group for which the video frame has not yet been added to a keyframe group model for the particular keyframe group and the new features are not yet present in the keyframe group model of the particular keyframe group; and

processing keyframe group models of disjoint keyframe groups to determine whether two or more keyframe group models can be joined and joining disjoint keyframe group models that can be joined.

10. The non-transitory computer readable medium of claim 9, wherein features are identified according to a corner detector algorithm.

11. The non-transitory computer readable medium of claim 9, wherein estimating the type of motion from the previously received video frame to the received video frame includes:

for each of at least two motion types, estimating the respective motion type;

applying a metric to each of the at least two estimated motion types; and

selecting a motion type that represents motion from the previously received video frame to the received video frame.

12. The non-transitory computer readable medium of claim 11, wherein the at least two motion types are parallax-inducing motion and rotation-only motion.

13. The non-transitory computer readable medium of claim 11, wherein the metric applied to each of the at least two estimated motion types is a modified Geometric Robust Information Criterion metric.

14. The non-transitory computer readable medium of claim 9, wherein the video frame captured by the camera device into memory is captured by a camera of and into a memory of a first computing device, the data processing activities further comprising:

transmitting at least a portion of at least one keyframe group model held in memory via a network interface device over a network to at least one second computing device;

receiving, via the network interface device from the network, annotation data including data identifying a location within an identified keyframe group model held in memory; and

adding the annotation data to the identified keyframe group model, the annotation data presentable within a view presented on a display of the first computing device when the view includes the location identified in the received annotation data.

15. A device comprising:

at least one processor;

at least one memory; and

an instruction set, stored in the at least one memory and executable by the at least one processor to perform data processing activities, the data processing activities comprising;

executing a tracking process including;

receiving a video frame into the at least one memory;

estimating a type of motion from a previously received video frame held in the at least one memory to the received video frame, the estimating including;

for each of at least parallax-inducing motion and rotation-only motion types, estimating the respective motion type;

applying a modified Geometric Robust Information Criterion metric to each of the estimated motion types; and

selecting a motion type that represents motion from the previously received video frame to the received video frame;

when the type of motion is the same as motion type of a current keyframe group held in the at least one memory, adding the received video frame to the current keyframe group; and

when the type of motion is not the same motion type of the current keyframe group, creating a new keyframe group in the at least one memory and adding the received video frame to the new keyframe group.

16. The device of claim 15, wherein after receipt of the video frame and prior to estimating the type of motion, the data processing activities further comprising:

identifying features in the received video frame according to a corner detector algorithm.

17. The device of claim 15, the data processing activities further comprising:

executing a modeling process in one or more processing threads distinct from one or more processing thread in which the tracking process is executed, the modeling process executed against keyframe group data of each of one or more keyframe groups held in the at least one memory to generate and maintain keyframe group models held in the at least one memory, the modeling process including;

triangulating new features of a video frame of a particular keyframe group for which the video frame has not yet been added to a keyframe group model for the particular keyframe group and the new features are not yet present in the keyframe group model of the particular keyframe group; and

processing keyframe group models of disjoint keyframe groups to determine whether two or more keyframe group models can be joined and joining disjoint keyframe group models that can be joined.