METHOD AND APPARATUS FOR DEVICE ORIENTATION TRACKING USING A VISUAL GYROSCOPE

• US 20150103183A1
• Filed: 10/10/2013
• Published: 04/16/2015
• Est. Priority Date: 10/10/2013
• Status: Abandoned Application

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Abbott, Josh, Heinly, Jared, van Welzen, Jim
• Application Number:
  US14/051,282
• Publication Number:
  US 20150103183A1
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method for tracking device orientation on a portable device is disclosed. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.

Claims

1. A method for tracking device orientation on a portable device, said method comprising:

initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;

initiating visual tracking using a camera on said portable device;

capturing a frame using said camera;

determining a plurality of visual features in said frame;

matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;

computing a rotation amount between said frame and said keyframe;

responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and

performing an adjustment of said frame with all prior captured keyframes.

2. The method of claim 1, wherein said determining comprises:

computing repeatable and distinct features in said frame using a feature detecting procedure; and

detecting and describing said features using a feature description procedure.

3. The method of claim 2, wherein said feature detecting procedure is selected from a group consisting of:

FAST, Harris &

Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.

4. The method of claim 2, wherein said feature description procedure is selected from a group consisting of:

SURF, SIFT and BRIEF.

5. The method of claim 1, further comprising:

responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.

6. The method of claim 1, wherein said matching is performed using RANSAC.

7. The method of claim 1, wherein said computing is performed using Horn'"'"'s procedure.

8. The method of claim 1 further comprising:

responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and

building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.

9. The method of claim 8, further comprising:

determining if said second orientation map overlaps with keyframes from said orientation map; and

responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.

10. The method of claim 8, further comprising:

determining if said second orientation map overlaps with keyframes from said first orientation map; and

responsive to a determination of overlap, merging said second orientation map with said first orientation map.

11. The method of claim 8, further comprising:

determining if said second orientation map overlaps with keyframes from said first orientation map; and

responsive to a determination of no overlap, continuing to build said second orientation map.

12. A computer-readable storage medium having stored thereon instructions that, if executed by a computer system cause the computer system to perform a method for tracking device orientation on a portable device, said method comprising:

initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;

initiating visual tracking using a camera on said portable device;

capturing a frame using said camera;

determining a plurality of visual features in said frame;

matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;

computing a rotation amount between said frame and said keyframe;

responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and

performing an adjustment of said frame with all prior captured keyframes.

13. The computer-readable medium as described in claim 12, wherein said determining comprises:

computing repeatable and distinct features in said frame using a feature detecting procedure; and

detecting and describing said features using a feature description procedure.

14. The computer-readable medium as described in claim 13, wherein said feature detecting procedure is selected from a group consisting of:

FAST, Harris &

Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.

15. The computer-readable medium as described in claim 13, wherein said feature description procedure is selected from a group consisting of:

SURF, SIFT and BRIEF.

16. The computer-readable medium as described in claim 12, further comprising:

responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.

17. The computer-readable medium as described in claim 12, wherein said matching is performed using RANSAC.

18. The computer-readable medium as described in claim 12, wherein said computing is performed using Horn'"'"'s procedure.

19. The computer-readable medium as described in claim 12, wherein said method further comprises:

responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and

building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.

20. The computer-readable medium as described in claim 19, wherein said method further comprises:

determining if said second orientation map overlaps with keyframes from said first orientation map; and

responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.

21. The computer-readable medium as described in claim 19, further comprising:

determining if said second orientation map overlaps with keyframes from said first orientation map; and

responsive to a determination of overlap, merging said second orientation map with said first orientation map.

22. The computer-readable medium as described in claim 19, further comprising:

determining if said second orientation map overlaps with keyframes from said first orientation map; and

responsive to a determination of no overlap, continuing to build said second orientation map.

23. A system for tracking device orientation on a portable device, said system comprising:

a display screen;

a memory;

a camera; and

a processor configured to implement a visual gyroscope, wherein said visual gyroscope performs a method for tracking device orientation on said portable device, wherein said method comprises;

initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;

initiating visual tracking using a camera on said portable device;

capturing a frame using said camera;

determining a plurality of visual features in said frame;

matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;

computing a rotation amount between said frame and said keyframe;

responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and

performing an adjustment of said frame with all prior captured keyframes.