Spatially registered augmented video

• US 9,153,073 B2
• Filed: 03/13/2013
• Issued: 10/06/2015
• Est. Priority Date: 05/23/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Langlotz, Tobias Martin, Zingerle, Mathaus, Reitmayr, Gerhard
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US13/802,065
• Publication Number:
  US 20130314442A1
• Time in Patent Office: 937 Days
• Field of Search: 345/619, 345/633
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A source video stream is processed to extract a desired object from the remainder of video stream to produce a segmented video of the object. Additional relevant information, such as the orientation of the source camera for each frame in the resulting segmented video of the object, is also determined and stored. During replay, the segmented video of the object, as well as the source camera orientation are obtained. Using the source camera orientation for each frame of the segmented video of the object, as well as target camera orientation for each frame of a target video stream, a transformation for the segmented video of the object may be produced. The segmented video of the object may be displayed over the target video stream, which may be a live video stream of a scene, using the transformation to spatially register the segmented video to the target video stream.

Claims

1. A method comprising:

obtaining a plurality of segmented image frames of an object, a first panoramic image associated with the plurality of segmented image frames of the object, and an orientation of a source camera with respect to the first panoramic image for each frame in the plurality of segmented image frames, the plurality of segmented image frames of the object captured with the source camera;

causing a plurality of target image frames to be captured with a target camera;

generating a second panoramic image from the plurality of target image frames;

determining an orientation of the target camera with respect to the second panoramic image for each frame of the plurality of target image frames;

matching the first panoramic image to the second panoramic image to align the first panoramic image to the second panoramic image;

calculating a transformation for each frame of the plurality of segmented image frames of the object with the first panoramic image aligned to the second panoramic image using the orientation of the source camera with respect to the first panoramic image for each frame in the plurality of segmented image frames and the orientation of the target camera with respect to the second panoramic image for each respective frame in the plurality of target image frames; and

causing the plurality of segmented image frames to be displayed over the plurality of target image frames using the transformation for each frame.

2. The method of claim 1, wherein causing the plurality of segmented image frames to be displayed over the plurality of target image frames using the transformation for each frame spatially registers each frame from of the plurality of segmented image frames of the object with each frame of the plurality of target image frames.

3. The method of claim 1, wherein determining the orientation of the target camera with respect to the second panoramic image for each frame comprises using data from inertial sensors.

4. The method of claim 1, wherein determining the orientation of the target camera with respect to the second panoramic image for each frame comprises using vision based tracking.

5. The method of claim 1, wherein the plurality of segmented image frames of the object and the plurality of target image frames are captured from a same geographical location.

6. The method of claim 1, wherein the transformation for each frame of the plurality of segmented image frames of the object compensates for a difference in position of the source camera and the target camera.

7. The method of claim 1, further comprising:

comparing the first panoramic image and the second panoramic image to determine a lighting difference; and

adjusting the plurality of segmented image frames of the object based on the lighting difference before displaying the plurality of segmented image frames of the object over the plurality of target image frames.

8. The method of claim 1, wherein the orientation of the source camera and the orientation of the target camera are absolute or relative orientations.

9. The method of claim 1, wherein the object in the plurality of segmented image frames is a foreground object or actor or a moving element from a plurality of source frames captured by the source camera and designated by a user.

10. A mobile device comprising:

a camera capable of capturing a plurality of target image frames;

a display capable of displaying the plurality of target image frames; and

a processor coupled to the camera and the display, the processor configured to obtain a plurality of segmented image frames of an object, a first panoramic image associated with the plurality of segmented image frames of the object, and an orientation of a source camera with respect to the first panoramic image for each frame in the plurality of segmented image frames, the plurality of segmented image frames of the object captured with the source camera;

generate a second panoramic image from the plurality of target image frames;

determine an orientation of the camera with respect to the first panoramic image for each frame of the plurality of target image frames;

match the first panoramic image to the second panoramic image to align the first panoramic image to the second panoramic image;

calculate a transformation for each frame of the plurality of segmented image frames of the object with the first panoramic image aligned to the second panoramic image using the orientation of the source camera with respect to the first panoramic image and the orientation of the camera with respect to the second panoramic image; and

display the plurality of segmented image frames of the object over the plurality of target image frames on the display using the transformation for each frame.

11. The mobile device of claim 10, wherein the processor is configured to display the plurality of segmented image frames of the object over the plurality of target image frames on the display using the transformation for each frame to spatially register the plurality of segmented image frames of the object with the plurality of target image frames.

12. The mobile device of claim 10, further comprising inertial sensors coupled to the processor, wherein the processor is configured to determine the orientation of the camera with respect to the second panoramic image for each frame of the plurality of target image frames using data received from the inertial sensors.

13. The mobile device of claim 10, wherein the processor is configured to determine the orientation of the camera with respect to the second panoramic image for each frame of the plurality of target image frames using vision based tracking with the plurality of target image frames.

14. The mobile device of claim 10, wherein the plurality of segmented image frames of the object and the plurality of target image frames are captured from a same geographical location.

15. The mobile device of claim 10, wherein the transformation for each of the plurality of segmented image frames of the object compensates for a difference in position of the source camera and the camera.

16. The mobile device of claim 10, wherein the processor is further configured to compare the first panoramic image and the second panoramic image to determine a lighting difference, and adjust the plurality of segmented image frames of the object based on the lighting difference before displaying the plurality of segmented image frames of the object over the plurality of target image frames.

17. The mobile device of claim 10, wherein the orientation of the source camera and the orientation of the camera are absolute or relative orientations.

18. The mobile device of claim 10, wherein the object in the plurality of segmented image frames is a foreground object or actor or a moving element from a plurality of source frames captured by the source camera and designated by a user.

19. A mobile device comprising:

means for obtaining a plurality of segmented image frames of an object, a first panoramic image associated with the plurality of segmented image frames of the object, and an orientation of a source camera with respect to the first panoramic image for each frame in the plurality of segmented image frames, the plurality of segmented image frames of the object captured with the source camera;

means for capturing a plurality of target image frames with a target camera;

means for generating a second panoramic image from the plurality of target image frames;

means for determining an orientation of the target camera with respect to the second panoramic image for each frame of the plurality of target image frames;

means for matching the first panoramic image to the second panoramic image to align the first panoramic image to the second panoramic image;

means for calculating a transformation for each frame of the plurality of segmented image frames of the object with the first panoramic image aligned to the second panoramic image using the orientation of the source camera with respect to the first panoramic image and the orientation of the target camera with respect to the second panoramic image; and

means for displaying the plurality of segmented image frames of the object over the plurality of target image frames using the transformation for each frame.

20. The mobile device of claim 19, wherein the means for displaying the plurality of segmented image frames of the object over the plurality of target image frames using the transformation for each frame spatially registers the plurality of segmented image frames of the object with the plurality of target image frames.

21. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to obtain a plurality of segmented image frames of an object, a first panoramic image associated with the plurality of segmented image frames of the object, and an orientation of a source camera with respect to the first panoramic image for each frame in the plurality of segmented image frames, the plurality of segmented image frames of the object captured with the source camera;

program code to generate a second panoramic image from a plurality of target image frames;

program code to determine an orientation of a target camera with respect to the second panoramic image for each frame of a plurality of target image frames captured with the target camera;

program code to match the first panoramic image to the second panoramic image to align the first panoramic image to the second panoramic image;

program code to calculate a transformation for each frame of the plurality of segmented image frames of the object with the first panoramic image aligned to the second panoramic image using the orientation of the source camera with respect to the first panoramic image and the orientation of the target camera with respect to the second panoramic image; and

program code to display the plurality of segmented image frames of the object over the plurality of target image frames using the transformation.

22. A method comprising:

obtaining a plurality of source image frames including an object and a background that is captured with a moving camera;

segmenting the object from the background in the plurality of source image frames to produce a plurality of segmented image frames of the object;

generating a panoramic image with the background from the plurality of source image frames;

determining an orientation of the moving camera for each frame of the plurality of segmented image frames of the object; and

storing the plurality of segmented image frames of the object with the panoramic image and the orientation of the moving camera for each frame of the plurality of segmented image frames of the object.

23. The method of claim 22, wherein segmenting the object comprises separating the object from the background in each frame of the plurality of source image frames.

24. The method of claim 22, further comprising:

capturing a plurality of target image frames;

determining a target camera orientation for each frame of the plurality of target image frames;

calculating a transformation for each frame of the plurality of segmented image frames of the object using the panoramic image, the orientation of the moving camera for each frame of the plurality of segmented image frames and the target camera orientation for each frame of the plurality of target image frames; and

displaying the plurality of segmented image frames of the object over the plurality of target image frames using the transformation for each frame.

25. The method of claim 22, wherein the orientation of the moving camera is an absolute or relative orientation.

26. The method of claim 22, wherein the moving camera is rotating.

27. The method of claim 22, wherein segmenting the object from the plurality of source image frames to produce the plurality of segmented image frames of the object comprises:

identifying the object in a first frame of the plurality of source image frames based on user selection;

segmenting the object from the first frame;

identifying the object in each subsequent frame of the plurality of source image frames based on optical flow using the identification of the object from a proceeding frame of the plurality of source image frames;

segmenting the object from each subsequent frame;

combining the object segmented from the first frame and the object segmented from each subsequent frame to produce the plurality of segmented image frames of the object.

28. The method of claim 22, wherein the object is a foreground object or actor or a moving element from the plurality of source image frames captured by the moving camera and designated by a user.

29. An apparatus comprising:

a database; and

a processor coupled to the database, the processor being configured to obtain a plurality of source image frames including an object an a background that is captured with a moving camera, segment the object from the background in the plurality of source image frames to produce a plurality of segmented image frames of the object;

generate a panoramic image with the background from the plurality of source image frames;

determine an orientation of the moving camera for each frame of the plurality of segmented image frames of the object, and store the plurality of segmented image frames of the object with the panoramic image and the orientation of the moving camera for each frame of the plurality of segmented image frames of the object in the database.

30. The apparatus of claim 29, wherein the processor is configured to segment the object from the plurality of source image frames by being configured to separate the object from the background in each frame of the plurality of source image frames.

31. The apparatus of claim 29, wherein the orientation of the moving camera is an absolute or relative orientation.

32. The apparatus of claim 29, wherein the moving camera is rotating.

33. The apparatus of claim 29, wherein the processor is configured to segment the object from the plurality of source image frames to produce the plurality of segmented image frames of the object by being configured to segment the object from a first frame of the plurality of source image frames based on a user identification of the object in the first frame, perform optical flow to identify the object in each subsequent frame of the plurality of source image frames using an identification of the object from a proceeding frame of the plurality of source image frames, segment the object from each subsequent frame, and combine the object segmented from the first frame and the object segmented from each subsequent frame to produce the plurality of segmented image frames of the object.

34. The apparatus of claim 29, wherein the object is a foreground object or actor or a moving element from the plurality of source image frames captured by the moving camera and designated by a user.

35. An apparatus comprising:

means for obtaining a plurality of source image frames including an object and a background that is captured with a moving camera;

means for segmenting the object from the background in the plurality of source image frames to produce a plurality of segmented image frames of the object;

means for generating a panoramic image with the background from the plurality of source image frames;

means for determining an orientation of the moving camera for each frame of the plurality of segmented image frames of the object; and

means for storing the plurality of segmented image frames of the object with the panoramic image and the orientation of the moving camera for each frame of the plurality of segmented image frames of the object.

36. The apparatus of claim 35, wherein the means for segmenting the object separates the object from the background in each frame of the plurality of source image frames.

37. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to obtain a plurality of source image frames including an object and a background that is captured with a moving camera;

program code to segment the object from the background in the plurality of source image frames to produce a plurality of segmented image frames of the object;

program code to generate a panoramic image with the background from the plurality of source image frames;

program code to determine an orientation of the moving camera for each frame of the plurality of segmented image frames of the object; and

program code to store the plurality of segmented image frames of the object with the panoramic image and the orientation of the moving camera for each frame of the plurality of segmented image frames of the object.