Parallax compensation

• US 10,008,021 B2
• Filed: 12/14/2011
• Issued: 06/26/2018
• Est. Priority Date: 12/14/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wantland, Timothy, Ramos, Gonzalo Alberto, Barnett, Donald
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): HASAN, SYED Y
• Application Number:
  US13/325,430
• Publication Number:
  US 20130155180A1
• Time in Patent Office: 2,386 Days
• Field of Search: 348 36, 348 26, 348 42, 348 56, 348154, 348 E7001, 348 51, 348142, 348744, 348 E7085, 348 E9025, 348E13001, 348 47, 348155, 348169, 348239, 34833312, 345419, 345629, 715786, 715854, 382106, 382195
• CPC Class Codes: G01S 3/7865 using correlation of the live video image with a stored image
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Among other things, one or more techniques and/or systems are disclosed for rendering imagery that compensates for parallax. Image data may comprise a sequence of images of a location, collected at known intervals (e.g., distances), such as along a path. The image data can comprise a first layer comprising a first depth, and a second layer comprising a second depth, that is different from the first depth. The image data can be composed into resulting imagery, where the first layer is composed at the first depth and the second layer is composed at the second depth. The resulting imagery can be rendered to compensate for, or rather make use of, parallax, such that the first layer is rendered at a first movement speed, based at least upon the first depth, and the second layer is rendered at a second movement speed, based at least upon the second depth.

Claims

1. A computer-based method for rendering imagery that utilizes parallax, comprising:

receiving image data comprising a first image captured from a first observational position and a second image captured from a second observational position different than the first observational position, wherein the first and second image each include a first object and a second object and the first and second images are captured by one or more cameras;

identifying the first object and the second object in the first image;

determining a first location of the first object in the first image and a second location of the second object in the first image;

identifying the first object and the second object in the second image;

determining a third location of the first object in the second image and a fourth location of the second object in the second image;

determining a first depth of the first object based at least upon the first location and the third location;

determining a second depth of the second object based at least upon the second location and the fourth location, the second depth different than the first depth;

composing a first layer for objects within a first depth range;

composing a second layer for objects within a second depth range;

associating the first object with a first layer based at least upon the determined first depth being within the first depth range for the first layer;

associating the second object with a second layer based at least upon the determined second depth being within the second depth range for the second layer; and

rendering resulting imagery to compensate for parallax by rendering the first layer at a first movement speed and the second layer at a second movement speed, the second movement speed different than the first movement speed.

2. The method of claim 1, the resulting imagery comprising a lateral panorama.

3. The method of claim 1, the rendering based at least upon a received indication to pan the resulting imagery.

4. The method of claim 1, the first layer comprising an object tag comprising information associated with the first object, and wherein the object tag is rendered at the first movement speed such that the object tag remains in a same relative position to the first object.

5. The method of claim 4, further comprising annotating the information to the first object in the resulting imagery.

6. The method of claim 1, the first layer comprising a foreground and the second layer comprising a background.

7. The method of claim 1:

receiving image data comprising an object tag layer, the object tag layer comprising information associated with an object identified in the image data;

composing the object tag layer at a third depth in the resulting imagery; and

rendering the resulting imagery comprising rendering the object tag layer at a third movement speed, the third movement speed based at least upon the third depth.

8. The method of claim 7, the third movement speed based at least upon an object weighting factor.

9. The method of claim 1, comprising:

receiving an indication of a panning speed for the resulting imagery; and

determining the first movement speed and the second movement speed based at least upon the panning speed.

10. The method of claim 1, comprising:

identifying a non-desired view of an object in the image data; and

composing the resulting imagery to mitigate the non-desired view of the object based at least upon the first depth and the second depth.

11. The method of claim 1, wherein the first layer is a foreground imagery layer including a plurality of objects having varying depths within the first range and the second layer is a background imagery layer including a plurality of objects having varying depths within the second range;

wherein rendering the first layer at the first movement speed causes all objects within the first layer to be rendered as moving at the first movement speed; and

wherein rendering the second layer at the second movement speed causes all objects within the second layer to be rendered as moving at the second first movement speed.

12. A system for rendering imagery that utilizes parallax, comprising:

one or more processing units; and

memory comprising instructions that when executed by at least one of the one or more processing units perform operations comprising;

receiving image data comprising a first image captured from a first observational position and a second image captured from a second observational position different than the first observational position, wherein the first and second image each include a first object and a second object and the first and second images are captured by one or more cameras;

identifying the first object and the second object in the first image;

identifying the first object and the second object in the second image;

determining a first depth of the first object based at least upon a position of the first object in the first image and the second image;

determining a second depth of the second object based at least upon a position of the second object in the first image and the second image;

composing a first layer including objects within a first depth range;

composing a second layer including objects within a second depth range;

associating the first object with the first layer based at least upon the first depth being within the first depth range for the first layer;

associating the second object with the second layer based at least upon the second depth being within the second depth range for the second layer; and

rendering the resulting imagery to compensate for parallax by rendering the first layer at a first movement speed and the second layer at a second movement speed, the second movement speed different than the first movement speed.

13. The system of claim 12, the resulting imagery comprising a lateral panorama.

14. The system of claim 12, the second image captured consecutive to the first image.

15. The system of claim 12, the operations comprising

determining at least one of the first movement speed or the second movement speed based at least upon a received indication of a panning speed for the resulting imagery.

16. The system of claim 12, the operations comprising

annotating an object in the resulting imagery with an object tag comprising information associated with the object.

17. The system of claim 16, the information associated with the object comprising at least one of:

geo-location data identifying a location of the object;

identification data that identifies the object;

or

descriptor information that provides information about the object.

18. The system of claim 16, the operations comprising determining a third movement speed for an object tag layer comprising the object tag.

19. The system of claim 12, the operations comprising:

identifying a non-desired view of an object in the image data; and

adjusting the resulting imagery to mitigate the non-desired view of the object based at least upon the first depth and the second depth.

20. A computer readable device comprising instructions that when executed perform a method for rendering imagery that utilizes parallax, comprising:

receiving image data comprising a first image captured from a first observational position and a second image captured from a second observational position different than the first observational position, wherein the first and second image each include a first object and a second object and the first and second images are captured by one or more cameras;

identifying the first object and the second object in the first image;

determining a first line of sight angle to the first object in the first image and a second line of sight angle to the second object in the first image;

identifying the first object and the second object in the second image;

determining a third line of sight angle to the first object in the second image and a fourth line of sight angle to the second object in the second image;

determining a first depth of the first object based at least upon the first line of sight angle and the third line of sight angle;

determining a second depth of the second object based at least upon the second line of sight angle and the fourth line of sight angle, the second depth different than the first depth;

composing a first layer including objects within a first depth range;

composing a second layer including objects within a second depth range;

associating the first object with a first layer based at least upon the first depth;

associating the second object with a second layer based at least upon the second depth; and

rendering the resulting imagery to compensate for parallax by rendering the first layer at a first movement speed and the second layer at a second movement speed, the second movement speed different than the first movement speed.