Viewing media in the context of street-level images

• US 8,447,136 B2
• Filed: 03/04/2010
• Issued: 05/21/2013
• Est. Priority Date: 01/12/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Kroepfl, Michael, Walker, Julian R., Ramos, Gonzalo A., Aguera y Arcas, Blaise Hilary
• Primary Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US12/717,846
• Publication Number:
  US 20110173565A1
• Time in Patent Office: 1,174 Days
• Field of Search: 382/113, 715/790, 715/781
• US Class Current: 382/284
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G09B 29/00 Maps; Plans; Charts; Diagrams, e.g. route diagram star maps G09B27/04; devices for holding or supporting maps A47B97/02; for computing purposes G06G1/14, G09B1/16; display boards G09F

Abstract

A system for displaying hybrid image data produced by embedding additional media objects within street-level panoramic images includes a user interface through which a user may view, search for, and/or navigate through additional media objects in the context of browsing a virtual environment of a location at street level. In response to user input indicating a request to view a geographic location and/or an additional media object, street-level panoramic image data associated with the geographic location, in which one or more additional media objects also associated with the geographic location have been embedded, may be provided for display through the user interface. The user interface may be provided by a client device including one or more processors that receive hybrid image data produced by one or more processors of a server and display the image data to the user.

Claims

1. A computer-implemented method for providing hybrid image data for display, the method comprising:

receiving user input;

determining a geographic location, a view direction, and at least one media image frame of at least one additional media object for display based at least on the user input, the at least one additional media object having a first depiction of at least a portion of a subject proximate the geographic location along the view direction;

selecting a target panoramic image based at least on the geographic location, the target panoramic image having a second depiction of at least a portion of the subject;

integrating, using at least one computer system, the at least one media image frame into the target panoramic image to produce a hybrid panoramic image having the first depiction of the subject generally aligned with the second depiction of the subject; and

providing hybrid image data for display, the hybrid image data comprising at least a portion of the hybrid panoramic image in accordance with the view direction;

wherein the user input comprises at least one indication selected from the group consisting of at least one indication of the geographic location, at least one indication of the view direction, and at least one indication of a first additional media object desired for display.

2. The computer-implemented method of claim 1, wherein the at least one additional media object comprises at least one photographic image.

3. The computer-implemented method of claim 1, wherein the at least one additional media object comprises a plurality of video image frames.

4. The computer-implemented method of claim 1, wherein the at least one additional media object is obtained from a media sharing website.

5. The computer-implemented method of claim 1, wherein the geographic location is determined based at least on metadata associated with the at least one media image frame.

6. The computer-implemented method of claim 1, further comprising providing at least one indication of at least one available additional media object based at least on the geographic location, wherein the at least one indication is selectable by a user to view the at least one available additional media object.

7. The computer-implemented method of claim 1, further comprising repeating the determining, selecting, integrating and providing for at least one next media image frame of the at least one additional media object in a time progression.

8. The computer-implemented method of claim 1, further comprising providing at least one indication of at least one overlapping media image frame not integrated into the hybrid image data, wherein a footprint of the at least one overlapping media image frame overlaps with a footprint of the at least one integrated media image frame, wherein the at least one indication is selectable by a user to view the at least one overlapping media image frame.

9. A system for providing hybrid image data for display, the system comprising at least one processor programmed to:

receive first user input;

determine a geographic location, a first view direction, and at least one media image frame of at least one additional media object for display based at least on the first user input, the at least one additional media object having a first depiction of at least a portion of a subject proximate the geographic location along the first view direction;

select a target panoramic image based at least on the geographic location, the target panoramic image having a second depiction of at least a portion of the subject;

produce hybrid image data comprising the at least one media image frame in an alignment with at least a first portion of the target panoramic image in accordance with the first view direction, the alignment at least approximately aligning the first depiction of the subject in the at least one media image frame with the second depiction of the subject in the target panoramic image; and

provide the hybrid image data for display;

wherein the first user input comprises at least one indication selected from the group consisting of at least one indication of the geographic location, at least one indication of the first view direction, and at least one indication of a first additional media object desired for display.

10. The system of claim 9, wherein the at least one processor is further programmed to:

receive second user input indicating a second view direction; and

provide updated hybrid image data for display, the updated hybrid image data comprising at least a portion of the at least one media image frame in the alignment with at least a second portion of the target panoramic image in accordance with the second view direction.

11. The system of claim 10, wherein the second user input comprises a selection of a second additional media object desired for display.

12. The system of claim 9, wherein the at least one processor is further programmed to:

receive second user input indicating at least one parameter of additional media objects desired for display; and

provide at least one indication of at least one available additional media object based at least on the at least one parameter, wherein the at least one indication is selectable by a user to view the at least one available additional media object.

13. The system of claim 12, wherein the at least one parameter is selected from the group consisting of at least one location at which at least one additional media object was captured, at least one author of at least one additional media object, at least one subject of at least one additional media object, and at least one time at which at least one additional media object was captured.

14. The system of claim 9, wherein the at least one processor is further programmed to provide at least one indication of at least one overlapping media image frame not included in the hybrid image data, wherein a footprint of the at least one overlapping media image frame overlaps with a footprint of the at least one media image frame included in the hybrid image data.

15. The system of claim 14, wherein the at least one processor is further configured to:

receive second user input comprising a selection of at least one overlapping media image frame; and

provide updated hybrid image data for display, the updated hybrid image data comprising the at least one overlapping image frame in an alignment with at least the first portion of the target panoramic image, the alignment aligning features of the at least one overlapping media image frame with features of the target panoramic image.

16. At least one non-transitory computer-readable storage medium encoded with a plurality of computer-executable instructions that, when executed, perform a method for providing hybrid image data for display, the method comprising:

receiving user input;

determining a geographic location, a view direction, and at least one media image frame of at least one additional media object for display based at least on the user input, the at least one additional media object having a first depiction of at least a portion of a subject proximate the geographic location along the view direction;

selecting a target street-level panoramic image based at least on the geographic location, the target panoramic image having a second depiction of at least a portion of the subject;

determining a projection to project the at least one media image frame onto the target street-level panoramic image, the projection mapping the first depiction of the subject in the at least one media image frame to the second depiction of the subject in the target street-level panoramic image;

embedding the at least one media image frame in the target street-level panoramic image to produce a hybrid panoramic image, wherein the embedding comprises projecting the at least one media image frame onto the target street-level panoramic image and replacing at least a portion of the target street-level panoramic image with the projected at least one media image frame in accordance with the determined projection; and

providing hybrid image data for display, the hybrid image data comprising at least a portion of the hybrid panoramic image in accordance with the view direction;

wherein the user input comprises at least one indication selected from the group consisting of at least one indication of the geographic location, at least one indication of the view direction, and at least one indication of a first additional media object desired for display.

17. The at least one non-transitory computer-readable storage medium of claim 16, wherein the target street-level panoramic image represents a 360-degree range of view.

18. The at least one non-transitory computer-readable storage medium of claim 16, wherein the projection comprises a relative position and at least one geometric transform.

19. The at least one non-transitory computer-readable storage medium of claim 18, wherein the at least one geometric transform comprises a homography.

20. The at least one non-transitory computer-readable storage medium of claim 16, wherein the view direction is determined based at least on the projection.