Systems, devices, and methods for generating a social street view

• US 10,380,726 B2
• Filed: 03/21/2016
• Issued: 08/13/2019
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: University of Maryland, College Park
• Original Assignee: University of Maryland, College Park
• Inventors: Du, Ruofei, Varshney, Amitabh
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US15/559,955
• Publication Number:
  US 20180075591A1
• Time in Patent Office: 1,240 Days
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

Disclosed are systems, devices, and methods for creating a rendering of real-world locations with embedded multimedia elements. An exemplary method includes receiving image data of a real-world location, identifying geographic coordinates of the real-world location and/or a point of view from which the image data was acquired, acquiring multimedia elements relevant to the real-world location based on the geographic coordinates and/or the point of view, and creating a rendering of the image data with the multimedia elements embedded therein.

Claims

1. A method for creating a rendering of real-world locations with embedded multimedia elements, the method comprising:

receiving image data of a real-world location;

identifying geographic coordinates of the real-world location and/or a point of view from which the image data was acquired;

acquiring multimedia elements relevant to the real-world location based on the geographic coordinates and/or the point of view; and

creating a rendering of the image data with the multimedia elements embedded therein,

wherein the multimedia elements are embedded in the image data in a manner which fuses the multimedia elements with the image data,

wherein fusing the multimedia elements with the image data includes;

determining a geometry of the real-world location based on the image data; and

aligning the multimedia elements with the determined geometry of the real-world location,

wherein the aligning includes maximal Poisson sampling.

2. The method according to claim 1, further comprising displaying the rendering of the image data with the multimedia elements embedded therein.

3. The method according to claim 1, wherein the multimedia elements are geo-tagged, and wherein the multimedia elements are acquired based on geo-tag information of the multimedia elements.

4. The method according to claim 1, wherein the multimedia elements include pictures.

5. The method according to claim 1, wherein the multimedia elements include videos.

6. The method according to claim 1, wherein the multimedia elements include textual messages, gestures, graffiti, or audio messages.

7. The method according to claim 1, wherein the multimedia elements include sentiments derived from pictures, videos, textual messages, gestures, graffiti, or audio messages.

8. The method according to claim 1, wherein aligning the multimedia elements with the determined geometry of the real-world location includes displaying the multimedia elements on the sides of buildings in the image data.

9. The method according to claim 1, wherein aligning the multimedia elements with the determined geometry of the real-world location includes displaying the multimedia elements on virtual billboards in the image data.

10. The method according to claim 1, wherein the image data include still images of the real-world location.

11. The method according to claim 1, wherein the image data include video images of the real-world location.

12. The method according to claim 1, wherein displaying the rendering of the image data with the multimedia elements embedded therein includes displaying the rendering using a virtual reality or augmented reality display.

13. The method according to claim 1, wherein the multimedia elements are received from social media platforms.

14. The method according to claim 1, further comprising filtering the acquired multimedia elements based on user-defined parameters.

15. The method according to claim 14, wherein the user-defined parameters include a date, a time, a season, a person, group of people, a social media profile, a keyword, a distance from the real-world location, or a data type of the multimedia elements.

16. The method according to claim 1, wherein the image data includes 360 degree panoramic video images of the real-world location.

17. A system for creating a rendering of real-world locations with embedded multimedia elements, the system comprising:

a computing device including a processor and a memory storing instructions which, when executed by at least one processor, causes the computing device to;

receive image data of a real-world location;

identify geographic coordinates of the real-world location and/or a point of view from which the image data was acquired;

acquire multimedia elements relevant to the real-world location based on the geographic coordinates and/or the point of view;

create a rendering of the image data with the multimedia elements embedded therein; and

send the rendering of the image data with the multimedia elements embedded therein; and

a display device configured to;

receive the rendering of the image data with the multimedia elements embedded therein; and

display the rendering of the image data with the multimedia elements embedded therein,

wherein the multimedia elements are embedded in the image data in a manner which fuses the multimedia elements with the image data,

wherein when fusing the multimedia elements with the image data, the instructions, when executed by the processor, cause the computer to;

determine a geometry of the real-world location based on the image data; and

align the multimedia elements with the determined geometry of the real-world location,

wherein the aligning includes maximal Poisson sampling.

18. The system according to claim 17, wherein the display device is a virtual reality or augmented reality display device.

19. A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause a computer to:

receive image data of a real-world location;

identify geographic coordinates of the real-world location and a point of view from which the image data was acquired;

acquire multimedia elements relevant to the real-world location based on the geographic coordinates and the point of view;

create a rendering of the image data with the multimedia elements embedded therein; and

display the rendering of the image data with the multimedia elements embedded therein,

wherein the multimedia elements are embedded in the image data in a manner which fuses the multimedia elements with the image data,

wherein when fusing the multimedia elements with the image data, the instructions, when executed by the processor, cause the computer to;

determine a geometry of the real-world location based on the image data; and

align the multimedia elements with the determined geometry of the real-world location,

wherein the aligning includes maximal Poisson sampling.