System and Method for Synchronizing, Merging, and Utilizing Multiple Data Sets for Augmented Reality Application

• US 20140168243A1
• Filed: 12/19/2012
• Published: 06/19/2014
• Est. Priority Date: 12/19/2012
• Status: Active Grant

• Current Assignee: Jeffrey Huang
• Original Assignee: Jeffrey Huang
• Inventors: Huang, Jeffrey
• Granted Patent: US 9,330,431 B2
• Time in Patent Office: Days
• US Class Current: 345/522
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612

Abstract

Systems and methods for synchronizing, merging, and utilizing multiple data sets for augmented reality application are disclosed. In one example, an electronic system receives and processes live recorded video information, GPS information, map data information, and points of interest information to produce a data set comprising merged graphical and/or audio information and non-graphical and non-audio information metadata that are referenced to the same clock and timestamp information. This data set can be stored in a cloud network storage. By retaining numerical and textual values of non-graphical and non-audio information (e.g. camera viewing angle information, GPS coordinates, accelerometer values, and compass coordinates) as metadata that are referenced to the same clock and timestamp information within the data set, an augmented reality application that replays information or augments information in real time can dynamically select or change how the data set is presented in augmented reality based on dynamically-changeable user preferences.

Claims

1. An electronic system for synchronizing and merging multiple data sets for an augmented reality application, the electronic system comprising:

a CPU and a memory unit configured to execute the augmented reality application, wherein the augmented reality application provides a user interface on a display screen for display of video and other information associated with a real-world environment;

a first data set comprising live recorded video and/or audio information referenced to clock and timestamp information, wherein the first data set is loaded to the memory unit of the electronic system;

a second data set comprising GPS information, map data information, and points of interest information, wherein the second data set is also loaded to the memory unit of the electronic system;

a third data set comprising merged graphical and/or audio information and non-graphical and non-audio information metadata, wherein the merged graphical and/or audio information and the non-graphical and non-audio information metadata are created from the first data set and the second data set using the augmented reality application executed on the CPU, and wherein the merged graphical and/or audio information and the non-graphical and non-audio information metadata of the third data set are both referenced to the clock and timestamp information of the first data set to enable correct time synchronization of the merged graphical and/or audio information and the non-graphical and non-audio information metadata during a full or selective replay; and

an external communication input and output interface configured to transmit the third data set as one or more data packets to another electronic system via a data network.

2. The electronic system of claim 1, wherein the non-graphical and non-audio information metadata includes the clock and timestamp information, camera viewing angle information, GPS coordinates, accelerometer values, compass coordinates, and/or other non-graphical and non-audio information from the first data set and the second data set.

3. The electronic system of claim 1, wherein the merged graphical and/or audio information of the third data set includes the live recorded video information and a graphical portion of the map data information from the first data set and the second data set.

4. The electronic system of claim 1, wherein another electronic system connected via the data network operates a social media website, a video content access website, a geographic location search-based augmented reality application, or a multimedia analytics program.

5. The electronic system of claim 1, further comprising a local data storage integrated in the electronic system, wherein the local data storage is configured to save a local copy of at least one of the first data set, the second data set, and the third data set.

6. The electronic system of claim 1, further comprising a GPS and location tracking unit capable of receiving GPS signals via a GPS antenna and generating the GPS information relative to a current location of the electronic system.

7. The electronic system of claim 1, further comprising a camera processing unit operatively connected to a camera lens, wherein the camera processing unit is capable of recording the video information from the real-world environment through the camera lens, and wherein the camera processing unit is also at least partly capable of converting the video information to the live recorded video information of the first data set in association with the CPU.

8. The electronic system of claim 1, further comprising a power management unit operatively connected to a battery, a power adapter, or another power source.

9. The electronic system of claim 1, further comprising a graphics unit capable of controlling and driving a display driver to display the user interface on the display screen.

10. The electronic system of claim 1, further comprising a digital signal processing unit for a cloud network access via a wireless data network.

11. The electronic system of claim 1, wherein the external communication input and output interface includes a network interface card for a data network connection.

12. A method for synchronizing, merging, and utilizing multiple data sets for an augmented reality application executed on a CPU and a memory unit of an electronic system, the method comprising:

displaying a live video footage via a user display panel while the live footage is being recorded into a local and/or a network-attached storage;

displaying map graphics and underlying non-graphical and non-audio data including GPS coordinates, points of interest, and/or current timestamps; and

if the augmented reality application is configured to merge the live video footage with the map graphics and the underlying non-graphical and non-audio data;

creating a merged graphical and/or audio information data set comprising the live video footage and the map graphics;

creating a metadata set for non-graphical and non-audio information which includes the underlying non-graphical and non-audio data, wherein the metadata set is time-synchronized with the merged graphical and/or audio information data set by referencing to same timestamps; and

storing the merged graphical and/or audio information data set and the metadata set for non-graphical and non-audio information as separate files or as a combined file in a local and/or a cloud network storage.

13. The method of claim 12, further comprising a step of executing or logging into a video content access application on the electronic system or on another electronic system operatively connected to the cloud network storage.

14. The method of claim 13, further comprising a step of selecting the separate files or the combined file for replay.

15. The method of claim 14, further comprising a step of creating a dynamically-changeable replay environment of augmented reality by processing the merged graphical and/or audio information data set and the metadata set for non-graphical and non-audio information from the separate files or from the combined file stored in the cloud network storage.

16. The method of claim 15, further comprising a step of incorporating a user'"'"'s general or specific augmented reality preferences to generate one or more graphical views and non-graphical and non-audio information for the dynamically-changeable replay environment.

17. The method of claim 16, further comprising a step of displaying the dynamically-changeable replay environment of augmented reality, wherein the dynamically-changeable replay environment includes a user interface to trigger a dynamic change to the one or more graphical views and the non-graphical and non-audio information.

18. A method for providing a geographic location search-based augmented reality application executed on a CPU and a memory unit of an electronic system, the method comprising:

loading the geographic location search-based augmented reality application on the CPU and the memory of the electronic system, wherein the electronic system is an electronic goggle with an embedded display and an eye-movement tracking sensor for location-pointing, another wearable computer, or another electronic device, which is configured to retrieve previously-stored information associated with a particular geographic location;

selecting or pointing to the particular geographic location using the electronic system during an operation of the geographic location search-based augmented reality application;

checking whether a video file is associated with the particular geographic location in a cloud network storage or a local storage operatively connected to the electronic system; and

if the video file is found, checking whether the video file includes metadata for non-graphical and non-audio information;

if the metadata for non-graphical and non-audio information is included in the video file;

extracting the metadata for the geographic location search-based augmented reality application using a user'"'"'s general and/or specific preferences, and displaying dynamically-changeable geographic location-associated graphical information and non-graphical and non-audio information;

else if the metadata for non-graphical and non-audio information is not included in the video file;

replaying the video file which does not have separate metadata for non-graphical and non-audio information.

19. The method of claim 18, wherein the electronic system is operatively connected to the cloud network storage by a cellular network or another wireless data network.

20. The method of claim 18, wherein the dynamically-changeable geographic location-associated graphical information and non-graphical and non-audio information include previously-recorded live video footage, previously-recorded map graphics associated with GPS data, points of interest near the particular geographic location selected or pointed by the user, or other stored information specific to the particular geographic location.