Hybrid media viewing application including a region of interest within a wide field of view

• US 10,440,329 B2
• Filed: 05/24/2010
• Issued: 10/08/2019
• Est. Priority Date: 05/22/2009
• Status: Active Grant

• Current Assignee: Immersive Ventures Inc. (Digital Domain Holdings Ltd.)
• Original Assignee: Immersive Media Company
• Inventors: McCutchen, David, Siroshton, Benjamen, Humphrey, Benjamin, Exner, James, McCandless, Casey
• Primary Examiner(s): Bilgrami, Asghar H
• Application Number:
  US12/786,270
• Publication Number:
  US 20100299630A1
• Time in Patent Office: 3,424 Days
• Field of Search: 709226
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A content delivery and display solution includes a viewing application for displaying immersive images with a region of interest, in addition to conventional fixed-aspect-ratio media. The display can include the layered display of metadata, multiple windows, and images or hotspots embedded into the immersive image. The viewing application can be used for the display of either live or prerecorded images, from local or online sources.

Claims

1. A system for the display of media and associated metadata, comprising:

network resources including;

at least one media server having;

a computing platform; and

storage for storing immersive image media, region of interest subdivisions of the immersive image media and identification information for said immersive image media and its associated region of interest subdivisions;

wherein the at least one media server is operable to deliver the immersive image media or a region of interest subdivision upon a request from a client according to its identification information;

at least one metadata server having;

a computing platform; and

storage for storing a plurality of metadata associated with the immersive image media or the region of interest subdivisions of the immersive image media and identification information for said metadata;

wherein the at least one metadata server is operable to deliver the metadata upon a request from a client according to its identification information;

a client including;

a computing platform having;

a memory storing a client viewing application, the client viewing application having machine-readable instructions that, when executed by the computing platform, cause the client to provide;

a user interface service for generating requests for specific network resources, including the identification information for the requested network resources, and for communicating to the servers representing the requested network resources;

a data management service for receiving data comprising requested immersive image media and requested metadata or a requested region of interest subdivision of the requested immersive image media and requested metadata from the servers, including identification information, and for storing the received data in the memory;

a data formatting service for identifying a source formatting of the received data and for formatting the received data;

an application display service for presenting data formatted by the data formatting service on a display, including presenting, on the display, the requested region of interest; and

a playback control service for controlling a speed and a direction of movement of the requested region of interest within the immersive image media presented on the display by the application display service according to user input, the movement of the requested region of interest comprising spatial movement of the region of interest from a first user-selected spatial portion of the immersive image media to a second user-selected spatial portion of the immersive image media;

wherein the application display service also presents, on the display, at least one of;

(i) the requested metadata; and

(ii) the requested immersive image media and the requested metadata.

2. The system of claim 1, wherein the requested image media comprises immersive video media comprising an image sequence and associated audio data.

3. The system of claim 1, wherein the data formatting service spatially rearranges the received data.

4. The system of claim 1 wherein the playback control service provides for movement of the requested region of interest within the requested immersive image media on the display solely by movement of a cursor within the display.

5. The system of claim 1 wherein the application display service is operable to present, on the display, one of:

the requested region of interest and the requested metadata; and

the requested region of interest, the requested immersive image media and the requested metadata, according to the source formatting of the received data.

6. The system of claim 5 wherein the application display service is operable to present an indication of when the application display service is presenting the requested region of interest and the requested metadata.

7. The system of claim 5 wherein the presented image fills the available area of the display.

8. The system of claim 1 wherein the application display service includes indicators of other media which can be selected for display by said user interface.

9. The system of claim 1 wherein the application display service includes the capability of adding plug in services.

10. The system of claim 1 wherein the display comprises a head mounted display, wherein the client viewing application further has machine-readable instructions that cause the client to provide a head tracker service for controlling the direction of a region of interest window and displaying the region of interest window as a separate window in the head mounted display.

11. The system of claim 1 wherein the application display service presents, on the display, an overlay of metadata representing the identification information stored on the media server.

12. The system of claim 1 wherein the application display service copies and stores the requested region of interest and the requested metadata.

13. The system of claim 1 wherein the application display service includes an ability to present additional images, meshes and text on the display according to latitude and longitude coordinates.

14. The system of claim 13 wherein geospatial information is derived from the appearance of pixels along projected rays as seen from successive video frames.

15. The system of claim 5 wherein the application display service includes an ability to stabilize the requested region of interest according to camera pose information.

16. The system of claim 15 wherein the camera pose information is derived from visual analysis of objects in view over successive video frames.

17. The system of claim 5 wherein the application display service includes an ability to record and play back a path taken by a manually controlled region of interest over time as a view track, and to select from among multiple stored view tracks to automatically control the movement of a region of interest window.

18. The system of claim 1 wherein the at least one media server is a live streaming immersive video server.

19. The system of claim 5 wherein the application display service presents the requested region of interest and a vertically cropped worldview image of the requested immersive image media, with the worldview image having an indication of the location of the region of interest window being displayed.

20. A method for the display of media and associated metadata, comprising:

storing and reading machine-readable instructions for a client viewing application which includes a user interface service, a data management service, a data formatting service, an application display service, and a playback control service;

using the user interface service to generate requests for specific network resources according to identification information for the requested network resources, and to communicate to the servers representing the requested network resources;

using the data management service to receive data comprising requested immersive image media and requested metadata or a requested region of interest subdivision of the requested immersive image media and requested metadata from the servers, including the identification information, and to store the received data in a memory;

using the data formatting service to format the received data according to its source formatting;

using the application display service to present data formatted by the data formatting service on a display including presenting, on the display, the requested region of interest; and

using the playback control service to control a speed and a direction of movement of the requested region of interest presented on the display by the application display service according to user input, the movement of the requested region of interest comprising spatial movement of the region of interest from a first user-selected spatial portion of the immersive image media to a second user-selected spatial portion of the immersive image media;

wherein the application display service also presents, on the display, at least one of;

(i) the requested metadata; and

(ii) the requested immersive image media and the requested metadata.

21. A tangible article of machine-readable storage media having machine-readable instructions for a computing platform stored thereon, the storage medium selected from the group consisting of magnetic storage, optical storage, quantum storage, and phase change storage, said machine-readable instructions, when executed by the computing platform, results in the computing platform performing the method recited in claim 20.

22. A system for a client including:

a storage medium for the storage of machine-readable instructions for a computing platform, the storage medium selected from the group consisting of magnetic storage, optical storage, quantum storage, and phase change storage, said machine-readable instructions, when executed by the computing platform, cause the client to provide;

a user interface service for generating requests for specific network resources, including the identification information for the requested network resources, and for communicating to the servers representing the requested network resources;

a data management service for receiving data comprising requested immersive image media and requested metadata or a requested region of interest subdivision of the requested immersive image media and requested metadata from the servers, including identification information, and for storing the received data in a memory of the computing platform;

a data formatting service for identifying a source formatting of the received data and for formatting the received data;

an application display service for presenting data formatted by the data formatting service on a display, including presenting, on the display, the requested region of interest; and

a playback control service for controlling a speed and a direction of movement of the requested region of interest within the immersive image media presented on the display by the application display service according to user input, the movement of the requested region of interest comprising spatial movement of the region of interest from a first user-selected spatial portion of the immersive image media to a second user-selected spatial portion of the immersive image media;

wherein the application display service also presents, on the display, at least one of;

(i) the requested metadata; and

(ii) the requested immersive image media and the requested metadata.