Method and system for panoramic multimedia streaming

• US 10,419,770 B2
• Filed: 09/08/2016
• Issued: 09/17/2019
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: 3649954 Canada Incorporated
• Original Assignee: Vantrix Corporation
• Inventors: Mayrand, Jean
• Primary Examiner(s): Keller, Michael A
• Application Number:
  US15/259,962
• Publication Number:
  US 20170070553A1
• Time in Patent Office: 1,104 Days
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Methods and apparatus for panoramic multimedia streaming where viewers may control spatial coverage of panoramic video components of multimedia signals are disclosed. A novel flexible streaming server is devised to perform client-specific content filtering in addition to adapting multimedia signals to characteristics of individual client devices as well as to varying capacities of network paths to client devices. The server may distribute software modules to client devices to enable viewers to communicate preferred view regions of a panoramic scene. The server includes a learning module devised to retain viewing-preference data, correlate viewing preference to characteristics of client devices, and determine a default viewing preference for each client device. The server implements computationally efficient schemes of generating and distributing content-filtered multimedia signals to clients. The server may be implemented using hardware processing units and memory devices allocated within a shared cloud-computing network.

Claims

1. A method of video signal streaming implemented at a server comprising at least one physical processor, the method comprising:

obtaining from a panoramic multimedia source a panoramic video signal;

transcoding the panoramic video signal to produce a transcoded signal compatible with a specific type of client devices;

generating a signal sample comprising a set of distant frames of the transcoded signal conforming to a predetermined permissible flow rate lower than a flow rate of said transcoded signal, consecutive frames of said set of distant frames being separated by a time interval exceeding a duration of a single frame;

sending said signal sample to a client device of the specific type;

receiving from said client device an identifier of a respective preferred view region;

content-filtering the transcoded signal to produce a client-specific signal corresponding to the respective preferred view region at said predetermined permissible flow rate; and

transmitting the client-specific signal to said client device.

2. The method of claim 1 further comprising:

obtaining from the panoramic multimedia source indications of signal processing applied to the panoramic video signal;

decompressing the panoramic video signal subject to a determination that the panoramic video signal has been compressed; and

de-warping the panoramic video signal subject to a determination that the panoramic video signal has not been de-warped.

3. The method of claim 1 further comprising compressing the client-specific signal, under constraint of the predetermined permissible flow rate, prior to said transmitting.

4. The method of claim 1 further comprising sending to the client device view-selection processor executable instructions to enable a user of the client device to communicate identifications of the preferred view region to the server.

5. The method of claim 1 wherein the panoramic video signal is produced by a single camera.

6. The method of claim 1 wherein the panoramic video signal is produced by combining video signals produced by multiple cameras.

7. The method of claim 1 wherein the at least one physical processor and associated memory devices are allocated within a shared cloud-computing network.

8. The method of claim 1 wherein said obtaining comprises communicating with at least one of:

a signal source comprising a panoramic camera;

a signal source comprising a panoramic camera and a de-warping module;

a signal source comprising a panoramic camera and a compression module; and

a signal source comprising a panoramic camera, a de-warping module, and a compression module.

9. The method of claim 1 further comprising accessing a client-profile database storing characteristics of each client-device type of a set of client-device types, the characteristics comprising upper bounds of frame rate, frame resolution, and flow rate for use in said transcoding.

10. A server for video signal streaming, comprising:

a processor;

a memory device storing computer executable instructions causing the processor to;

obtain a panoramic video signal from a panoramic multimedia source;

transcode the panoramic video signal to produce a transcoded signal compatible with a specific type of client devices belonging to a plurality of client devices;

generate a signal sample comprising a set of distant frames of the transcoded signal conforming to a predetermined permissible flow rate lower than a flow rate of the transcoded signal, consecutive frames of the set of distant frames being separated by a time interval exceeding a duration of a single frame;

send the signal sample to a client device of the specific type;

receive from the client device an identifier of a respective preferred view region;

content-filter the transcoded signal to produce a client-specific signal corresponding to the respective preferred view region at the predetermined permissible flow rate; and

transmit the client-specific signal to the client device.

11. The server of claim 10 further comprising a memory device storing software modules for distribution to the plurality of client devices to enable users of the client devices to communicate identifications of selected viewing regions to the server.

12. The server of claim 10 further comprising a client-profile database, communicatively coupled to the processor, storing characteristics of each client-device type of a set of client-device types, the characteristics comprising upper bounds of frame rate and frame resolution.

13. The server of claim 10 further comprising network access ports for communicating with a plurality of video sources and the plurality of client devices.

14. The server of claim 10 further comprising:

a decompression module for decompressing the panoramic video signal subject to a determination that the panoramic video signal has been compressed at the panoramic multimedia source;

a de-warping module for de-warping the panoramic video signal subject to a determination that the panoramic video signal has not been de-warped at the panoramic multimedia source; and

a compression module for compressing the client-specific signal.

15. A server for video signal streaming, comprising:

a processor;

a memory device storing computer executable instructions organized into;

a network interface for communicating with;

a panoramic multimedia source to acquire a panoramic video signal; and

a plurality of client devices;

a transcoding module for transcoding the panoramic video signal to produce a transcoded signal compatible with a specific type of client devices;

a frame-sampling module for generating a signal sample comprising a set of distant frames of the transcoded signal conforming to a predetermined permissible flow rate lower than a flow rate of the transcoded signal, consecutive frames of the set of distant frames being separated by a time interval exceeding a duration of a single frame;

a client control-data module for sending the signal sample to a client device of the specific type and receiving an identifier of a respective preferred view region from the client device through the network interface;

a client-specific adaptation module for content-filtering the transcoded signal to produce a client-specific signal at the predetermined permissible flow rate for transmission to the client device through the network interface, the client-specific signal corresponding to the respective preferred view region.

16. The server of claim 15 further comprising a source signal processing module equipped with:

a de-warping module for de-warping the panoramic video signal subject to a determination that the panoramic video signal has not been de-warped at the panoramic multimedia source; and

a decompression module for decompressing the panoramic video signal subject to a determination the panoramic video signal has been compressed at the panoramic multimedia source.

17. The server of claim 15 wherein the memory device further comprises software modules for distribution to the plurality of client devices to enable users of the client devices to communicate identifications of selected viewing regions to the server.

18. A system for video signal streaming, comprising:

a plurality of multimedia sources providing panoramic video signals;

a server, comprising;

a processor;

a memory device storing computer executable instructions causing the processor to;

obtain a panoramic video signal from one of the panoramic multimedia sources;

transcode the panoramic video signal to produce a transcoded signal compatible with a specific type of client devices;

generate a signal sample comprising a set of distant frames of the transcoded signal conforming to a predetermined permissible flow rate lower than a flow rate of the transcoded signal, consecutive frames of the set of distant frames being separated by a time interval exceeding a duration of a single frame;

send the signal sample to a client device of the specific type;

receive from the client device an identifier of a respective preferred view region;

content-filter the transcoded signal to produce a client-specific signal corresponding to the respective preferred view region at the predetermined permissible flow rate; and

transmit the client-specific signal to the client device.

19. The system of claim 18 wherein at least one of the panoramic multimedia sources comprises:

a panoramic camera of up to 4π

solid angle coverage;

a de-warping module; and

a compression module.

20. The system of claim 19 wherein the at least one of the panoramic multimedia sources is configured to provide:

a raw signal as captured at said panoramic camera;

a corrected signal derived from the raw signal using the de-warping module;

a compressed signal derived from the raw signal using the compression module; and

a compact signal derived from the raw signal using the de-warping module and the compression module.

21. The system of claim 20 wherein the at least one of the panoramic multimedia sources is configured to provide indication of signal-processing functions applied to the raw signal.

22. The system of claim 18 further comprising a dedicated transmission medium connecting the server to the one of the panoramic multimedia sources.

23. The system of claim 18 further comprising access to a network connecting the server to the one of the panoramic multimedia sources.

24. The system of claim 23 further comprising at least one additional server connecting to the plurality of multimedia sources through the network.

25. The server of claim 18 wherein the memory device further comprises software modules for distribution to the client devices to enable users of the client devices to communicate identifications of selected viewing regions to the server.