Client-server communication for live filtering in a camera view

• US 10,070,154 B2
• Filed: 02/07/2017
• Issued: 09/04/2018
• Est. Priority Date: 02/07/2017
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Munaro, Matteo, Kar, Abhishek, Trevor, Alexander Jay Bruen, Chande, Krunal Ketan, Rusu, Radu Bogdan
• Primary Examiner(s): Huynh, An Son Phi
• Application Number:
  US15/427,027
• Publication Number:
  US 20180227601A1
• Time in Patent Office: 574 Days
• Field of Search: None
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/6547 comprising parameters, e.g. for client setup

Abstract

Provided are mechanisms and processes for performing live filtering in a camera view via client-server communication. In one example, a first video frame in a raw video stream is transmitted from a client device to a server. The client device receives a filter processing message associated with the first video frame that includes filter data for applying a filter to the first video frame. A processor at the client device creates a filtered video stream by applying the filter to a second video frame that occurs in the video stream later than the first video frame.

Claims

1. A method comprising:

automatically transmitting, using an electronic client device, a first video frame in a raw video stream from the electronic client device to a server via a communication network in response to determining the first video frame meets a designated criterion, the raw video stream being captured live by the electronic client device;

receiving, by the electronic client device, from the server a filter processing message associated with the first video frame, the filter processing message including filter data for applying a filter to the first video frame;

creating a filtered video stream in real time via a processor at the electronic client device by applying the filter to a second video frame, wherein the first video frame precedes the second video frame in the raw video stream, wherein the filter is applied to the second video frame by propagating information from the first video frame to the second video frame based on the filter data, wherein the second video frame is neither transmitted to nor received from the server; and

presenting the filtered video stream live at the electronic client device;

wherein applying the filter to the second video frame comprises;

identifying a first one or more image features in the first video frame via the processor at the electronic client device;

propagating the first one or more image features to the second video frame;

identifying a second one or more image features in the second video frame via the processor at the electronic client device; and

identifying a correspondence between the first one or more image features and the second one or more image features.

2. The method recited in claim 1, wherein the raw video stream is a camera view associated with a camera operating at the electronic client device.

3. The method recited in claim 1, the method further comprising:

identifying a respective one or more image features in each of one or more intervening video frames, each of the intervening video frames temporally located between the first video frame and the second video frame in the video stream, wherein creating the filtered video stream involves identifying a correspondence between the image features in each successive video frame from the first video frame through the intervening video frames to the second video frame.

4. The method recited in claim 1, wherein the filtered video stream is presented at an augmented reality or virtual reality device in communication with the electronic client device.

5. The method recited in claim 1, wherein the raw video stream includes spatial information selected from the group consisting of:

depth information, visual flow between a plurality of viewpoints, and three-dimensional location information.

6. The method recited in claim 1, wherein the raw video stream includes scale information estimated using accelerometer information or inertial measurement unit (IMU) data.

7. An electronic client device comprising:

a communications interface that automatically transmits a first video frame in a raw video stream to a server via a communication network in response to determining the first video frame meets a designated criterion and receives from the server a filter processing message associated with the first video frame, the filter processing message including filter data for applying a filter to the first video frame, the raw video stream being captured live by the electronic client device; and

a processor that creates a filtered video stream in real time by applying the filter to a second video frame, wherein the first video frame precedes the second video frame in the raw video stream, wherein the filter is applied to the second video frame by propagating information from the first video frame to the second video frame based on the filter data, wherein the second video frame is neither transmitted to nor received from the server; and

a display screen that presents the filtered video stream live at the electronic client device;

wherein by applying the filter to the second video frame, the processor further;

identifies a first one or more image features in the first video frame via the processor at the electronic client device;

propagates the first one or more image features to the second video frame;

identifies a second one or more image features in the second video frame via the processor at the electronic client device; and

identifies a correspondence between the first one or more image features and the second one or more image features.

8. The electronic client device recited in claim 7, wherein the raw video stream is a camera view associated with a camera operating at the electronic client device.

9. The electronic client device recited in claim 7, wherein the processor further identifies a respective one or more image features in each of one or more intervening video frames, each of the intervening video frames temporally located between the first video frame and the second video frame in the video stream, wherein the correspondence between the first one or more image features and the second one or more image features comprises identifying a correspondence between the image features in each successive video frame from the first video frame through the intervening video frames to the second video frame.

10. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

automatically transmitting, using an electronic device, a first video frame in a raw video stream from the electronic client device to a server via a communication network in response to determining the first video frame meets a designated criterion, the raw video stream being captured live by the electronic client device;

receiving, by the electronic client device, from the server a filter processing message associated with the first video frame, the filter processing message including filter data for applying a filter to the first video frame;

creating a filtered video stream in real time via a processor at the electronic client device by applying the filter to a second video frame, wherein the first video frame precedes the second video frame in the video stream, wherein the filter is applied to the second video frame by propagating information from the first video frame to the second video frame based on the filter data, wherein the second video frame is neither transmitted to nor received from the serve; and

presenting the filtered video stream live at the client device presenting the filtered video stream live at the electronic client device;

wherein applying the filter to the second video frame comprises;

identifying a first one or more image features in the first video frame via the processor at the electronic client device;

propagating the first one or more image features to the second video frame;

identifying a second one or more image features in the second video frame via the processor at the electronic client device; and

identifying a correspondence between the first one or more image features and the second one or more image features.

11. The one or more non-transitory computer readable media recited in claim 10, wherein the raw video stream is a camera view associated with a camera operating at the electronic client device.