Method and system for 360-degree video playback

• US 10,462,449 B2
• Filed: 08/02/2017
• Issued: 10/29/2019
• Est. Priority Date: 02/23/2017
• Status: Active Grant

• Current Assignee: Novatek Microelectronics Corporation (Novatek Microelectronics Corp Limited)
• Original Assignee: Novatek Microelectronics Corporation (Novatek Microelectronics Corp Limited)
• Inventors: Sun, Min, Hu, Hou-Ning, Lin, Yen-Chen, Cheng, Hsien-Tzu, Chen, Chieh-Cheng, Wang, Hung-Ming
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US15/666,589
• Publication Number:
  US 20180241984A1
• Time in Patent Office: 818 Days
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: H04N 13/144 for flicker reduction
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 9/87 Regeneration of colour television signals H04N9/80 takes precedence

Abstract

The disclosure proposes a method and a system for 360-degree video playback. The method is applicable to a video playback system and includes at least the following steps. A current frame of a 360-degree video having a sequence of frames is received. Candidate objects in the current frame are detected, and a main object is selected from the candidate objects by using a selector recurrent neural network (RNN) model based on information of the candidate objects in the current frame. A viewing angle corresponding to the current frame is computed by using a regressor RNN model based on the main object in the current frame. The current frame is displayed on the screen according to the viewing angle.

Claims

1. A method for 360-degree video playback, applicable to a video playback system having a screen, comprising:

receiving a current frame of a 360-degree video having a sequence of frames;

detecting a plurality of candidate objects in the current frame;

selecting a main object from the candidate objects by using a selector recurrent neural network (RNN) model based on information of the candidate objects in the current frame and a previous frame of the current frame comprising;

computing a current state of the selector RNN model corresponding to the current frame based on the information of each of the candidate objects and a previous state of the selector RNN model corresponding to the previous frame; and

classifying the main object from the candidate objects according to the current state of the selector RNN model;

computing a viewing angle corresponding to the current frame by using a regressor RNN model based on the main object in the current frame and the previous frame comprising;

obtaining an action of the main object in the current frame according to a position of the main object and a viewing angle corresponding to the previous frame and obtaining a motion feature of the main object in the current frame;

computing a current state of the regressor RNN model corresponding to the current frame based on the action and the motion feature of the main object in the current frame and a previous state of the regressor RNN model corresponding to the previous frame; and

computing the viewing angle corresponding to the current frame according to the current state of the regressor RNN model; and

displaying the current frame on the screen according to the viewing angle.

2. The method according to claim 1, wherein the step of detecting the candidate objects in the current video frame comprises:

detecting a plurality of foreground objects in the current video frame; and

setting the foreground objects as the candidate objects.

3. The method according to claim 1, wherein the information of each of the candidate objects comprises an appearance feature, a motion feature, and a position of each of the candidate objects.

4. The method according to claim 1, wherein the step of classifying the main object from the candidate objects according to the current state of the selector RNN model comprises:

mapping the current state of the selector RNN model to a probability distribution according to assigned weights of the candidate objects by using a softmax layer; and

classifying the main object according to the probability distribution.

5. The method according to claim 1, wherein the step of computing the viewing angle corresponding to the current frame according to the current state of the regressor RNN model comprises:

mapping the current state of the regressor RNN model to a refined action of the main object according to an assigned weight in associated with a moving speed of the main object; and

computing the viewing angle corresponding to the current frame according to the refined action of the main object in the current frame and the viewing angle corresponding to the previous frame.

6. The method according to claim 1, wherein before the step of receiving any frame of the 360-degree video, the method further comprises:

training the selector RNN model and the regressor RNN model by using a plurality of 360-degree videos comprising human-identifiable objects of interest.

7. The method according to claim 1 further comprising:

displaying another frame on the screen according to another viewing angle based on a viewer'"'"'s input.

8. The method according to claim 1, wherein the step of displaying the current frame on the screen according to the viewing angle comprises:

displaying only a guided field of the current frame on the screen, wherein the guided field is a natural field of view with respect to the viewing angle of the current frame.

9. The method according to claim 8, wherein the prompt object indicates a location of the guided field.

10. The method according to claim 8, wherein the prompt object directs a viewer to the guided field according to a location of a viewer'"'"'s gaze relative to the screen.

11. The method according to claim 1, wherein the step of displaying the current frame on the screen according to the viewing angle comprises:

displaying a prompt object along with the current frame on the screen according to a guided field of the current frame, wherein the guided field is a natural field of view with respect to the viewing angle of the current frame.

12. A system for 360-degree video playback comprising:

a screen;

a data storage device; and

a processor, coupled to the screen and the data storage device, wherein the processor is configured to;

receive a current frame of a 360-degree video having a sequence of frames;

detect a plurality of candidate objects in the current frame;

select a main object from the candidate objects by using a selector recurrent neural network (RNN) model based on information of the candidate objects in the current frame and a previous frame of the current frame by being configured to;

compute current state of the selector RNN model corresponding to the current frame based on the information of each of the candidate objects and a previous state of the selector RNN model corresponding to the previous frame; and

classify the main object from the candidate objects according to the current state of the selector RNN model;

compute a viewing angle corresponding to the current frame by using a regressor RNN model based on the main object in the current frame and the previous frame by being configured to;

obtain an action of the main object in the current frame according to a position of the main object and a viewing angle corresponding to the previous frame and obtaining the motion feature of the main object in the current frame;

compute a current state of the regressor RNN model corresponding to the current frame based on the action and a motion feature of the main object in the current frame and a previous state of the regressor RNN model corresponding to the previous frame; and

compute the viewing angle corresponding to the current frame according to the current state of the regressor RNN model; and

display the current frame on the screen according to the viewing angle.

13. The system according to claim 12, wherein the information of each of the candidate objects comprises an appearance feature, a motion feature, and a position of each of the candidate objects.

14. The system according to claim 12 further comprising:

an input device, configured to receive an input of a viewer, wherein the processor is further coupled to the input device and configured to;

displaying another frame on the screen according to another viewing angle based on the input of the viewer.

15. The system according to claim 12, wherein the processor is configured to:

display only a guided field of the current frame on the screen, wherein the guided field is a natural field of view with respect to the viewing angle of the current frame.

16. The system according to claim 12, wherein the processor is configured to:

displaying a prompt object along with the current frame on the screen according to a guided field of the current frame, wherein the guided field is a natural field of view with respect to the viewing angle of the current frame.

17. The system according to claim 16, wherein the processor is configured to:

display the prompt object that indicates a location of the guided field on the current frame.

18. The system according to claim 16 further comprising:

an eye tracking device, configured to determine a location of a gaze of a viewer relative to the screen, wherein the processor is coupled to the eye tracking device and configured to;

display the prompt object that directs a viewer to the guided field on the current frame according to the location of the gaze of the viewer relative to the screen.