Behavioral directional encoding of three-dimensional video

• US 10,681,342 B2
• Filed: 04/17/2019
• Issued: 06/09/2020
• Est. Priority Date: 09/19/2016
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Walkingshaw, Andrew David
• Primary Examiner(s): Chio, Tat C
• Application Number:
  US16/387,376
• Publication Number:
  US 20190246096A1
• Time in Patent Office: 419 Days
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A method includes receiving head-tracking data that describes one or more positions of one or more people while the one or more people are viewing a three-dimensional video. The method further includes generating video segments from the three-dimensional video. The method further includes, for each of the video segments: determining a directional encoding format that projects latitudes and longitudes of locations of a surface of a sphere onto locations on a plane, determining a cost function that identifies a region of interest on the plane based on the head-tracking data, and generating optimal segment parameters that minimize a sum-over position for the region of interest.

Claims

1. A method comprising:

receiving head-tracking data that includes yaw, pitch, and roll values that describe one or more positions of people as a function of time while the people are viewing a three-dimensional video;

generating one or more user profiles of one or more persons of the people based on characteristics associated with the people;

generating video segments from the three-dimensional video; and

for each of the video segments for the one or more user profiles of the one or more persons;

determining a directional encoding format that projects latitudes and longitudes of locations of a surface of a sphere onto locations on a plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons;

determining a cost function that identifies a region of interest on the plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons; and

generating optimal segment parameters for the one or more user profiles that minimize a sum-over position for the region of interest.

2. The method of claim 1, further comprising:

determining a user profile from the one or more user profiles that corresponds to a user;

re-encoding the three-dimensional video to include the optimal segment parameters for each of the video segments based on the user profile; and

providing a re-encoded video and the optimal segment parameters for each of the video segments to a client device, wherein the client device uses the re-encoded video and the optimal segment parameters for each of the video segments to generate a two-dimensional video that automates head movement.

3. The method of claim 2, further comprising:

providing the user with an option to modify the two-dimensional video by at least one of selecting different optimal segment parameters and selecting a different region of interest for one or more of the video segments.

4. The method of claim 1, wherein the one or more user profiles are generated based on at least demographic information associated with the people and an amount of movement associated with the people.

5. The method of claim 1, further comprising:

cropping the region of interest for one or more video segments based on the optimal segment parameters to form one or more thumbnails of one or more cropped regions of interest; and

generating a timeline of the three-dimensional video with the one or more thumbnails.

6. The method of claim 1, further comprising:

generating a two-dimensional video from the three-dimensional video based on the optimal segment parameters that depicts head tracking movement as automatic panning within the two-dimensional video.

7. The method of claim 1, wherein the cost function incorporates hinge loss.

8. The method of claim 1, wherein generating the video segments from the three-dimensional video includes generating equal-length video segments of a predetermined length.

9. The method of claim 1, wherein generating the video segments from the three-dimensional video includes generating the video segments based on a detection of scene boundaries in the three-dimensional video.

10. A system comprising:

one or more processors coupled to a memory;

a head-tracking module stored in the memory and executable by the one or more processors, the head-tracking module operable to receive head-tracking data that includes yaw, pitch, and roll values that describe one or more positions of people as a function of time while the people are viewing a three-dimensional video and generate one or more user profiles of one or more persons of the people based on characteristics associated with the people;

a segmentation module stored in the memory and executable by the one or more processors, the segmentation module operable to generate video segments from the three-dimensional video; and

a parameterization module stored in the memory and executable by the one or more processors, the parameterization module operable to, for each of the video segments for the one or more user profiles of the one or more persons;

determine a directional encoding format that projects latitudes and longitudes of locations of a surface of a sphere onto locations on a plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons;

determine a cost function that identifies a region of interest on the plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons; and

generate optimal segment parameters for the one or more user profiles that minimize a sum-over position for the region of interest.

11. The system of claim 10, wherein head-tracking module is further operable to:

determine a user profile from the one or more user profiles that corresponds to a user;

re-encode the three-dimensional video to include the optimal segment parameters for each of the video segments based on the user profile; and

provide a re-encoded video and the optimal segment parameters for each of the video segments to a client device, wherein the client device uses the re-encoded video and the optimal segment parameters for each of the video segments to generate a two-dimensional video that automates head movement.

12. The system of claim 10, further comprising a user interface module stored in the memory and executable by the one or more processors, the user interface module operable to:

provide the user with an option to modify the two-dimensional video by at least one of selecting different optimal segment parameters and selecting a different region of interest for one or more of the video segments.

13. The system of claim 10, wherein the one or more user profiles are generated based on at least demographic information associated with the people and an amount of movement associated with the people.

14. The system of claim 10, further comprising a user interface module stored in the memory and executable by the one or more processors, the user interface module operable to:

crop the region of interest for one or more video segments based on the optimal segment parameters to form one or more thumbnails of one or more cropped regions of interest; and

generate a timeline of the three-dimensional video with the one or more thumbnails.

15. The system of claim 10, wherein the segmentation module is configured to generate the video segments from the three-dimensional video by generating equal-length video segments of a predetermined length.

16. The system of claim 10, wherein the segmentation module is configured to generate the video segments from the three-dimensional video by generating the video segments based on a detection of scene boundaries in the three-dimensional video.

17. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving head-tracking data that includes yaw, pitch, and roll values that describe one or more positions of people as a function of time while the people are viewing a three-dimensional video;

generating one or more user profiles of one or more persons of the people based on characteristics associated with the people;

generating video segments from the three-dimensional video; and

for each of the video segments for the one or more user profiles of the one or more persons;

determining a directional encoding format that projects latitudes and longitudes of locations of a surface of a sphere onto locations on a plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons;

determining a cost function that identifies a region of interest on the plane based on the yaw, pitch, and roll values from the head-tracking data for the one or more persons; and

generating optimal segment parameters for the one or more user profiles that minimize a sum-over position for the region of interest.

18. The computer storage medium of claim 17, wherein the operations further comprise:

determining a user profile from the one or more user profiles that corresponds to a user;

re-encoding the three-dimensional video to include the optimal segment parameters for each of the video segments based on the user profile; and

providing a re-encoded video and the optimal segment parameters for each of the video segments to a client device, wherein the client device uses the re-encoded video and the optimal segment parameters for each of the video segments to generate a two-dimensional video that automates head movement.

19. The computer storage medium of claim 18, wherein the operations further comprise:

providing the user with an option to modify the two-dimensional video by at least one of selecting different optimal segment parameters and selecting a different region of interest for one or more of the video segments.

20. The computer storage medium of claim 17, wherein generating the video segments from the three-dimensional video includes generating equal-length video segments of a predetermined length.