Systems and methods for provisioning content using barrel projection representation

• US 10,579,898 B2
• Filed: 04/13/2018
• Issued: 03/03/2020
• Est. Priority Date: 04/16/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Kuzyakov, Evgeny V., Peng, Renbin, Chen, Chien-Nan
• Primary Examiner(s): Demeter, Hilina K
• Application Number:
  US15/953,164
• Publication Number:
  US 20180300848A1
• Time in Patent Office: 690 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 2207/20068 Projection on vertical or horizontal image axis
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Systems, methods, and non-transitory computer-readable media can determine at least one salient point of interest in a frame of a content item based at least in part on a saliency prediction model, the saliency prediction model being trained to identify salient points of interest that appear in content items; determine a barrel projection representation for the frame; and apply a view-based projection to the barrel projection representation for the frame, wherein the view-based projection enhances a quality in which a region corresponding to the at least one salient point of interest is presented.

Claims

1. A computer-implemented method comprising:

determining, by a computing system, at least one salient point of interest in a frame of a content item based at least in part on a saliency prediction model, the saliency prediction model being trained to identify salient points of interest that appear in content items;

generating, by the computing system, a barrel projection representation for the frame, wherein the barrel projection representation comprises portions including a circular top portion, a circular bottom portion, and a middle portion, wherein each of the portions includes at least one pixel of the frame; and

applying, by the computing system, a view-based projection to the barrel projection representation for the frame, wherein the view-based projection enhances a quality in which a region corresponding to the at least one salient point of interest is presented.

2. The computer-implemented method of claim 1, wherein the plurality of distinct portions comprises a top portion of the frame associated with a first circular projection, a bottom portion of the frame associated with a second circular projection, and a middle portion of the frame associated with a rectangular projection.

3. The computer-implemented method of claim 2, wherein the middle portion of the frame represents a middle 90 degrees field-of-view of a scene represented in the frame.

4. The computer-implemented method of claim 2, wherein applying the view-based projection to the barrel projection representation of the frame further comprises:

increasing, by the computing system, a pixel density associated with a region in the middle portion of the frame that corresponds to the at least one salient point of interest.

5. The computer-implemented method of claim 4, wherein the region is stretched horizontally by a threshold amount.

6. The computer-implemented method of claim 2, wherein applying the view-based projection to the barrel projection representation of the frame further comprises:

decreasing, by the computing system, respective pixel densities associated with one or more other regions in the middle portion of the frame.

7. The computer-implemented method of claim 6, wherein the quality is enhanced in a portion of the plurality of distinct portions in the region.

8. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform;

determining at least one salient point of interest in a frame of a content item based at least in part on a saliency prediction model, the saliency prediction model being trained to identify salient points of interest that appear in content items;

generating a barrel projection representation for the frame, wherein the barrel projection representation comprises portions including a circular top portion, a circular bottom portion, and a middle portion, wherein each of the portions includes at least one pixel of the frame; and

applying a view-based projection to the barrel projection representation for the frame, wherein the view-based projection enhances a quality in which a region corresponding to the at least one salient point of interest is presented.

9. The system of claim 8, wherein the plurality of distinct portions comprises a top portion of the frame associated with a first circular projection, a bottom portion of the frame associated with a second circular projection, and a middle portion of the frame associated with a rectangular projection.

10. The system of claim 9, wherein the middle portion of the frame represents a middle 90 degrees field-of-view of a scene represented in the frame.

11. The system of claim 9, wherein applying the view-based projection to the barrel projection representation of the frame further causes the system to perform:

increasing a pixel density associated with a region in the middle portion of the frame that corresponds to the at least one salient point of interest.

12. The system of claim 11, wherein the region is stretched horizontally by a threshold amount.

13. The system of claim 9, wherein applying the view-based projection to the barrel projection representation of the frame further causes the system to perform:

decreasing respective pixel densities associated with one or more other regions in the middle portion of the frame.

14. The system of claim 11, wherein the quality is enhanced in a portion of the plurality of distinct portions in the region.

15. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:

determining at least one salient point of interest in a frame of a content item based at least in part on a saliency prediction model, the saliency prediction model being trained to identify salient points of interest that appear in content items;

generating a barrel projection representation for the frame, wherein the barrel projection representation comprises portions including a circular top portion, a circular bottom portion, and a middle portion, wherein each of the portions includes at least one pixel of the frame; and

applying a view-based projection to the barrel projection representation for the frame, wherein the view-based projection enhances a quality in a portion of the plurality of distinct portions in which a region corresponding to the at least one salient point of interest is presented.

16. The non-transitory computer-readable storage medium of claim 15, wherein the plurality of distinct portions comprises a top portion of the frame associated with a first circular projection, a bottom portion of the frame associated with a second circular projection, and a middle portion of the frame associated with a rectangular projection.

17. The non-transitory computer-readable storage medium of claim 16, wherein the middle portion of the frame represents a middle 90 degrees field-of-view of a scene represented in the frame.

18. The non-transitory computer-readable storage medium of claim 16, wherein applying the view-based projection to the barrel projection representation of the frame further causes the computing system to perform:

increasing a pixel density associated with a region in the middle portion of the frame that corresponds to the at least one salient point of interest.

19. The non-transitory computer-readable storage medium of claim 16, wherein applying the view-based projection to the barrel projection representation of the frame further causes the system to perform:

decreasing respective pixel densities associated with one or more other regions in the middle portion of the frame.

20. The non-transitory computer-readable storage medium of claim 15, wherein the applying the view-based projection to the barrel projection representation for the frame enhances the quality in the portion of the plurality of distinct portions by allocating increased number of pixels to the region.