Automated cinematic decisions based on descriptive models

• US 10,523,864 B2
• Filed: 10/05/2018
• Issued: 12/31/2019
• Est. Priority Date: 04/10/2018
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Harrison, Jason Francis, Hwang, Eric W., Nallamothu, Rahul, Razzaq, Shahid
• Primary Examiner(s): Woo, Stella L.
• Application Number:
  US16/153,595
• Publication Number:
  US 20190313013A1
• Time in Patent Office: 452 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20044 Skeletonization; Medial axis transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/45455 applied to a region of the image
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/695 : Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/155 : involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

In one embodiment, a method includes identifying, from a set of coordinate points that correspond to a plurality of surface points of a person in an environment, a coordinate point that corresponds to a facial feature of the person; generating a facial structure for a face of the person, wherein the facial structure: covers a plurality of facial features of the person; and substantially matches a pre-determined facial structure; generating a body skeletal structure for the person, wherein the body skeletal structure substantially matches a predetermined body skeletal structure and substantially aligns with the generated facial structure in at least one dimension of a two-dimensional coordinate plane; and associating the generated body skeletal structure and facial structure with the person in the environment.

Claims

1. A method comprising, by a computing device:

identifying, from a set of coordinate points that correspond to a plurality of surface points of a person in an environment, one or more coordinate points that correspond to one or more facial features of the person;

generating a facial structure associated with the person based on the one or more coordinate points, wherein the facial structure maps the one or more facial features, wherein generating the facial structure comprises;

comparing the one or more coordinate points to a pre-determined facial structure to determine a likelihood that the pre-determined facial structure corresponds to a facial structure of the person;

generating a body skeletal structure for the person based on the generated facial structure, wherein generating the body skeletal structure comprises;

aligning a candidate body skeletal structure with the generated facial structure; and

comparing the candidate body skeletal structure to a pre-determined body skeletal structure to determine a likelihood that the pre-determined body skeletal structure corresponds to a body skeletal structure of the person;

associating the generated body skeletal structure and facial structure with the person in the environment; and

determining one or more instructions for a camera, microphone, speaker, or display screen based on the generated body skeletal structure and facial structure.

2. The method of claim 1, wherein generating the facial structure associated with the person based on the one or more coordinate points further comprises:

determining a matching score for the generated facial structure, wherein the matching score indicates how closely the facial structure matches the predetermined facial structure.

3. The method of claim 2, further comprising penalizing the generated facial structure if the matching score is below a threshold score.

4. The method of claim 1, wherein the facial structure is a first facial structure, the method further comprising:

comparing the first facial structure to a second facial structure generated by the computing device;

determining that the first facial structure is identical to the second facial structure; and

associating the second facial structure with the person in the environment.

5. The method of claim 1, wherein the person in the environment is a participant in an audio-video communication session.

6. The method of claim 1, wherein the facial structure is a first facial structure, the method further comprising:

determining a first location for the first facial structure;

determining a second location for the second facial structure;

determining that the first location and the second location are identical; and

discarding the second facial structure based on the determination that the first location and the second location are identical.

7. The method of claim 1, wherein the one or more instructions comprise zooming in on the generated facial structure.

8. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

identify, from a set of coordinate points that correspond to a plurality of surface points of a person in an environment, one or more coordinate points that correspond to one or more facial features of the person;

generate a facial structure associated with the person based on the one or more coordinate points, wherein the facial structure maps the one or more facial features, wherein generating the facial structure comprises;

comparing the one or more coordinate points to a pre-determined facial structure to determine a likelihood that the pre-determined facial structure corresponds to a facial structure of the person;

generate a body skeletal structure for the person based on the generated facial structure, wherein generating the body skeletal structure comprises;

aligning the candidate body skeletal structure with the generated facial structure; and

comparing a candidate body skeletal structure to a pre-determined body skeletal structure to determine a likelihood that the pre-determined body skeletal structure corresponds to a body skeletal structure of the person;

associate the generated body skeletal structure and facial structure with the person in the environment; and

determine one or more instructions for a camera, microphone, speaker, or display screen based on the generated body skeletal structure and facial structure.

9. The media of claim 8, wherein, when executed to generate the facial structure associated with the person based on the one or more coordinate points, the software is further operable when executed to:

determine a matching score for the generated facial structure, wherein the matching score indicates how closely the facial structure matches the predetermined facial structure.

10. The media of claim 9, wherein the software is further operable when executed to penalize the generated facial structure if the matching score is below a threshold score.

11. The media of claim 8, wherein the facial structure is a first facial structure, wherein the software is further operable when executed to:

compare the first facial structure to a second facial structure generated by the computing device;

determine that the first facial structure is identical to the second facial structure; and

associate the second facial structure with the person in the environment.

12. The media of claim 8, wherein the person in the environment is a participant in an audio-video communication session.

13. The media of claim 8, wherein the facial structure is a first facial structure, wherein the software is further operable when executed to:

determine a first location for the first facial structure;

determine a second location for the second facial structure;

determine that the first location and the second location are identical; and

discard the second facial structure based on the determination that the first location and the second location are identical.

14. The media of claim 8, wherein the one or more instructions comprise zooming in on the generated facial structure.

15. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to;

identify, from a set of coordinate points that correspond to a plurality of surface points of a person in an environment, one or more coordinate points that correspond to one or more facial features of the person;

generate a facial structure associated with the person based on the one or more coordinate points, wherein the facial structure maps the one or more facial features, wherein generating the facial structure comprises;

comparing the one or more coordinate points to a pre-determined facial structure to determine a likelihood that the pre-determined facial structure corresponds to a facial structure of the person;

generate a body skeletal structure for the person based on the generated facial structure, wherein generating the body skeletal structure comprises;

aligning a candidate body skeletal structure with the generated facial structure; and

comparing the candidate body skeletal structure to a pre-determined body skeletal structure to determine a likelihood that the pre-determined body skeletal structure corresponds to a body skeletal structure of the person;

associate the generated body skeletal structure and facial structure with the person in the environment; and

determine one or more instructions for a camera, microphone, speaker, or display screen based on the generated body skeletal structure and facial structure.

16. The system of claim 15, wherein, when executing the instructions to generate the facial structure associated with the person based on the one or more coordinate points, the processors are further operable when executing the instructions to:

determine a matching score for the generated facial structure, wherein the matching score indicates how closely the facial structure matches the predetermined facial structure.

17. The system of claim 16, wherein the processors are further operable when executing the instructions to penalize the generated facial structure if the matching score is below a threshold score.

18. The system of claim 15, wherein the facial structure is a first facial structure, wherein the processors are further operable when executing the instructions to:

compare the first facial structure to a second facial structure generated by the computing device;

determine that the first facial structure is identical to the second facial structure; and

associate the second facial structure with the person in the environment.

19. The system of claim 15, wherein the person in the environment is a participant in an audio-video communication session.

20. The system of claim 15, wherein the facial structure is a first facial structure, wherein the processors are further operable when executing the instructions to:

determine a first location for the first facial structure;

determine a second location for the second facial structure;

determine that the first location and the second location are identical; and

discard the second facial structure based on the determination that the first location and the second location are identical.