Automated cinematic decisions based on descriptive models

• US 10,659,731 B2
• Filed: 10/05/2018
• Issued: 05/19/2020
• Est. Priority Date: 04/10/2018
• Status: Active Grant

• Current Assignee: Corium Pharma Solutions, Inc.
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Harrison, Jason Francis, Hwang, Eric W., Nallamothu, Rahul, Razzaq, Shahid
• Primary Examiner(s): Woo, Stella L.
• Application Number:
  US16/153,613
• Publication Number:
  US 20190313058A1
• Time in Patent Office: 592 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20044 Skeletonization; Medial axis transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/45455 applied to a region of the image
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/695 : Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/155 : involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

In one embodiment, a method includes accessing input data from one or more different input sources. The input sources include: one or more cameras, one or more microphones, and a social graph maintained by a social-networking system. Based on the input data, generating a current descriptive model for a current audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the current audio-video communication session, (2) one or more people within the environment, or (3) one or more contextual elements associated with the current audio-video communication session. The method also includes generating one or more instructions for the current audio-video communication session that are based the one or more descriptive characteristics; and sending the one or more instructions to a computing device associated with the one or more cameras and the one or more microphones.

Claims

1. A method comprising:

accessing input data from one or more different input sources, the input sources comprising;

one or more cameras, one or more microphones, and a social graph maintained by a social-networking system;

based on the input data, generating a current descriptive model for a current audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the current audio-video communication session, (2) one or more people within the environment, or (3) one or more contextual elements associated with the current audio-video communication session;

generating one or more instructions for the current audio-video communication session that are based the one or more descriptive characteristics; and

sending the one or more instructions to a computing device associated with the one or more cameras and the one or more microphones.

2. The method of claim 1, wherein the one or more descriptive characteristics about the environment associated with the current audio-video communication session comprises:

a location for each of one or more walls within the environment;

a location for each of one or more doors within the environment;

a category of the environment; and

a location for each of one or more objects within the environment.

3. The method of claim 1, wherein the one or more descriptive characteristics about the one or more people comprises, for each person of the one or more people:

an engagement metric with the current audio-video communication session; and

an affinity between the person and at least one other person of the one or more people.

4. The method of claim 3, wherein the engagement metric for each person of the one or more people is based at least in part on a count of words spoken by the person, a physical distance between the person and the computing device, an amount of time the person has been in the environment during the current audio-video communication system, or social graph data associated with the person.

5. The method of claim 1, further comprising generating a historical descriptive model based on a plurality of past audio-video communication sessions that have occurred within the environment, wherein the one or more descriptive characteristics are based at least in part on the historical descriptive model.

6. The method of claim 1, wherein the one or more instructions comprise instructions to zoom in on a visual target, pan toward a visual target, beamform a microphone to an audio target, or cut a scene to a visual target.

7. The method of claim 1, wherein the one or more contextual elements associated with the current audio-video communication session comprises a time of day for the current audio-video communication session, a data of the current audio-video communication session, or an event associated with the current audio-video communication session.

8. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access input data from one or more different input sources, the input sources comprising;

one or more cameras, one or more microphones, and a social graph maintained by a social-networking system;

based on the input data, generate a current descriptive model for a current audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the current audio-video communication session, (2) one or more people within the environment, or (3) one or more contextual elements associated with the current audio-video communication session;

generate one or more instructions for the current audio-video communication session that are based the one or more descriptive characteristics; and

send the one or more instructions to a computing device associated with the one or more cameras and the one or more microphones.

9. The media of claim 8, wherein the one or more descriptive characteristics about the environment associated with the current audio-video communication session comprises:

a location for each of one or more walls within the environment;

a location for each of one or more doors within the environment;

a category of the environment; and

a location for each of one or more objects within the environment.

10. The media of claim 8, wherein the one or more descriptive characteristics about the one or more people comprises, for each person of the one or more people:

an engagement metric with the current audio-video communication session; and

an affinity between the person and at least one other person of the one or more people.

11. The media of claim 10, wherein the engagement metric for each person of the one or more people is based at least in part on a count of words spoken by the person, a physical distance between the person and the computing device, an amount of time the person has been in the environment during the current audio-video communication system, or social graph data associated with the person.

12. The media of claim 8, wherein the software is further operable when executed to generate a historical descriptive model based on a plurality of past audio-video communication sessions that have occurred within the environment, wherein the one or more descriptive characteristics are based at least in part on the historical descriptive model.

13. The media of claim 8, wherein the one or more instructions comprise instructions to zoom in on a visual target, pan toward a visual target, beamform a microphone to an audio target, or cut a scene to a visual target.

14. The media of claim 8, wherein the one or more contextual elements associated with the current audio-video communication session comprises a time of day for the current audio-video communication session, a data of the current audio-video communication session, or an event associated with the current audio-video communication session.

15. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to;

access input data from one or more different input sources, the input sources comprising;

one or more cameras, one or more microphones, and a social graph maintained by a social-networking system;

based on the input data, generate a current descriptive model for a current audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the current audio-video communication session, (2) one or more people within the environment, or (3) one or more contextual elements associated with the current audio-video communication session;

generate one or more instructions for the current audio-video communication session that are based the one or more descriptive characteristics; and

send the one or more instructions to a computing device associated with the one or more cameras and the one or more microphones.

16. The system of claim 15, wherein the one or more descriptive characteristics about the environment associated with the current audio-video communication session comprises:

a location for each of one or more walls within the environment;

a location for each of one or more doors within the environment;

a category of the environment; and

a location for each of one or more objects within the environment.

17. The system of claim 15, wherein the one or more descriptive characteristics about the one or more people comprises, for each person of the one or more people:

an engagement metric with the current audio-video communication session; and

an affinity between the person and at least one other person of the one or more people.

18. The system of claim 17, wherein the engagement metric for each person of the one or more people is based at least in part on a count of words spoken by the person, a physical distance between the person and the computing device, an amount of time the person has been in the environment during the current audio-video communication system, or social graph data associated with the person.

19. The system of claim 15, wherein the processors are further operable when executing the instructions to generate a historical descriptive model based on a plurality of past audio-video communication sessions that have occurred within the environment, wherein the one or more descriptive characteristics are based at least in part on the historical descriptive model.

20. The system of claim 15, wherein the one or more instructions comprise instructions to zoom in on a visual target, pan toward a visual target, beamform a microphone to an audio target, or cut a scene to a visual target.