Systems and methods for videoconferencing input and display management based on activity

• US 9,491,374 B1
• Filed: 12/11/2015
• Issued: 11/08/2016
• Est. Priority Date: 12/11/2015
• Status: Expired due to Fees

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Everdingen, Eveline van, Avrahami, Daniel, Marlow, Jennifer
• Primary Examiner(s): Laekemariam, Yosef K
• Application Number:
  US14/966,365
• Time in Patent Office: 333 Days
• Field of Search: 348/14.01, 348/14.08, 348/14.09, 348/14.1, 348/211.12, 379/93.21, 379/158, 379/202.01, 379/205.01
• US Class Current: 1/1
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Systems and methods of managing a videoconference viewed by a first camera coupled to a first display are provided, including, for a face not detected by the first camera and a second camera coupled to a second display in a user environment, performing an action based on off-display analysis; for a face direction less head-on in the first camera than in the second camera, switching the videoconference to the second camera; and for the face direction not detectable in the first camera, and on-display user activity at the second display, switching the videoconference to the second camera.

Claims

1. A method of managing a videoconference viewed by a first camera coupled to a first display, the method comprising:

for a face not detected by the first camera and a second camera coupled to a second display in a user environment, performing an action based on off-display analysis;

for a face direction that is less head-on in the first camera than in the second camera, switching the videoconference to the second camera; and

for the face direction not detectable in the first camera, and on-display user activity detected at the second display, based on a time duration that the on-display user activity will be accessed, as estimated by applying a rule based on a function of the on-display user activity accessed at the second display, being greater than a prescribed period, switching the videoconference to the second camera.

2. The method of claim 1, wherein the on-display user activity comprises at least one of an input/output event and an application event.

3. The method of claim 2, wherein the input/output event comprises one or more of a pointing device action and a keyboard action, associated with cursor activity on the second display.

4. The method of claim 2, wherein the application event comprises:

determining that the function is associated with an online application being accessed at the second display;

estimating the time duration that the online application will be accessed by applying the rule that is based on the function of the application; and

based on the estimated time duration being greater than the prescribed period, performing the switching the videoconference to the second camera.

5. The method of claim 1, wherein the switching the videoconference to the second camera further comprises:

determining to switch to the second camera;

confirming that a first prescribed time period has elapsed since the determining;

based on the confirming, switching an input signal into the videoconference from the first camera to the second camera; and

outputting a video to the videoconference via the second camera.

6. The method of claim 1, wherein the switching the videoconference to the second camera further comprises:

determining to switch to the second camera;

confirming that no other switching operation was performed on the first camera and the second camera for a prescribed time prior to the determining;

based on the confirming, switching an input signal into the videoconference from the first camera to the second camera; and

outputting a video to the videoconference via the second camera.

7. The method of claim 1, wherein an environment of the user of the first camera and the second camera is annotated with at least one of a positive annotation indicative of a surface or object in the environment that is to be displayed in the videoconference, and a negative annotation indicative of a surface or object in the environment that is not to be displayed in the videoconference.

8. The method of claim 7, wherein the positive annotation is associated with a display board in the environment that is not coupled to the first camera and the second camera, and the negative annotation is associated with a doorway or window region not to be displayed, or another user.

9. The method of claim 1, wherein the off-display activity comprises:

analyzing a video of an off-display environment of the first camera and the second camera;

detecting a focus area to display in the videoconference; and

performing the action, wherein the action comprises determining to switch the videoconference from the first camera to the second camera when the second camera has a view of the focus area with less obstruction and a higher degree of view than the first camera.

10. The method of claim 1, wherein the switching the videoconference to the second camera is not performed if the face direction is less head-on in the first camera than in the second camera, or if the on-display user activity occurs at the second display, for less than a prescribed time period.

11. A system for managing a video conference, comprising:

a first camera coupled to a first display and a second camera coupled to a second display in a user environment;

a user environment adjacent to the first camera and the second camera, the user environment including at least one physical display surface not coupled to the first camera and the second camera; and

a processor configured to;

perform an off-display analysis when a face is not detected by the first camera and the second camera, and perform an action based on the off-display analysis,

switch the videoconference to the second camera when (a) a face direction is less head-on in the first camera than in the second camera, or (b) the face direction is not detectable in the first camera and on-display user activity is detected at the second display based on a time duration that the on-display user activity will be accessed, as estimated by applying a rule based on a function of the on-display user activity accessed at the second display, being greater than a prescribed period.

12. The system of claim 11, wherein the on-display user activity comprises one or more of a pointing device action and a keyboard action associated with cursor activity on the second display, or an application event that comprises the processor:

determining that the function is associated with an online application being accessed at the second display;

estimating the time duration that the online application will be accessed by applying a rule that is based on the function of the application; and

based on the estimated time duration greater than the prescribed period, performing the switching the videoconference to the second camera.

13. The system of claim 11, wherein the processor switches the videoconference to the second camera by determining to switch to the second camera, confirming that a first prescribed time period has elapsed since the determining, or that no other switching operation was performed on the first camera and the second camera for a prescribed time prior to the determining, based on the confirming, switching an input signal into the videoconference from the first camera to the second camera, and outputting a video to the videoconference via the second camera.

14. The system of claim 11, wherein an environment of the user of the first camera and the second camera is annotated with at least one of a positive annotation indicative of a surface or object in the environment that is to be displayed in the videoconference, and a negative annotation indicative of a surface or object in the environment that is not to be displayed in the videoconference.

15. The system of claim 11, wherein the off-display activity comprises:

analyzing a video of an off-display environment of the first camera and the second camera;

detecting a focus area to display in the videoconference; and

performing the action, wherein the action comprises determining to switch the videoconference from the first camera to the second camera when the second camera has a view of the focus area with less obstruction and a higher degree of view than the first camera.

16. A non-transitory computer readable medium, storing instructions for managing videoconference input and display, the instructions comprising:

for a face not detected by the first camera and a second camera coupled to a second display in a user environment, performing an action based on off-display analysis;

for a face direction that is less head-on in the first camera than in the second camera, switching the videoconference to the second camera; and

for the face direction not detectable in the first camera, and on-display user activity detected at the second display, based on a time duration that the on-display user activity will be accessed, as estimated by applying a rule based on a function of the on-display user activity accessed at the second display, being greater than a prescribed period, switching the videoconference to the second camera.

17. The non-transitory computer readable medium of claim 16, wherein the on-display user activity comprises:

determining that the function is associated with an online application being accessed at the second display;

estimating the time duration that the online application will be accessed by applying a rule that is based on the function of the application; and

based on the estimated time duration greater than the prescribed period, performing the switching the videoconference to the second camera.

18. The non-transitory computer readable medium of claim 16, wherein the switching the videoconference to the second camera further comprises:

determining to switch to the second camera;

confirming that a first prescribed time period has elapsed since the determining, or that no other switching operation has been performed for the first camera and the second camera for a prescribed time prior to the determining;

based on the confirming, switching an input signal into the videoconference from the first camera to the second camera; and

outputting a video to the videoconference via the second camera.

19. The non-transitory computer readable medium of claim 16, wherein an environment of the user of the first camera and the second cis annotated with at least one of a positive annotation indicative of a surface or object in the environment that is to be displayed in the videoconference, and a negative annotation indicative of a surface or object in the environment that is not to be displayed in the videoconference.

20. The non-transitory computer readable medium of claim 16, wherein the off-display activity comprises:

analyzing a video of an off-display environment of the first camera and the second camera;

detecting a focus area to display in the videoconference; and

performing the action, wherein the action comprises determining to switch the videoconference from the first camera to the second camera when the second camera has a view of the focus area with less obstruction and a higher degree of view than the first camera.

21. A method of managing a videoconference viewed by a first camera coupled to a first display, the method comprising:

for a face not detected by the first camera and a second camera coupled to a second display in a user environment, performing an action based on off-display analysis;

for a face direction that is less head-on in the first camera than in the second camera, switching the videoconference to the second camera; and

for the face direction not detectable in the first camera, and on-display user activity detected at the second display, switching the videoconference to the second camera,

wherein the on-display user activity comprises at least one of an input/output event and an application event that comprises,

determining that a function of an online application is being accessed at the second display,

estimating a time duration that the online application will be accessed by applying a rule that is based on the function of the application, and

based on the estimated time duration being greater than a prescribed period, performing the switching the videoconference to the second camera.

22. A system for managing a video conference, comprising:

a first camera coupled to a first display and a second camera coupled to a second display in a user environment;

a user environment adjacent to the first camera and the second camera, the user environment including at least one physical display surface not coupled to the first camera and the second camera; and

a processor configured to;

perform an off-display analysis when a face is not detected by the first camera and the second camera, and perform an action based on the off-display analysis,

switch the videoconference to the second camera when (a) a face direction is less head-on in the first camera than in the second camera, or (b) the face direction is not detectable in the first camera and on-display user activity is detected at the second display,

wherein the on-display user activity comprises one or more of a pointing device action and a keyboard action associated with cursor activity on the second display, or an application event that comprises the processor;

determining that a function of an online application is being accessed at the second display,

estimating a time duration that the online application will be accessed by applying a rule that is based on the function of the application, and

based on the estimated time duration greater than a prescribed period, performing the switching the videoconference to the second camera.

23. non-transitory computer readable medium, storing instructions for managing videoconference input and display, the instructions comprising:

for a face not detected by the first camera and a second camera coupled to a second display in a user environment, performing an action based on off-display analysis;

for a face direction that is less head-on in the first camera than in the second camera, switching the videoconference to the second camera; and

for the face direction not detectable in the first camera, and on-display user activity detected at the second display, switching the videoconference to the second camera wherein the on-display user activity comprises,

determining that a function of an online application is being accessed at the second display,

estimating a time duration that the online application will be accessed by applying a rule based on the function of the application, and

based on the estimated time duration greater than a prescribed period, performing the switching the videoconference to the second camera.