Virtual window

• US 9,485,459 B2
• Filed: 09/05/2014
• Issued: 11/01/2016
• Est. Priority Date: 12/14/2012
• Status: Expired due to Fees

• Current Assignee: Biscotti Incorporated
• Original Assignee: Biscotti Incorporated
• Inventors: Shoemake, Matthew B., Ahmed, Syed Nadeem
• Primary Examiner(s): Goodarzi, Nasser
• Assistant Examiner(s): PIERORAZIO, MICHAEL
• Application Number:
  US14/479,169
• Publication Number:
  US 20140375752A1
• Time in Patent Office: 788 Days
• Field of Search: 725/10, 348/169
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/14 Systems for two-way working H04N7/12, H04N7/173 take precedence
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Novel tools and techniques are provided for displaying video. In some embodiments, novel tools and techniques might be provided for sensing the presence and/or position of a user in a room, and/or for customizing displayed content (including video call content, media content, and/or the like) based on the sensed presence and/or position of the user. In particular, in some aspects, a user device (which might include, without limitation, a video calling device, an image capture device, a gaming console, etc.) might determine a position of a user relative to a display device in communication with the user device. The user device and/or a control server (in communication with the user device over a network) might adjust an apparent view of video or image(s) displayed on the display device, based at least in part on the determined position of the user relative to the display device.

Claims

1. A method, comprising:

displaying, with a presence detection device (“

PDD”

), video content on a display device that is communicatively coupled to the PDD;

determining, with a processor of the PDD based at least in part on one or more first images of the first user that are captured by at least one camera of the PDD, a first position of a first user relative to a reference point on the display device that displays the video content, the first user viewing the displayed video content, the first position of the first user comprising a first vertical position and a first lateral position, each relative to the reference point on the display device, the first vertical position and the first lateral position of the first user relative to the reference point on the display device providing the first user with a first apparent view of the displayed video content;

determining, with the processor of the PDD based at least in part on one or more second images of the first user that are captured by the at least one camera of the PDD, that position of the first user has changed from the first position to a second position relative to the reference point on the display device, the second position of the first user comprising a second vertical position and a second lateral position, each relative to the reference point on the display device;

calculating, with the processor of the PDD, a difference between the first position and the second position of the first user, by calculating the difference between the first vertical position and the second vertical position and the difference between the first lateral position and the second lateral position, by determining a first vertical direction from the first vertical position to the second vertical position and determining a first lateral direction from the first lateral position to the second lateral position, and by calculating a vertical angle of rotation about the reference point on the display device based on the calculated difference between the first vertical position and the second vertical position and calculating a lateral angle of rotation about the reference point on the display device based on the calculated difference between the first lateral position and the second lateral position;

adjusting, with the processor of the PDD, the first apparent view of the displayed video content to a second apparent view of the displayed video content, based at least in part on the calculated difference between the first position and the second position of the first user, wherein the second apparent view of the displayed video content is rotated by the calculated lateral angle of rotation about the reference point along a second lateral direction and rotated by the calculated vertical angle of rotation about the reference point along a second vertical direction, the second lateral direction being opposite to the first lateral direction with respect to the reference point, and the second vertical direction being opposite to the first vertical direction with respect to the reference point; and

displaying, with the PDD, the second apparent view of the displayed video content on the display device.

2. The method of claim 1, wherein adjusting the first apparent view of the displayed video content comprises adjusting an apparent field of view of the displayed video content.

3. The method of claim 2, wherein the reference point is a face of the display device, and wherein calculating the difference between the first lateral position and the second lateral position comprises determining a first horizontal position of the first user in a horizontal dimension of a plane parallel to the face of the display device based on the first lateral position, determining a second horizontal position of the first user in the horizontal dimension of the plane parallel to the face of the display device, and calculating a difference between the first horizontal position and the second horizontal position.

4. The method of claim 3, wherein adjusting an apparent field of view of the displayed video content comprises panning the video content in a horizontal direction, based on the calculated difference between the first lateral position and the second lateral position.

5. The method of claim 2, wherein the reference point is a face of the display device, and wherein calculating the difference between the first vertical position and the second vertical position comprises determining a third vertical position of the first user in a vertical dimension of a plane parallel to the face of the display device based on the first vertical position, determining a fourth vertical position of the first user in the vertical dimension of the plane parallel to the face of the display device based on the second vertical position, and calculating a difference between the third vertical position and the fourth vertical position.

6. The method of claim 5, wherein adjusting an apparent field of view of the displayed video content comprises panning the video in a vertical direction, based on the calculated difference between the first vertical position and the second vertical position.

7. The method of claim 2, wherein the first position of the first user further comprises a first proximal position that is also relative to the reference point on the display device, the combination of the first vertical position, the first horizontal position, and the first proximal position of the first user relative to the reference point on the display device providing the first user with a first apparent view of the displayed video content, wherein the second position of the first user further comprises a second proximal position that is also relative to the reference point on the display device, wherein calculating the difference between the first position and the second position of the first user comprises calculating, with the processor of the PDD, a difference between the first position and the second position of the first user, by calculating the difference between the first vertical position and the second vertical position, the difference between the first lateral position and the second lateral position, and the difference between the first proximal position and the second proximal position, and by determining a first vertical direction form the first vertical position to the second vertical position, determining a first lateral direction from the first lateral position to the second lateral position, and determining a first proximal direction from the first proximal position to the second proximal position.

8. The method of claim 7, wherein calculating the difference between the first proximal position and the second proximal position comprises determining a first distance of the first user from the reference point on the display device based on the first proximal position, determining a second distance of the first user from the reference point on the display device based on the second proximal position, and calculating a difference between the first distance and the second distance.

9. The method of claim 8, wherein adjusting an apparent field of view of the displayed video content comprises zooming the video content based on the calculated difference between the first proximal position and the second proximal position.

10. The method of claim 7, wherein the second apparent view of the displayed video content is further shifted to have a narrower field of view when the first proximal direction is away from the display device, wherein the second apparent view of the displayed video content is further shifted to have a wider field of view when the first proximal direction is toward the display device.

11. The method of claim 1, wherein adjusting the first apparent view of the displayed video content comprises modifying, at the PDD, a video signal received by the PDD.

12. The method of claim 1, wherein the PDD is a first video calling device, wherein the video content is received from a second video calling device, and wherein adjusting the first apparent view of the displayed video content comprises instructing the second video calling device to adjust a view of one or more cameras of the second video calling device.

13. The method of claim 12, wherein instructing the second video calling device to adjust a view of one or more cameras comprises instructing the second video calling device to adjust a perspective of the one or more cameras.

14. The method of claim 12, wherein instructing the second video calling device to adjust a view of one or more cameras comprises instructing the second video calling device to pan a camera in at least one of a horizontal dimension or a vertical dimension.

15. The method of claim 12, wherein instructing the second video calling device to adjust a view of a camera comprises instructing the second video calling device to zoom a camera.

16. The method of claim 12, wherein instructing the second video calling device to adjust a view of a camera comprises instructing the second video calling device to crop frames of a video stream captured by the camera.

17. The method of claim 12, wherein instructing the second video calling device to adjust a view of one or more cameras comprises instructing the second video calling device to adjust a field of view of the one or more cameras.

18. The method of claim 12, wherein the second video calling device comprises an array of cameras, and wherein the field of view of the one or more cameras comprises a field of view of a composite image captured by a plurality of cameras within the array of cameras.

19. The method of claim 18, wherein the apparent view of the video call comprises a virtual perspective of the composite image.

20. The method of claim 19, wherein the virtual perspective represents a perspective of the first party to the displayed video content relative to the display device.

21. The method of claim 1, wherein the PDD is a video game console, and wherein the displayed video content is at least one of video content, image content, or gaming content from a video game.

22. The method of claim 1, wherein adjusting the first apparent view of the displayed video content comprises adjusting the first apparent view of the displayed video content to the second apparent view of the displayed video content substantially in real time with the determined change in position of the first user from the first position to the second position.

23. The method of claim 1, wherein determining each of the first position and the second position of the first user comprises capturing one or more images of the first user with the camera.

24. The method of claim 23, wherein the one or more images comprise a video stream.

25. The method of claim 24, wherein the PDD is a first video calling device, wherein the method further comprises transmitting the video stream to a second video calling device as part of a video call between the first video calling device and the second video calling device.

26. The method of claim 23, wherein determining each of the first position and the second position of the first user further comprises analyzing the one or more images to identify the corresponding first position and second position of the first user.

27. The method of claim 26, wherein analyzing the one or more images comprises identifying, in the one or more images, positions of one or more eyes of the first user.

28. An apparatus, comprising:

at least one processor;

at least one camera; and

a non-transitory computer readable medium having encoded thereon a set of instructions executable by the at least one processor to cause the apparatus to;

display video content on a display device that is communicatively coupled to the apparatus;

determine, based at least in part on one or more first images of the first user that are captured by the at least one camera, a first position of a first user relative to a reference point on the display device that displays the video content, the first user viewing the displayed video content, the first position of the first user comprising a first vertical position and a first lateral position, each relative to the reference point on the display device, the first vertical position and the first lateral position of the first user relative to the reference point on the display device providing the first user with a first apparent view of the displayed video content;

determine, based at least in part on one or more second images of the first user that are captured by the at least one camera, that position of the first user has changed from the first position to a second position relative to the reference point on the display device, the second position of the first user comprising a second vertical position and a second lateral position, each relative to the reference point on the display device;

calculate the difference between the first position and the second position of the first user, by calculating the difference between the first vertical position and the second vertical position and the difference between the first lateral position and the second lateral position, by determining a first vertical direction from the first vertical position to the second vertical position and determining a first lateral direction from the first lateral position to the second lateral position, and by calculating a vertical angle of rotation about the reference point on the display device based on the calculated difference between the first vertical position and the second vertical position and calculating a lateral angle of rotation about the reference point on the display device based on the calculated difference between the first lateral position and the second lateral position;

adjust the first apparent view of the displayed video content to a second apparent view of the displayed video content, based at least in part on the calculated difference between the first position and the second position of the first user, wherein the second apparent view of the displayed video content is rotated by the calculated lateral angle of rotation about the reference point along a second lateral direction and rotated by the calculated vertical angle of rotation about the reference point along a second vertical direction, the second lateral direction being opposite to the first lateral direction with respect to the reference point, and the second vertical direction being opposite to the first vertical direction with respect to the reference point; and

display the second apparent view of the displayed video content on the display device.

29. The apparatus of claim 28, wherein the apparatus comprises a video calling device.

30. The apparatus of claim 28, wherein the apparatus comprises a video game console.

31. A system, comprising:

a presence detection device (“

PDD”

), comprising;

at least one first processor;

at least one camera;

a first network interface; and

a first non-transitory computer readable medium in communication with the at least one first processor, the first non-transitory computer readable medium having encoded thereon a first set of instructions executable by the at least one first processor to cause the video calling device to;

display video content on a display device that is communicatively coupled to the PDD;

determine, based at least in part on one or more first images of the first user that are captured by the at least one camera, a first position of a first user relative to a reference point on the display device that displays the video content, the first user viewing the displayed video content, the first position of the first user comprising a first vertical position and a first lateral position, each relative to the reference point on the display device, the first vertical position and the first lateral position of the first user relative to the reference point on the display device providing the first user with a first apparent view of the displayed video content;

determine, based at least in part on one or more second images of the first user that are captured by the at least one camera, that position of the first user has changed from the first position to a second position relative to the reference point on the display device, the second position of the first user comprising a second vertical position and a second lateral position, each relative to the reference point on the display device; and

send, over a network via the first network interface, the first position and the second position of the first user to a computer;

the computer, comprising;

one or more second processors;

a second network interface; and

a second non-transitory computer readable medium in communication with the one or more second processors, the second non-transitory computer readable medium having encoded thereon a second set of instructions executable by the one or more second processors to cause the computer to;

receive, over the network via the second network interface, the first position and the second position of the first user;

calculate the difference between the first position and the second position of the first user, by calculating the difference between the first vertical position and the second vertical position and the difference between the first lateral position and the second lateral position, by determining a first vertical direction from the first vertical position to the second vertical position and determining a first lateral direction from the first lateral position to the second lateral position, and by calculating a vertical angle of rotation about the reference point on the display device based on the calculated difference between the first vertical position and the second vertical position and calculating a lateral angle of rotation about the reference point on the display device based on the calculated difference between the first lateral position and the second lateral position;

adjust the first apparent view of the displayed video content to a second apparent view of the displayed video content, based at least in part on the calculated difference between the first position and the second position of the first user, wherein the second apparent view of the displayed video content is rotated by the calculated lateral angle of rotation about the reference point along a second lateral direction and rotated by the calculated vertical angle of rotation about the reference point along a second vertical direction, the second lateral direction being opposite to the first lateral direction with respect to the reference point, and the second vertical direction being opposite to the first vertical direction with respect to the reference point; and

send, over the network and via the second network interface, the second apparent view of the displayed video content to the PDD;

wherein the first set of instructions, when executed by the at least one first processor, cause the video calling device to;

receive, over the network and via the first network interface, the second apparent view of the displayed video content; and

display the second apparent view of the displayed video content on the display device.

32. The system of claim 31, wherein the PDD comprises a video game console.

33. The system of claim 31, wherein the PDD comprises a first video calling device, the system further comprising a second video calling device that comprises a camera that records the video of a second user who is party to a video call established between the first video calling device and the second video calling device.

34. The system of claim 33, wherein adjusting the first apparent view of the displayed video content comprises transmitting, to the second video calling device, instructions for adjusting a field of view of the camera of the second video calling device.

35. The system of claim 33, wherein the computer is a control server separate from the first video calling device and the second video calling device.

36. The system of claim 33, wherein the computer is incorporated within the second video calling device.

37. The system of claim 31, wherein the computer is a control server separate from the PDD.