Remote control operations in a video conference

• US 8,874,090 B2
• Filed: 06/06/2010
• Issued: 10/28/2014
• Est. Priority Date: 04/07/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Abuan, Joe S., Normile, James O., Wu, Hsi-Jung, Guo, Haitao, Price, Douglas S., Zhou, Xiaosong, Zhang, Dazhong, Tung, Berkat S., Garcia, Roberto Jr., Jeong, Hyeonkuk, Yang, Yan, Eldred, David A., Cranfill, Elizabeth C., Yanowitz, Andrew, Chen, Ting, Kuo, David D.
• Primary Examiner(s): El-Zoobi, Maria
• Application Number:
  US12/794,772
• Publication Number:
  US 20110249075A1
• Time in Patent Office: 1,605 Days
• Field of Search: 455/416, 348 1401- 1408, 348/14.09
• US Class Current: 455/416
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04M 1/72469 for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/617 Upgrading or updating of programs or applications for camera control
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems

Abstract

Some embodiments provide a method for allowing a first device that is in a video conference with a second mobile device to remotely control the second mobile device. The method sends images captured by a camera of the first device to the second device. The method receives images captured by a camera of the second device. The method sends a command through a communication channel of a real-time communication session to the second device. The command is for instructing the second device to perform an operation that modifies the images captured by the camera of the second device.

Claims

1. A non-transitory machine-readable medium of a first mobile device, the machine-readable medium storing a program for allowing the first mobile device that is in a video conference with a second mobile device to remotely control the second mobile device, the program comprising sets of instructions for:

sending video images captured by a camera of the first mobile device to the second mobile device for display on the second mobile device during the video conference;

receiving video images captured by a camera of the second mobile device for display on the first mobile device during the video conference;

specifying a region of interest as a portion of the received video images based on a touch input at a location within a received video image displayed on the first mobile device; and

sending, while sending video images captured by the camera of the first mobile device and receiving video images captured by the camera of the second mobile device during the video conference, a command through a communication channel of a real-time communication session to the second mobile device, said command for instructing the second mobile device to perform an operation that uses the specified region of interest to modify the video images captured by the camera of the second mobile device.

2. The non-transitory machine-readable medium of claim 1, wherein the communication channel is a control channel.

3. The non-transitory machine-readable medium of claim 1, wherein the operation is an exposure operation on the region of interest.

4. The non-transitory machine-readable medium of claim 1, wherein the operation is a focus operation on the region interest.

5. The non-transitory machine-readable medium of claim 1, wherein the operation is a zoom operation on the region of interest.

6. The non-transitory machine-readable medium of claim 1, wherein the program further comprises sets of instructions for:

displaying on the first mobile device the video images captured by the camera of the second mobile device; and

displaying on the first mobile device the video images captured by the camera of the first mobile device such that the video images captured by the camera of the second mobile device overlap a portion of the video images captured by the camera of the first mobile device.

7. The non-transitory machine-readable medium of claim 1, wherein the set of instructions for sending the command comprises a set of instructions for sending the command in response to receiving a touch input of a selectable user interface (“

UI”

) tool on the display screen of the first mobile device.

20. The non-transitory machine-readable medium of claim 1, wherein the command is a first command, wherein the program further comprises a set of instructions for receiving, while sending video images captured by the camera of the first mobile device and receiving video images captured by the camera of the second mobile device during the video conference, a second command through the communication channel of the real-time communication session from the second mobile device, the second command for instructing the first mobile device to perform an operation that modifies the images captured by the camera of the first mobile device.

21. The non-transitory machine-readable medium of claim 1, wherein the region of interest is a shape that bounds the portion of the received video images displayed on the first mobile device.

26. The non-transitory machine-readable medium of claim 1, wherein the specified region of interest comprises the location of the touch input.

27. The non-transitory machine-reasonable medium of claim 1, wherein the specified region of interest is centered at the location of the touch input.

8. A non-transitory machine-readable medium of a first mobile device, the machine-readable medium storing a program for allowing the first mobile device during a video conference with a second mobile device to remotely control the second mobile device, the program comprising a graphical user interface (“

GUI”

), the GUI comprising;

a first display area for displaying video images captured by a camera of the first mobile device during a real-time communication session;

a second display area for displaying video images captured by a camera of the second mobile device during the real-time communication session; and

a selectable user interface (“

UI”

) tool for display upon receiving a touch input on a location in the second display area to specify a region of interest as a portion of captured video images, wherein selection of the selectable UI tool causes the first mobile device to send a command to the second mobile device to instruct the second mobile device to perform an operation during the real-time communication session, said operation using the specified region of interest to modify the video images captured by the camera of the second mobile device.

9. The non-transitory machine-readable medium of claim 8, wherein the command is sent through a communication channel of the real-time communication session.

10. The non-transitory machine-readable medium of claim 9, wherein the communication channel is a control channel of the real-time communication session.

11. The non-transitory machine-readable medium of claim 8, wherein the first display area is inset in the second display area.

22. The non-transitory machine-readable medium of claim 8, wherein the region of interest is a shape that bounds the portion of the captured video images displayed on the first mobile device.

23. The non-transitory machine-readable medium of claim 8, wherein the touch input is a tap input on the location to specify the region of interest centered at the location.

24. The non-transitory machine-readable medium of claim 8, wherein the touch input comprises a drag operation across the portion of the video images to specify the region of interest.

25. The non-transitory machine-readable medium of claim 8, wherein the touch input specifies the region of interest by drawing a boundary of the region of interest.

12. A method of defining a program for allowing a first mobile device to remotely control a second mobile device during a video conference, the method comprising:

defining a display area for displaying on the first mobile device a video comprising images captured by a camera of the second mobile device based on a set of settings of the camera comprising an exposure setting that specifies an exposure level at which the camera of the second mobile device captures the images, the set of settings specifying a first mode for the camera of the second mobile device to capture the images at a first exposure level; and

defining a camera adjustment operation for receiving an input on the first mobile device during the video conference that directs the second mobile device to automatically modify, without user intervention on the second mobile device, the set of settings to specify a second mode for the camera of the second mobile device to capture the images at a second exposure level different from the first exposure level.

13. The method of claim 12 further comprising defining a first network manager for the first mobile device that relays the received input to a second network manager of the second mobile device during the video conference.

14. The method of claim 13, wherein the first network manager relays data to the second network manager during the video conference through packets transmitted over the Internet Protocol (IP), said packets including control packets for relaying input data received on the first mobile device for controlling the operation of the second mobile device.

15. The method of claim 12 further comprising defining a user interface for receiving the input by detecting that a location on the display area has been selected during the video conference.

16. A method of defining a program for allowing a first mobile device to remotely control a second mobile device during a video conference, the method comprising:

defining a display area for displaying on the first mobile device a video comprising images captured by a camera of the second mobile device based on a set of settings of the camera comprising a focus setting that specifies a focal point at which the camera of the second mobile device captures the images, the set of settings specifying a first mode for the camera of the second mobile device to capture the images at a first focal point; and

defining a camera adjustment operation for receiving an input on the first mobile device during the video conference that directs the second mobile device to automatically modify, without user intervention on the second mobile device, the set of settings to specify a second mode for the camera of the second mobile device to capture the images at a second focal point different from the first focal point.

17. The method of claim 16, wherein the input comprises a selection of a portion of the video displayed on the first mobile device, the input for directing the second mobile device to automatically modify, without user intervention on the second mobile device, the set of settings to specify the second mode for the camera of the second mobile device to capture the images at a second focal point based on the selected portion of the video.

18. The method of claim 17, wherein the first mobile device comprises a touch sensitive display screen, wherein the camera adjustment operation is further for receiving the input through the touch sensitive display screen of the first mobile device.

19. The method of claim 16, wherein the video is a first video, wherein the display area is further for displaying on the first mobile device a second video comprising images captured by a camera of the first mobile device.