Apparatus and method for remote control using a visual information stream

• US 5,652,849 A
• Filed: 03/16/1995
• Issued: 07/29/1997
• Est. Priority Date: 03/16/1995
• Status: Expired due to Term

• Current Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Original Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Inventors: Cohen, Charles J., Conway, Lynn A.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): KATBAB, ABDOLLAH
• Application Number:
  US08/404,886
• Time in Patent Office: 866 Days
• Field of Search: 395/155-161, 395/327, 348/115, 348/114, 348/113, 364/424.029
• US Class Current: 715/719
• CPC Class Codes: G08B 25/14 Central alarm receiver or annunciator arrangements
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems and methods for controlling devices remotely by embedding command information within video signals. The command information includes gestures or icons which are imbedded into the video signal and enable the user to interpret the command information by viewing the icon or gesture on a monitor or display device. By viewing the monitor or display device, the user can determine what command information is conveyed within the video signal. Further, imbedding the visually interpretable icon or gesture within the video signal facilitates transmitting the video signal having the command information imbedded therein using analog to digital conversions and reconversions and transmission through any of a various number of switching networks. The remote devices may by any of a number of electronically controlled electrical or mechanical devices. A device controller at the remote end receives, the decoded command information and generates the appropriate commands to operate the controlled device.

Claims

1. An apparatus for enabling an operator to receive an incoming video signal and to imbed command information in an outgoing video signal in order to control a device, comprising:

means for generating the command information in the form of a visually iconic gesture overlaid onto the incoming video signal to define the outgoing video signal, the visually iconic gesture being interpretable by the operator and conveying the command information in accordance with its location within the outgoing video signal;

processing means for receiving the outgoing video signal having the visually iconic gesture and decoding the iconic gesture into the command information conveyed by the iconic gesture, the processing means outputting the command information; and

device control means for receiving the command information output by the processing means and transforming the command information into control commands for operating the device in accordance with the command information conveyed in the visually iconic gesture.

2. The apparatus as defined in claim 1 further comprising:

a local site at which is located the operator and the generating means; and

a remote site at which is located the processing means and the device control means.

3. The apparatus as defined in claim 2 further comprising a communications interface for establishing a communications link between the local site and the remote site, the communications link enabling transmission of the incoming video signal from the remote site to the local site and enabling transmission of the overlaid video signal from the local site to the remote site.

4. The apparatus as defined in claim 3 further comprising a monitor means interconnected to the means for generating command information, the monitor means displaying the received video signal and the visually iconic gesture overlaid thereon in order to enable an operator to view the iconic gesture and its relative position within the received video signal.

5. The apparatus as defined in claim 4 wherein the means for generating command information further comprises:

means for positioning the iconic gesture, the iconic gesture being overlaid into the receiving video signal and viewable by the operator on the monitoring means; and

means for selecting at least one of a plurality of candidate iconic gestures, the selected at least one iconic gesture being positionable in accordance with operation of the positioning means.

6. The apparatus as defined in claim 1 wherein the means for generating command information further comprises:

means for positioning the iconic gesture, the iconic gesture being overlaid into the receiving video signal and viewable by the operator on the monitoring means; and

means for selecting at least one of a plurality of candidate iconic gestures, the selected at least one iconic gesture being positionable in accordance with operation of the positioning means.

7. The apparatus as defined in claim 6 wherein the means for generating the iconic gesture further comprises a video overlay mixer for overlaying onto the incoming video signal the command information in the form of the iconic gesture and the position within the incoming video signal, the means for generating the iconic gesture outputting an overlaid video signal to a monitor.

8. The apparatus as defined in claim 1 wherein the means for generating the iconic gesture further comprises a video overlay mixer for overlaying onto the incoming video signal the command information in the form of the iconic gesture and the position within the incoming video signal, the means for generating the iconic gesture outputting an overlaid video signal to a monitor and to the processing means.

9. The apparatus as defined in claim 1 wherein the processing means for decoding the outgoing video signal further comprises:

means for extracting from the overlaid video signal information including a command icon and a position of the command icon; and

means for interpreting the iconic information extracted from the overlaid video signal.

10. The apparatus as defined in claim 9 wherein said means for extracting further comprises:

memory for storing a digitized video signal;

pattern memory for storing in digitized form at least a portion of a candidate icon recognizable by the extracting means; and

icon recognizer means for comparing the at least a portion of the candidate icon stored in pattern memory with the digitized video signal stored in memory, the icon recognizer outputting a signal in accordance with a recognized icon.

11. The apparatus as defined in claim 9 wherein the incoming signal is an analog signal and the apparatus further comprises a digitizing means for digitizing the analog incoming video signal to yield a digital representation of the incoming video signal.

12. The apparatus as defined in claim 9 wherein the incoming video signal is a digital signal and the apparatus further includes signal processing means to condition the incoming video signal for storage in memory.

13. The apparatus as defined in claim 1 wherein the device to be controlled comprises:

a controllable camera base arranged to enable motion in at least one degree of freedom, said camera base comprising;

a camera mounted to said camera base;

a first means for enabling the camera base to operate in the at least one degree of freedom; and

communication means for communicating with the device control means, the communication means sending and receiving control signals to operate the camera base in the at least one degree of freedom.

14. The apparatus as defined in claim 13 wherein the enabling means comprises a motor.

15. The apparatus as defined in claim 13 wherein the means for generating command information is operable to insert visual information defining a centering icon, and the control means is operable to position the controllable base to center the camera view area about a point selected in accordance with the centering icon.

16. The apparatus as defined in claim 13 wherein the means for generating command information is operable to insert a visual iconic gesture defining a pan/tilt icon, and the control means is operable to position the controllable base in a direction and a rate in accordance with the direction and magnitude, respectively, of a displacement of the pan/tilt icon from a center of the camera view area.

17. The apparatus as defined in claim 13 wherein the means for generating command information is operable to insert a visual iconic gesture information defining a move icon, and the control means is operable to position the controllable base in accordance with the position of a first and second move gesture where the first move icon defines a starting point for the move and a second move gesture defines a final point for the move where camera base is displaced such that the final point of the view area is displaced to the initial point of the view area.

18. The apparatus as defined in claim 1 wherein the controlled device is a robot arm which receives control signals from the device control means and is manipulable in accordance with the control signal.

19. The apparatus as defined in claim 1 wherein the controlled device is an electronically controlled device which receives control signals from the device control means and is operable in accordance with the control signal.

20. A remote control video apparatus in which an operator at a local site which receives an incoming video signal from a remote site can generate commands to operate a device located at the remote cite, comprising:

a visual signaller for receiving the incoming video signal from the remote site, the visual signaller operable to overlay visual command information onto the incoming video signal to define an outgoing video signal, the visual signaler being operable to output the outgoing video signal for transmission to the remote site, the visual command information being visually interpretable by a user at the local or remote site;

a visual interpreter for decoding the overlaid video signal in order to determine the visual command information overlaid onto the incoming video signal and for outputting the decoded visual command information; and

a device controller for receiving visual command information decoded by the visual interpreter, the device controller generating a command for operating the device in accordance with the visual command information overlaid onto the overlaid video signal.

21. The apparatus as defined in claim 20 wherein the apparatus further comprises a monitor for displaying the overlaid video signal output by the visual signaller.

22. The apparatus as defined in claim 21 wherein the visual signaller further comprises:

a graphics locator means for manipulating a pointer overlaid onto the video signal to enable the operator to view the position of the visual command information; and

an icon selector to enable an operator to select at least one of a plurality of icons corresponding to at least one of a plurality of candidate commands to operate the device at the remote site.

23. The apparatus as defined in claim 20 wherein the visual signaller further comprises:

a graphics locator means for manipulating a pointer overlaid onto the video signal to enable the operator to view the position of the visual command information; and

an icon selector to enable an operator to select at least one of a plurality of icons corresponding to at least one of a plurality of candidate commands to operate the device at the remote site.

24. The apparatus as defined in claim 20 wherein the device to be operated at the remote cite is a controllable camera base for supporting a camera, where the controllable base is controllable for positioning the camera.

25. The apparatus as defined in claim 24 wherein the controllable base is operable to pan in a horizontal direction and tilt in a vertical direction.

26. A visual controller for receiving a video signal and generating visually interpretable control commands related to the received video signal and inserted into an outgoing video signal for controlling a remote device, comprising:

a selector for inserting into the outgoing video signal at least one visually interpretable gesture, the gesture being decodable into a partial control command for controlling the remote device; and

a gesture positioner for positioning the at least one selected gesture onto the outgoing video signal where the position of the at least one selected gesture supplements the control command in accordance with the position of the at least one selected gesture.

27. The visual control device defined in claim 26 further comprising a display means for displaying the incoming video signal and the at least one selected gesture overlaid onto the incoming video signal, wherein an operator may view on the display means the command information to be inserted into the outgoing video signal.

28. The visual control device as defined in claim 26 wherein the outgoing signal includes the incoming video signal having overlaid thereon the gesture information inserted by the visual control device.

29. The visual control device as defined in claim 26 further comprising a video overlay mixer for receiving the incoming video signal and a signal representing the at least one selected gesture positioned by the gesture positioner, and for providing the outgoing video signal including the incoming signal having inserted thereon the at least one selected gesture.

30. The visual signaller as defined in claim 26 wherein the gesture positioner is one of a joy stick, a graphics locator pad, and a track ball.

31. A visual interpreter for detecting the presence of at least one gesture in an incoming video signal and generating device control commands in accordance with the control commands in the incoming video signal, comprising:

means for extracting the at least one gesture from the incoming video signal, the gesture being visually interpretable when displayed on a display means; and

means for interpreting the extracted gesture and providing device control commands to a device controller, the device controller operating the controlled device in accordance with the device control commands.

32. The visual interpreter as defined in claim 31 wherein the incoming video signal is an analog signal, and the means for extracting further comprises a digitizer for creating a digitized representation of the analog signal and memory for storing the digitized representation of the analog signal.

33. The visual interpreter as defined in claim 31 wherein the incoming video signal is a digital signal, and the means for extracting further comprises a memory for storing the digital incoming video signal.

34. The visual interpreter as defined in claim 33 further comprising signal processing means for conditioning the incoming video signal.

35. The visual interpreter as defined in claim 31 further comprising a memory for storing the incoming video signal.

36. The visual interpreter as defined in claim 31 further comprising a gesture pattern memory having stored therein at least one of a plurality of candidate gestures.

37. The visual interpreter as defined in claim 31 wherein:

the incoming video signal is an analog signal, and the means for extracting further comprises a digitizer for creating a digitized representation of the analog signal and memory for storing the digitized representation of the analog signal; and

a gesture pattern memory having stored therein at least one of a plurality of candidate gestures, where the means for extracting the at least one gesture compares at least a portion of the digitized representation of the analog signal with at least one of the plurality of candidate gestures.

38. The apparatus as defined in claim 37 further wherein:

the incoming video signal is a digital signal, and the means for extracting further comprises a memory for storing the digital incoming video signal; and

a gesture pattern memory having stored therein at least one of a plurality of candidate gestures recognized by the means for extracting, where the means for extracting the at least one gesture compares at least a portion of the digitized representation of the analog signal with at least one of the plurality of candidate gestures.

39. An apparatus for enabling an operator to remotely control a device, comprising:

a received video signal input to the apparatus, the video signal having an object located therein;

means for generating command information in the form of a visually iconic gesture overlaid onto the received video signal to define an output video signal, where the visually iconic gesture has a meaning which varies in accordance with the position of the visually iconic gesture relative to the object, and where the visually iconic gesture is interpretable by the operator;

processing means for receiving outgoing video signal and decoding the outgoing video signal into the command information conveyed by the visually iconic gesture and the position of the visually iconic gesture relative to the object in the incoming video signal, the processing means outputting decoded command information to control the object on which the visually iconic gesture is located; and

device control means for receiving the decoded command information output by the processing means and transforming the command information into control commands for operating the object in accordance with the command information conveyed in the visually iconic gesture.

40. The apparatus as defined in claim 39 further comprising:

an imaging device to provide the received video signal, the imaging device scanning and capturing a portion of a predetermined area defined in accordance with a specific orientation of the imaging device;

memory having stored therein candidate objects and a corresponding location of each candidate object, the object being determinable in accordance with its predetermined location;

mapping means for correlating the orientation of the imaging device and the objects stored in memory in accordance with the location of the object and the orientation of the imaging means.

41. A method for controlling a remotely located device, comprising the steps of:

generating command information in the form of a visually iconic gesture overlaid onto an outgoing video signal, the visually iconic gesture being interpretable by the operator and conveying command information in accordance with its location within the outgoing video signal;

processing the outgoing video signal having the visually iconic gesture to decode the iconic gesture into the command information conveyed by the iconic gesture, the processing means outputting the command information; and

transforming the command information into control commands for operating the device in accordance with the command information conveyed in the visually iconic gesture.