Video Processing and Telepresence System and Method

• US 20100007773A1
• Filed: 05/12/2009
• Published: 01/14/2010
• Est. Priority Date: 07/14/2008
• Status: Abandoned Application

• Current Assignee: Alex Howes, Ian O'Connell
• Original Assignee: Alex Howes, Ian O'Connell
• Inventors: O'Connell, Ian, Howes, Alex
• Application Number:
  US12/464,224
• Publication Number:
  US 20100007773A1
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G03B 15/02 Illuminating scene
• CPC Class Codes: G03B 21/00 Projectors or projection-type viewers; Accessories therefor devices for changing pictures G03B23/00
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/112 according to a given display mode, e.g. for interlaced or progressive display mode
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A codec having a video input for receiving a continuous video stream, an encoder for encoding the continuous video stream to result in an encoded video stream, a video output for transmitting the encoded video stream, and switching means for switching the encoder during encoding between a first mode, in which the continuous video stream is encoded in accordance with a first encoding format, to a second mode, in which the continuous video stream is encoded in accordance with a second encoding format.

Claims

1. A codec, comprising:

a video input for receiving a continuous video stream;

an encoder for encoding the continuous video stream to result in an encoded video stream;

a video output for transmitting the encoded video stream; and

switching means for switching the encoder during encoding between a first mode, in which the continuous video stream is encoded in accordance with a first encoding format, to a second mode, in which the continuous video stream is encoded in accordance with a second encoding format.

3. The codec of claim 1, wherein the switching means is responsive to an external control signal for switching the encoder between the first mode and the second mode.

4. The codec of claim 1, wherein the codec is capable of changing resolution and/or size of a video image of the continuous video stream.

2. A codec, comprising:

a video input for receiving an encoded video stream;

a decoder for decoding the encoded video stream to result in a decoded video stream;

a video output for transmitting the decoded video stream; and

switching means for switching the decoder during decoding between a first mode, in which the encoded video stream is decoded in accordance with a first encoding format, to a second mode, in which the encoded video stream is decoded in accordance with a second encoding format.

5. A telepresence system, comprising:

a camera for filming a subject to be displayed as an isolated subject image and/or Pepper'"'"'s Ghost;

a first codec for receiving a video stream generated by the camera and outputting an encoded video stream, the first codec including a first video input for receiving the video stream, an encoder for encoding the video stream to result in the encoded video stream, a first video output for transmitting the encoded video stream, and switching means for switching the encoder during encoding between a first encoding mode, in which the video stream is encoded in accordance with a first encoding format, to a second encoding mode, in which the video stream is encoded in accordance with a second encoding format;

means for transmitting the encoded video stream to a second codec at a remote location, the second codec including a second video input for receiving the encoded video stream, a decoder for decoding the encoded video stream to result in a decoded video stream, a second video output for transmitting the decoded video stream, and switching means for switching the decoder during decoding between a first decoding mode, in which the encoded video stream is decoded in accordance with the first encoding format, to a second decoding mode, in which the encoded video stream is decoded in accordance with the second encoding format, the second codec arranged to decode the encoded video stream and output the decoded video stream to apparatus for producing the isolated subject image and/or Pepper'"'"'s Ghost based on the decoded video stream; and

a user operated switch arranged to generate a control signal to cause the first codec to switch between the first encoding mode and the second encoding mode.

6. The telepresence system of claim 5, wherein the user operated switch is further arranged to generate a control signal to cause the second codec to switch between the first decoding mode and the second decoding mode.

7. The telepresence system of claim 6, wherein the second codec is arranged to automatically determine an encoding format of the encoded video stream and switch to decode the encoded video stream using a decoding mode corresponding to the encoding format.

8. A method of generating a telepresence of a subject, comprising:

filming the subject to generate a continuous video stream;

transmitting the continuous video stream to a remote location to generate a transmitted video stream; and

producing an isolated subject image and/or Pepper'"'"'s Ghost at the remote location based on the transmitted video stream;

wherein transmitting the continuous video stream comprises selecting different ones of a plurality of encoding formats during transmission of the continuous video stream based on changes in action being filmed and changing encoding format to selected encoding formats during transmission.

9. The method of claim 8, wherein the changes in action are changes in amount of movement of the subject, changes in lighting of the subject, changes in level of interaction of the subject with a person at the remote location, and/or inclusion of text or graphics in an image to be displayed.

10. A video processer, comprising:

a video input for receiving a video stream;

a video output for transmitting processed video stream;

wherein the video processor is arranged to identify an outline of a subject in each frame of the video stream by scanning pixels of each frame to identify pixels or sets of pixels that have a contrast above a predetermined level and defining the outline of the subject as a continuous line between the pixels or sets of pixels, and make pixels that fall outside the outline of the subject a preselected colour.

11. The video processor of claim 10, wherein the video processor is arranged to process the video stream in substantially real time such that the video stream can be transmitted or displayed in a continuous manner.

12. The video processor of claim 10, wherein identifying the outline of the subject comprises determining a preset number of consecutive pixels that have a first attribute that contrasts a second attribute of an adjacent preset number of consecutive pixels.

13. The video processor of claim 12, comprising means for adjusting the preset number.

14. The video processor of claim 10, wherein the video processor is arranged to modify a frame to provide a line of pixels with high relative luminescence along the outline of the subject.

15. The video processor of claim 14, wherein each pixel of high relative luminescence replaces a corresponding pixel having an original colour and has a colour that is the same as the original colour.

16. A data carrier having stored thereon instructions, which, when executed by a processor, cause the processor to receive a video stream;

process the video stream to generate a processed video stream by identifying an outline of a subject in each frame of the video stream by scanning pixels of each frame to identify pixels or sets of pixels that have a contrast above a predetermined level, defining the outline of the subject as a continuous line between the pixels or sets of pixels, and making pixels that fall outside the outline of the subject a preselected colour; and

to transmit the processed video stream.

17. A codec, comprising:

a video input for receiving a video stream of a subject;

an encoder for encoding the video stream to result in an encoded video stream; and

a video output for transmitting the encoded video stream;

wherein the encoder is arranged to process each frame of the video stream by identifying an outline of the subject and encoding pixels that fall within the outline of the subject while disregarding pixels that fall outside the outline of the subject to form the encoded video stream.

18. The codec of claim 17, wherein the pixels that fall outside the outline of the subject are identified from high luminescence pixels that define the outline of the subject and pixels outside of the high luminescence pixels are disregarded.

19. The codec of claim 17, wherein the encoder comprises a multiplexer for multiplexing the video stream.

20. The codec of claim 19, wherein pixels that fall within the outline of the subject are split into a number of segments and each segment transmitted on a separate carrier as a frequency division multiplexed (FDM) signal.

21. A codec, comprising:

a video input for receiving a video stream and an associated audio stream;

an encoder for encoding the video and associated audio streams; and

a video output for transmitting the encoded video and associated audio streams to a second codec;

wherein the codec is arranged to, during transmission of the video and associated audio streams, periodically transmit to the second codec a test signal, receive an echo response to the test signal from the second codec, determine from a time between sending the test signal and receiving the echo response a signal latency for transmission to the second codec and introduce a suitable delay to the video and associated audio streams for the signal latency.

22. A codec, comprising:

a video input for receiving from a second codec an encoded video stream and associated audio stream;

a decoder for decoding the encoded video stream and associated audio stream to generate a decoded video stream and associated audio stream; and

a video output for transmitting the decoded video and associated audio streams;

wherein the codec is arranged to, during transmission of the decoded video and associated audio streams, transmit an echo response to the second codec in response to receiving a test signal.

23. A system for transmitting a plurality of video streams to be displayed as an isolated subject and/or Pepper'"'"'s Ghost, comprising:

a codec for receiving a plurality of video streams, encoding the plurality of video streams to generate a plurality of encoded video streams, and transmitting the plurality of encoded video streams to a remote location;

wherein the plurality of video streams are generation locked based on one of the plurality of video streams.

24. A video processor, comprising:

a video input for receiving a video stream; and

a video output for transmitting a processed video stream;

wherein the video processor is arranged to identify an outline of a subject in each frame of the video stream by scanning each line of pixels of each frame to identify pixels or sets of pixels that have a contrast above a predetermined level due to a dark background compared to a bright subject and modifying the pixels or sets of pixels to have a higher luminescence than an original luminescence of the pixel or set of pixels.

25. A telepresence system, comprising:

a filming system at a first location for filming a subject to be projected as an isolated subject or Pepper'"'"'s Ghost image and generating a corresponding video stream, the filming system including a codec for encoding the corresponding video stream using one of a plurality of encoding formats to generate an encoded video stream and varying encoding of the corresponding video stream from any one of the plurality of encoding formats to another in response to changes in action being filmed;

a projecting system at a second location for producing the isolated subject or Pepper'"'"'s Ghost image filmed with the filming system, the projecting system including a second codec for receiving the encoded video stream from the filming system and for decoding the encoded video stream to generate a decoded video stream, the second codec operable to switch to a decoding mode capable of decoding the encoded video stream in response to a control signal received from a user operated switch or in response to the second codec automatically determining which one of the plurality of encoding formats was used to encode the encoded video stream; and

a bi-directional communication link connected to the filming system and the projecting system for communicating data between the first location and the second location.

26. The telepresence system of claim 25, wherein:

the filming system further includes a camera for filming the subject and generating the corresponding video stream, and a microphone for recording sound at the first location and generating a corresponding audio stream;

the first codec is further operable to encode the corresponding audio stream and transmit both the encoded audio and video streams over the bi-directional communication link to the projecting system;

the second codec is further operable to decode the encoded audio stream to generate a decoded audio stream; and

the projecting system further includes a speaker for generating sound based on the decoded audio stream.

27. The telepresence system of claim 26, wherein the projecting system includes:

a projector connected to the second codec for receiving the decoded video stream and projecting an image based on the decoded video stream toward a semi-transparent screen; and

a stage positioned relative to the semi-transparent screen so that an audience member perceives a virtual image reflected by the semi-transparent screen on the stage.

28. The telepresence system of claim 27, wherein:

the projecting system includes a second camera for filming audience members or action on the stage and generating a second corresponding video stream;

a second microphone for recording sound at the second location and generating a second corresponding audio stream;

the second corresponding audio and video streams are fed into the second codec, encoded by the second codec to generate a second encoded audio stream and a second encoded video stream, and transmitted by the second codec to the first location using the bi-directional communication link;

the codec is operable to decode the second encoded audio and video streams to generate a second decoded audio stream and a second decoded video stream;

the filming system further includes a second semi-transparent screen positioned between the camera and the subject;

the filming system further includes a heads up display for projecting a second image based on the second decoded video stream toward the second semi-transparent screen so that the subject perceives a second virtual image reflected by the second semi-transparent screen between the camera and the second semi-transparent screen; and

the filming system includes a second speaker for generating sound based on the second decoded audio stream.

29. The telepresence system of claim 28, wherein:

the filming system further includes black material positioned relative to the camera, subject, and second semi-transparent screen to prevent glare from being produced in the camera by the second semi-transparent screen; and

the subject is a performer or a participant in a meeting.

30. The telepresence system of claim 29, wherein the cameras include wide angle zoom lenses with adjustable shutter speeds, have frame rates adjustable between 25 and 120 frames per second interlaced, and are capable of shooting at up to 60 frames per second progressive.

31. The telepresence system of claim 30, wherein the codecs are integrated with the cameras and the cameras are operable to output progressive, interlaced, or other preformatted video streams.

32. The telepresence system of claim 31, wherein the projector is a 1080 high definition projector and is capable of processing both progressive and interlaced video streams, and the semi-transparent screens are foil screens.

33. The telepresence system of claim 32, wherein:

the audience member views the virtual image through upper and lower front masks positioned adjacent to the semi-transparent screen and a black drape is provided adjacent to the stage to provide a backdrop for the virtual image; and

the second semi-transparent screen is supported between a leg and a rigging point.

34. The telepresence system of claim 33, wherein the bi-directional communication link includes the internet or a virtual private network and the first location is a filming studio.