Foveation-based error resilience algorithm

• US 7,679,622 B2
• Filed: 03/30/2001
• Issued: 03/16/2010
• Est. Priority Date: 07/28/2000
• Status: Expired due to Fees

• Current Assignee: Alcatel-Lucent USA, Inc. (Nokia Corporation)
• Original Assignee: Alcatel-Lucent USA, Inc. (Nokia Corporation)
• Inventors: Podilchuk, Christine Irene, Lee, Sanghoon
• Primary Examiner(s): Brier; Jeffery A
• Application Number:
  US09/823,793
• Publication Number:
  US 20020180756A1
• Time in Patent Office: 3,273 Days
• Field of Search: 375/240.08, 375/240.27, 375/240.8, 345/619
• US Class Current: 345/619
• CPC Class Codes: H04N 19/67 involving unequal error protection [UEP], i.e. providing protection according to the importance of the data
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder

Abstract

A method for improving real-time video communications using a Foveation-based unequal error protection scheme (UEP) and error resilience. In a preferred embodiment of the present invention, a real time processing module partitions a video image into at least two data bit stream signals based on a directional pointing device or a pattern filter definition of “areas of importance.” These different data bit stream signals are then processed via different algorithms based on the importance of correcting errors in the data signals. The data is then transmitted. After reception at a receiving device, the data signals are regenerated and corrected to their original form.

Claims

1. A method for partitioning a video image between a foveated area and a background area comprising the steps of:

defining a foveation point in the video image based on a focal point of an eye;

defining a foveared area in proximity to said foveation point;

extracting the first plurality of data signals from said video image representing said foveated area;

extracting a second plurality of data signals from said video image representing a background area;

encoding the extracted first plurality of data signals with a first error correction protocol to create a first encoded signal; and

encoding the extracted second plurality of data signals with a second error correction protocol different from the first error correction protocol to create a second encoded signal, wherein the first error correction protocol comprises a first FEC algorithm and a second error correction protocol comprises a second FEC algorithm, the first FEC algorithm being more powerful than the second FEC algorithm.

2. The method according to claim 1, wherein the step of defining said foveation point comprises the step of:

pointing a video device at a location of the image using a means for pointing.

3. The method according to claim 2, wherein the pointing means comprises at least one of:

a computer keyboard;

a computer mouse;

a joystick, and an eye tracking device.

4. The method according to claim 1 further comprising the step of:

calculating a local bandwidth threshold based on said foveation point; and

wherein the step of defining said foveation area comprises the steps of;

calculating a Local bandwidth for each pixel group in said video image; and

incorporating those pixel groups having a respective local bandwidth above said local bandwidth threshold into said foveation area.

5. The method according to claim 1 further comprising the steps of:

packetizing the first encoded signal with inserted synchronization markers occurring after a first predetermined number of bits; and

packetizing the second encoded signal with the inserted synchronization markers occurring after a second predetermined number of bits wherein the first number is smaller than the second number.

6. The method according to claim 1 wherein the first plurality of data signals comprises all pixel signals included in a high-resolution area of said video image.

7. The method according to claim 1 wherein the first plurality of data signals comprises all pixel signals that are included in a high motion area of said video image.

8. The method according to claim 1 wherein the first error correction protocol conforms to video communications industry standards H263++ and/or MPEG-4.

9. The method according to claim 1 wherein the second error correction protocol conforms to video communications industry standards H263++ and/or MPEG-4.

10. The method according to claim 1 further comprising the steps of:

transmitting the first encoded signal; and

transmitting a second encoded signal at a predetermined time after the transmitting of said first encoded signal.

11. A method for the processing of video image data received from a first electronic device, the first electronic device having performed the steps of:

defining a foveation point in a video image based on a focal point of an eye;

defining at least one foveated area around said foveation point;

extracting a first plurality of data signals representing said foveated area;

extracting a second plurality of data signals representing a background area;

encoding the extracted first plurality of data signals with a first error correction protocol to create a first encoded signal; and

encoding the extracted second plurality of data signals with a second error correction protocol different from the first error correction protocol to create a second encoded signal, the method comprising the steps of;

decoding the first transmitted encoded signal;

correcting errors within the first transmitted encoded signal with the use of a high-priority processing step to create a received foveated area;

decoding the second transmitted encoded signal; and

correcting errors within the second transmitted encoded signal with use of a low priority processing step to create a received a background area.

12. The method according to claim 11 further comprising the step of:

combining the received foveated area and the received background area to create the video image data.

13. A method for partitioning a video image between a foveated area and a background area comprising the steps of:

defining a foveation point in the video image based on a focal point of an eye;

defining a foveated area in proximity to said foveation point;

extracting a first plurality of data signals from said video image representing said foveated area;

extracting a second plurality of data signals from said video image representing a background area;

encoding the extracted first plurality of data signals with a first error correction protocol to create a first encoded signal; and

encoding the extracted second plurality of data signals with a second error correction protocol different from the first error correction protocol to create a second encoded signal wherein the first error correction protocol comprises a first ARQ communications protocol having a first allowable error threshold associated therewith and the second error correction protocol comprises a second ARQ communications protocol having a second allowable error threshold associated therewith, the first allowable error threshold being lower than the second allowable error threshold.

14. The method according to claim 13 wherein the step of defining said foveation point comprises the step of pointing a video device at a location of the image using a means for pointing.

15. The method according to claim 14 wherein the pointing means comprises at least one of:

a computer keyboard;

a computer mouse;

a joystick, and an eye tracking device.

16. The method according to claim 13 further comprising the step of:

calculating a local bandwidth threshold based on said foveation point; and

wherein the step of defining said foveation area comprises the steps of;

calculating a local bandwidth far each pixel group in said video image; and

incorporating those pixel groups having a respective local bandwidth above said local bandwidth threshold into said foveation area.

17. The method according to claim 13 further comprising the steps of:

packetizing the first encoded signal with inserted synchronization markers occurring after a first predetermined number of bits; and

packetizing the second encoded signal with the inserted synchronization markers occurring after a second predetermined number of bits wherein the first number is smaller than the second number.

18. A method, comprising:

defining a foveation point in a video image based on a focal point of an eye;

defining a foveated area in proximity to the foveation point;

extracting a first plurality of data signals indicative of the foveated area from the video image;

extracting a second plurality of data signals indicative of a background area from the video image;

encoding the extracted first plurality of data signals with a first error correction protocol to create a first encoded signal; and

encoding the extracted second plurality of data signals with a second error correction protocol different from the first error correction protocol to create a second encoded signal.

19. A method comprising:

decoding a first signal indicative of at least one foveation area around a foveation point in a video image, the foveation point defined based on the focal point of an eye, wherein the first signal is encoded according to a first error correction protocol;

correcting errors within the first signal using a high-priority processing step to create a received foveared area;

decoding a second signal indicative of a background area in the video image, wherein the second signal is encoded according to a second error correction protocol different from the first error correction protocol; and

correcting errors within the second signal using a low priority processing step to create a received background area.