Remote display of an image by transmitting compressed video frames representing back-ground and overlay portions thereof

• US 5,493,638 A
• Filed: 10/13/1994
• Issued: 02/20/1996
• Est. Priority Date: 12/22/1993
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Digital Equipment Corporation (HP Inc.)
• Inventors: Tongel, David M., Hooper, Donald F.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US08/322,662
• Time in Patent Office: 495 Days
• Field of Search: 395/135, 395/133, 345/113, 345/114, 348/584, 348/588, 348/598
• US Class Current: 345/636
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/507 using conditional replenishment
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4351 involving reassembling additional data, e.g. rebuilding an executable program from recovered modules
• CPC Class Codes: H04N 21/8166 involving executable data, e.g. software arrangements for executing specific programs G06F9/44; broadcasting computer programmes in broadcast systems H04H20/91; involving the movement of software or configuration parameters H04L67/34
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

An interactive video system employs Motion Picture Expert Group (MPEG) video compression to transfer images from a remote server to a television. The images correspond to dialog frames in a graphical user interface. During an authoring process, the dialog frames are created by first creating a background image and then adding foreground elements, such as buttons. A set of MPEG video frames is created by encoding the resulting images according to the MPEG algorithm. The MPEG video frames are delivered to the television in sequence, where an MPEG decoder uses them to reconstruct the dialog images that are subsequently displayed. The system also contains an object-oriented database that maintains the necessary MPEG file ordering and also carries out menu navigation commands received from the user. The object classes include NODE, BRANCH, and DISPLAY. NODE objects correspond to dialog frames, and BRANCH objects correspond to user-selectable features such as buttons. DISPLAY objects correspond to either complete or partial images, as described in corresponding MPEG files. Data and function elements within the objects allow the system to track the user'"'"'s navigation commands, and to highlight and unhighlight selectable features as navigation proceeds in order to provide visual feedback to the user.

Claims

1. A method of operating a computer, comprising the steps of:

instantiating application programs each being associated with a corresponding one of control objects appearing on a dialog video image to be subsequently displayed on a display device coupled to a communications medium connected to said computer;

transmitting over said communications medium a sequence of at least one compressed video frame collectively representing an unhighlighted version of said dialog video image, the transmitted video frames being encoded according to a compression algorithm that is capable of describing a single image by an independent video frame and also capable of describing the differences between a pair of images by a change video frame;

transmitting over said communications medium a change video frame representing changes to be made to said unhighlighted dialog video image in order to highlight an initial one of the control objects appearing thereon;

awaiting the receipt of a keystroke identifier from said communications medium;

executing the following navigation steps if the received keystroke identifier represents one of a set of navigation commands to be carried out by said computer;

(i) transmitting over said communications medium a change frame representing changes to the displayed dialog image that are necessary to unhighlight the control object that is currently highlighted;

(ii) selecting another of said control objects for highlighting as directed by said one navigation command; and

(iii) transmitting over said communications medium a change frame representing changes to said dialog image that are necessary to highlight the selected control object; and

invoking the application program associated with the currently-highlighted control object if said keystroke identifier represents a select command to be carried out by said computer.

2. A method according to claim 1, wherein said compression algorithm is the Motion Picture Expert Group (MPEG) algorithm, and wherein said independent video frame is an MPEG I-free and said change video frame is an MPEG P-frame.

3. A method according to claim 1, further comprising the steps of:

creating a text block to appear on said dialog video image, said text block including a text string and having descriptive text attributes associated therewith;

storing said text string and said descriptive text attributes on said computer; and

transmitting said text string and said descriptive text attributes over said communications medium to enable said video display device to reconstruct said text block for subsequent display thereon.

4. A method according to claim 1, wherein said dialog video image is a parent dialog video image having associated therewith a child dialog video image to appear when a corresponding control object on said parent dialog video image is activated, and further comprising the step of:

repeating the steps of claim 1 for said child video image upon activation of the corresponding control object on said parent dialog video image.

5. A method according to claim 1, further comprising the steps of:

composing said unhighlighted dialog video image on an authoring station coupled to said computer;

composing a set of highlighted dialog video images on said authoring station, each highlighted image resulting from corresponding changes to said unhighlighted dialog video image that highlight a corresponding one of said control objects;

encoding said composed images according to said compression algorithm to yield said sequence of at least one compressed video frame representing said unhighlighted dialog video image and all of said change video frames necessary to respectively highlight and unhighlight said control objects; and

storing the encoded video frames on said computer in a manner sufficient to preserve the sequentiality of said sequence of at least one compressed video frame and to preserve the association between each of said control objects and the corresponding pair of highlighting and unhighlighting change frames.

6. A method according to claim 1, wherein said set of navigation commands includes "up", "down", "left" and "right" navigation commands, and wherein the control object selected in said selecting step is the control object having the corresponding spatial relationship with the currently-highlighted control object.

7. A method according to claim 1, wherein:

(1) each of said control objects has associated therewith a corresponding one of a set of distinct predetermined response codes;

(2) said set of navigation commands comprises a command to select the control object whose response code matches a response code represented by the received keystroke identifier upon receipt thereof; and

(3) said selecting step comprises the steps of;

i) searching through said set of response codes to determine the control object whose corresponding response code matches the response code represented by the received keystroke identifier; and

ii) selecting the control object having the matching response code.

8. A method according to claim 1, further comprising the steps of:

1) instantiating a node object associated with said unhighlighted dialog video image and having the following attributes;

a) an initial frame pointer to point to a display object to be subsequently instantiated that is associated with said node object and also with said unhighlighted dialog video image;

b) a focus pointer to point to a selected one of branch objects to be subsequently instantiated that are associated with said node object and also respectively associated with corresponding ones of said control objects appearing on said dialog video image;

c) an initial focus data element to serve as the initial value of said focus pointer so that it initially points to an initial one of said branch objects that is associated with said initial control object; and

d) an initialize function to carry out said first transmitting step and the following steps upon execution;

i) obtaining from the display object pointed to by said initial frame pointer the names of video files stored on said computer that collectively contain said sequence of at least one compressed video frames in preparation for said first transmitting step;

ii) initializing said focus pointer with said initial focus data element; and

iii) carrying out said second transmitting step by executing a hilight function of the branch object pointed to by said initial focus data element;

2) instantiating said branch objects, each having the following attributes;

a) a hilight display object pointer to point to a hilight display object to be subsequently instantiated that is associated with the branch object being instantiated and also with a highlighted version of the control object associated therewith;

b) an unhilight display object pointer to point to an unhilight display object to be subsequently instantiated that is associated with the branch object being instantiated and also with an unhighlighted version of the control object associated therewith;

c) a hilight function to carry out the step of transmitting a change video frame to highlight the corresponding control object and also the following step upon execution;

i) obtaining from said hilight display object the name of a video file stored on said computer that contains the highlighting change video frame to be transmitted;

d) an unhilight function to carry out the step of transmitting a change video frame to unhighlight the corresponding control object and also the following step upon execution;

i) obtaining from said unhilight display object the name of a video file stored on said computer that contains the unhighlighting change video frame to be transmitted; and

e) a select function to invoke upon execution the one of said application programs that is associated with both the branch object being instantiated and the corresponding control object appearing on said dialog video image;

3) incorporating navigational attributes among said node object and said branch objects upon instantiation thereof, said navigational attributes including at least one navigation function to carry out said navigation steps by executing the following steps upon execution;

executing said unhilight function of the branch object pointed to by said focus pointer;

changing said focus pointer to point to a different one of said branch objects as directed by the navigation command represented by the received keystroke identifier; and

executing said hilight function of the branch object pointed to by said focus pointer after it has been changed;

4) instantiating all of said display objects associated with said node object and branch objects, each display object also being associated with a corresponding one of said independent and change video frames and having the following attribute;

a) a name of an associated video file containing the corresponding one of said independent and change video frames;

5) carrying out said first and second transmitting steps by executing said initialize function of said node object;

6) carrying out said navigation steps by executing said navigation function upon receipt of a keystroke identifier representing one of said set of navigation commands; and

7) carrying out said invoking step by executing the select function of the branch object pointed to by said focus pointer upon receipt of a keystroke identifier representing said select command.

9. A method according to claim 8, wherein said navigational attributes comprise:

(1) up, down, left, and right branch pointers in each of said branch objects to point respectively to corresponding up, down, left, and right ones of said branch objects that are correspondingly spatially related to the branch object containing the corresponding branch pointer;

(2) up, down, left, and right arrow functions in each of said branch objects to respectively carry out the following steps upon execution;

i) executing the unhilight function of the branch object containing the arrow functions;

ii) substituting the corresponding one of said up, down, left, and right branch pointers for said focus pointer; and

iii) executing the hilight function of the branch object pointed to by said focus pointer after said substituting step has been executed;

and wherein said step of executing said navigation function comprises the step of executing the corresponding one of said up, down, left and right functions of the branch object pointed to by said focus pointer if the received keystroke identifier represents a corresponding one of "up", "down", "left" and "right" navigation commands.

10. A method according to claim 8, wherein said navigational attributes comprise:

(1) a set of distinct predetermined response codes each contained in a corresponding one of said branch objects; and

(2) a find_-- branch function in said node object to carry out the following steps upon execution;

i) searching through the response codes contained in said branch objects to determine the branch object whose response code matches a response code received from said communications medium;

ii) executing the unhilight function of the branch object having the matching response code;

iii) changing said focus pointer to points to the branch object having the matching response code; and

iv) executing the hilight function of the branch object pointed to by said focus pointer after said changing step has been executed;

and wherein said step of executing said navigation function comprises the step of executing said find_-- branch function if the received keystroke identifier represents a response code.

11. A method of operating a computer, comprising the steps of:

1) instantiating application programs each being associated with a corresponding one of control objects appearing on a dialog video image to be subsequently displayed on a display device coupled to a communications medium connected to said computer;

2) instantiating a node object associated with said dialog video image and having the following attributes;

a) an initial frame pointer to point to a display object to be subsequently instantiated that is associated with said node object and also with said dialog video image;

b) a focus pointer to point to a selected one of branch objects to be subsequently instantiated that are associated with said node object and respectively associated with corresponding ones of said control objects;

c) an initial focus data element to serve as the initial value of said focus pointer; and

d) an initialize function to carry out the following steps upon execution;

i) transmitting over a communications medium connected to said computer a set of at least one video files stored on said computer whose file name is contained within the display object pointed to by said initial frame pointer, said set of video files representing an unhighlighted version of said dialog video image;

ii) initializing said focus pointer with said initial focus data element; and

iii) executing a hilight function of an initial branch object pointed to by said initial focus data element in order to highlight a corresponding initial control object;

3) instantiating said branch objects, each having the following attributes;

a) a hilight display object pointer to point to a hilight display object to be subsequently instantiated that is associated with the branch object being instantiated and with the corresponding control object;

b) an unhilight display object pointer to point to a unhilight display object to be subsequently instantiated that is associated with the branch object being instantiated and with the corresponding control object;

c) a hilight function to carry out the following steps upon execution;

i) transmitting over said communications medium a video file stored on said computer whose file name is contained within said hilight display object, the transmitted video file representing a corresponding version of said dialog video image in which the control object associated with the branch object being instantiated is highlighted;

d) an unhilight function to carry out the following steps upon execution;

i) transmitting over said communications medium a video file stored on said computer whose file name is contained within said unhilight display object, the transmitted video file representing a corresponding version of said dialog video image in which the control object associated with the branch object being instantiated is unhighlighted; and

e) a select function to invoke upon execution the one of said application programs that is associated with both the branch object being instantiated and the corresponding control object appearing on said dialog video image;

4) incorporating navigational attributes among said node object and said branch objects upon instantiation thereof, said navigational attributes including at least one navigation function to carry out the following steps upon execution;

executing said unhilight function of the branch object pointed to by said focus pointer;

changing said focus pointer to point to a different one of said branch objects as directed by the one of a set of navigation commands that is represented by a keystroke identifier whose receipt from said communications medium has resulted in the execution of the navigation function; and

executing said hilight function of the branch object pointed to by said focus pointer after it has been changed;

5) instantiating all of said display objects associated with said node object and branch objects, each display object having the following attributes;

a) a video file name of an associated video file stored on said computer associated with a corresponding one of said unhighlighted and highlighted versions of said dialog video image;

6) executing said initialize function of said node object;

7) awaiting the receipt of a keystroke identifier from said communications medium;

8) executing said navigation function upon receipt of a keystroke identifier representing one of said set of navigation commands; and

9) executing the select function of the branch object pointed to by said focus pointer upon receipt of a keystroke identifier representing said select command.

12. A method according to claim 11, wherein:

(1) said node object has an initial text pointer to point to a text object to be subsequently instantiated that is associated with said node object;

(2) said initialize function carries out upon execution the step of transmitting over said communications medium all of the data elements contained in the text object pointed to by said initial text pointer;

(3) said hilight function carries out upon execution the step of transmitting over said communications medium all of the data elements contained in another text object to be pointed to upon its subsequent instantiation by a text pointer contained within said hilight display object;

(4) said unhilight function carries out upon execution the step of transmitting over said communications medium a text string consisting of character blanks to be used to erase any text associated with said branch object that was displayed as a result of executing said hilight function; and

(5) each of said display objects has a text pointer to point to a text object to be subsequently instantiated that is associated therewith;

and further comprising the step of;

instantiating all of said text objects associated with said node object and display objects, each text object having the following attributes;

a) a text string associated with the text object being instantiated; and

b) X and Y coordinates to respectively indicate horizontal and vertical components of a screen position where said text string is to be displayed.

13. A method according to claim 12, wherein each of said text objects also contains a color value to indicate a color in which said text string is to be displayed.

14. A method according to claim 12, wherein each of said text objects also contains a font value to indicate a font in which said text string is to be displayed.

15. A method according to claim 12, wherein each of said text objects also contains a size value to indicate a size in which said text string is to be displayed.

16. A method according to claim 11, wherein said video files are encoded according to a video compression algorithm.

17. A method according to claim 11, wherein:

(1) said video files are encoded according to a compression algorithm that is capable of describing a single image by an independent video frame and also capable of describing the differences between a pair of images by a change video frame;

(2) said node object contains an initial overlay sequence pointer to point to ah ordered list of at least one display objects each of which contains the name of a corresponding one of said set of video files representing an unhighlighted version of said dialog video image;

(3) each of said display objects contains a previous display object pointer to point to an immediately-preceding display object in a corresponding sequence thereof; and

(4) each of said display objects contains a next display object pointer to point to an immediately-following display object in said corresponding sequence of display objects;

the previous and next object display object pointers of said ordered list of display objects serving as the means by which it is ordered.

18. A method according to claim 11, wherein said compression algorithm is the Motion Picture Expert Group (MPEG) algorithm, and wherein said independent video frame is an MPEG I-frame and said change video frame is an MPEG P-frame.

19. A method according to claim 11, wherein:

(1) said dialog video image is a parent dialog video image having associated therewith at least one child dialog video images each to appear when a corresponding one of said control objects is activated;

(2) said node object is a parent node object associated with said parent dialog video image;

(3) each of said branch objects has a destination node pointer, the destination node pointer in each of those branch objects that is associated with a corresponding child dialog video image pointing to a corresponding associated child node object to be subsequently instantiated having the same attributes as said parent node object, the destination node pointer in the remainder of said branch objects containing a null value;

(4) said select function contains an application_-- call function that invokes the corresponding application program upon execution; and

(5) said select function upon execution selects the child node object pointed to by said destination node pointer by executing the initialize function thereof if said destination node pointer does not contain said null value, and otherwise executes said application_-- call function;

and further comprising the steps of;

(1) repeating steps (1) through (5) of claim 11 for each of said child dialog video images prior to executing step (6) thereof; and

(2) repeating steps (7) through (9) of claim 11 for each of said child node objects that is selected by the execution of the corresponding select function in said parent node object.

20. A method according to claim 11, wherein said navigational attributes comprise:

(1) up, down, left, and right branch pointers in each of said branch objects to point respectively to corresponding up, down, left, and right ones of said branch objects that are correspondingly spatially related to the branch object containing the corresponding branch pointer;

(2) up, down, left, and right arrow functions in each of said branch objects to respectively carry out the following steps upon execution;

i) executing the unhilight function of the branch object containing the arrow functions;

ii) substituting the corresponding one of said up, down, left, and right branch pointers for said focus pointer; and

iii) executing the hilight function of the branch object pointed to by said focus pointer after said substituting step has been executed;

and wherein said step of executing said navigation function comprises the step of executing the corresponding one of said up, down, left and right functions of the branch object pointed to by said focus pointer if the received keystroke identifier represents a corresponding one of "up", "down", "left" and "right" navigation commands.

21. A method according to claim 11, wherein said navigational attributes comprise:

(1) a set of distinct predetermined response codes each contained in a corresponding one of said branch objects; and

(2) a find_-- branch function in said node object to carry out the following steps upon execution;

i) searching through the response codes contained in said branch objects to determine the branch object whose response code matches a response code received from said communications medium;

ii) executing the unhilight function of the branch object having the matching response code;

iii) changing said focus pointer to points to the branch object having the matching response code; and

iv) executing the hilight function of the branch object pointed to by said focus pointer after said changing step has been executed;

and wherein said step of executing said navigation function comprises the step of executing said find_-- branch function if the received keystroke identifier represents a response code.

22. A method of operating a computer, comprising the steps of:

1) instantiating application programs each being associated with a corresponding one of control objects appearing on a dialog video image to be subsequently displayed on a display device coupled to a communications medium connected to said computer;

2) instantiating a node object associated with said dialog video image and having the following attributes;

a) an initial overlay sequence pointer to point to an ordered list of display objects to be subsequently instantiated that are associated with said node object and also with said dialog video image;

b) an initial text pointer to point to a text object to be subsequently instantiated that is associated with said node object and also with said dialog video image;

c) a focus pointer to point to a selected one of branch objects to be subsequently instantiated that are associated with said node object and respectively associated with corresponding ones of said control objects;

d) an initial focus data element to serve as the initial value of said focus pointer;

e) an initialize function to carry out the following steps upon execution;

i) sequentially transmitting over a communications medium connected to said computer a set of Motion Picture Expert Group (MPEG) files stored on said computer whose respective file names are contained within corresponding ones of the ordered list of display objects pointed to by said initial overlay sequence pointer, said set of MPEG files representing an unhighlighted version of said dialog video image;

ii) executing a hilight function of an initial one of said branch objects pointed to by said initial focus data element;

iii) transmitting over said communications medium all of the data elements contained in the text object pointed to by said initial text pointer; and

iv) initializing said focus pointer with said initial focus data element; and

f) a find_-- branch function to carry out the following steps upon execution;

i) comparing a response code received from said communications medium with each of distinct predetermined response codes, each of said predetermined response codes being contained in a corresponding one of said branch objects;

ii) executing an unhilight function contained in the branch object whose response code matches the received response code;

iii) changing said focus pointer so that it points to the branch object containing the matching response code; and

iv) executing a hilight function of the branch object pointed to by said focus pointer after said changing step has been executed;

3) instantiating said branch objects, each having the following attributes;

a) a hilight display object pointer to point to a hilight display object to be subsequently instantiated that is associated with the branch object being instantiated and with the corresponding control object;

b) an unhilight display object pointer to point to a unhilight display object to be subsequently instantiated that is associated with the branch object being instantiated and with the corresponding control object;

c) a response code to be used by said find_-- branch function;

d) up, down, left, and right branch pointers to point respectively to corresponding up, down, left, and right ones of said branch objects that are correspondingly spatially related to the branch object being instantiated;

e) a destination node pointer to point to a child node object to be subsequently instantiated if its activation is intended to result from activation of the branch object being instantiated, and to otherwise contain a null value;

f) a hilight function to carry out the following steps upon execution;

i) transmitting over said communications medium an MPEG file stored on said computer whose file name is contained within said hilight display object, the transmitted MPEG file representing a corresponding version of said dialog video image in which the control object associated with the branch object being instantiated is highlighted; and

ii) transmitting over said communications medium all of the data elements contained in a text object to be pointed to upon its subsequent instantiation by a text pointer contained within said hilight display object;

g) an unhilight function to carry out the following steps upon execution;

i) transmitting over said communications medium an MPEG file stored on said computer whose file name is contained within said unhilight display object, the transmitted MPEG file representing a corresponding version of said dialog video image in which the control object associated with the branch object being instantiated is unhighlighted; and

ii) transmitting over said communications medium a text string consisting of character blanks to be used to erase any text associated with said branch object that was displayed as a result of executing said hilight function;

h) an application_-- call function to call upon execution one of said application programs that is associated with the branch object being instantiated;

i) a select function to carry out the following steps upon execution;

i) executing the initialize function of the child node object pointed to by said destination pointer if said destination pointer does not contain said null value; and

ii) executing said application_-- call function if said destination pointer contains said null value;

j) up, down, left, and right arrow functions to respectively carry out the following steps upon execution;

i) executing the unhilight function of the branch object being instantiated;

ii) changing the value of said focus pointer of said node object to the value of the corresponding one of said up, down, left, and right branch pointers; and

iii) executing the hilight function of the branch object pointed to by said focus pointer after said changing step has been executed;

4) instantiating all of said display objects associated with said node object and branch objects, each display object having the following attributes;

a) a Motion Picture Experts Group (MPEG) file name of an associated MPEG-format file associated with a corresponding one of said unhighlighted and highlighted versions of said dialog video image;

b) a previous display object pointer to point to a display object immediately preceding the display object being instantiated in a sequence of display objects;

c) a next display object pointer to point to a display object immediately following the display object being instantiated in said sequence of display objects; and

d) a text pointer to point to a text object to be subsequently instantiated that is associated with the display object being instantiated;

5) instantiating all of said text objects associated with said node object and display objects, each text object having the following attributes;

a) a text string associated with the text object being instantiated;

b) a color value to indicate a color in which said text string is to be displayed;

c) X and Y coordinates to respectively indicate horizontal and vertical components of a screen position where said text string is to be displayed;

d) a font value to indicate a font in which said text string is to be displayed; and

e) a size value to indicate a size in which said text string is to be displayed;

6) executing said initialize function of said node object;

7) awaiting the receipt of a keystroke identifier from said communications medium;

8) executing the select function of the branch object pointed to by said focus pointer if said keystroke identifier indicates that a "select" key has been pressed;

9) executing the corresponding one of said up, down, left and right functions of the branch object pointed to by said focus pointer if said keystroke identifier indicates that a corresponding one of "up", "down", "left" and "right" keys has been pressed;

10) executing said find_-- branch function if said keystroke identifier is a response code; and

11) repeating the preceding four steps until the receipt of a keystroke identifier results in the execution of the application_-- call function in one of said branch objects.