Gesture-modified diagram for retrieval of image resembling diagram, with parts selectable for further interactive retrieval

• US 5,010,500 A
• Filed: 01/26/1989
• Issued: 04/23/1991
• Est. Priority Date: 01/26/1989
• Status: Expired due to Term

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Stults, Robert A., Makkuni, Ranjit, Zdybel, Frank Jr.
• Primary Examiner(s): CLARK, DAVID
• Application Number:
  US07/303,351
• Time in Patent Office: 817 Days
• Field of Search: 382/13, 382/59, 382/56, 364/518, 364/521, 340/709, 340/706, 178/19
• US Class Current: 715/863
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041

Abstract

Recorded video segments are retrieved and displayed through a user interface that employs gestures. The user provides gestures by moving a mouse, and the resulting signals indicate a diagram. Data identifying the diagram is used to access a data structure to obtain data indicating which of a set of interactive line drawings includes features most closely resembling the diagram. The interactive line drawing with features closest to the diagram can then be displayed. When the user selects a part of the line drawing, a menu is displayed that includes a description of the video segments that relate to that part. To assist the user in providing gestural input, a preliminary diagram is displayed, reducing the amount of information the user must provide. The user can change parts of the diagram to obtain a modified diagram resembling features that appear in a desired line drawing. The diagrams and interactive line drawings can be displayed on a workstation display screen, while the video segments can be displayed on a video monitor screen by a video disk player under control of the workstation CPU.

Claims

1. An information retrieval system comprising:

presenting means for providing presentations to a user, the presenting means comprising display means for providing visual presentations to the user;

selection input means for providing signals indicating user selections within visual presentations provided by the display means;

gesture input means for providing data basaed on gestures of the user;

memory for storing a plurality of interactive items, the display means being capable of providing a respective interactive presentation based on each of the interactive items;

the respective interactive presentations of a first one of the interactive items including a first image, the first image including a plurality of parts including a first part;

the memory further being for storing, for each of the parts of the first image, a respective set of related information items based on each of which the presenting means can provide a respective presentation for providing information to the user;

the respective set of related information items of the first part including a first information item;

the first image further including, for each of the parts, a respective selectable bounded area that is selectable by the user with the selection input means in requesting presentations based on information items from the respective set of related information items;

first means for using visual feature data from the gesture input means to determine whether the first image includes features resembling a desired set of visual features and, if so, for causing the display means to provide the first interactive item'"'"'s respective interactive presentation that includes the first image;

the visual feature data being based on a gesture indicating the desired set of visual features;

the visual feature data indicating the gesture'"'"'s path of motion; and

second means for receiving a request signal requesting the first information item'"'"'s respective presentation and for responding to the request signal by causing the presenting means to provide the first information item'"'"'s respective presentation for providing information to the user;

the request signal including a selection signal from the selection input means, the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area.

2. The system of claim 1 in which the first means is further for using the visual feature data to obtain an identifier of the desired set of visual features, the system further comprising a data structure for providing data indicating the first interactive item when accessed with the identifier of the desired set of visual features;

the first means further being for accessing the data structure with the identifier of the desired set of visual features to obtain the data indicating the first interactive item.

3. The system of claim 1 in which the first informwtion item is a video item.

4. The system of claim 3 in which the display means comprises first and second display screens, the first display screen being a workstation screen for providing the first interactive item'"'"'s respective interactive presentation and the second display screen being a video screen for providing the first information item'"'"'s respective presentation.

5. The system of claim 3 in which memory includes a video storage medium storing the first information item, the second means comprising a video player connected for accessing the video storage medium, the video player being for accessing the first information item on the video storage medium in response to the request signal requesting the first information item'"'"'s respective presentation.

6. The system of claim 1 in which the gesture input means comprises a mouse with which the user can perform the gesture indicating the desired set of visual features, the mouse providing the visual feature data to the first means.

7. A system comprising:

presenting means for providing presentations to a user, the presenting means comprising display means for providing visual presentations to the user;

selection input means for providing signals indicating user selections within visual presentations provided by the display means;

gesture input means for providing data based on gestures of the user;

memory for storing a plurality of interactive items, the display means being capable of providing a respective interactive presentation based on each of the interactive items;

the respective interactive presentation of a first one of the interactive items including a first image, the first image including a plurality of parts including a first part;

the memory further being for storing, for each of the parts of the first image, a respective set of related information items based on each of which the presenting means can provide a respective presentation;

the respective set of related information items of the first part including a first information item;

the first image further including, for each of the parts, a respective selectable bounded area that is selectable by the user with the selection input means in requesting presentations based on information items from the respective set of related information items;

first means for using diagram data from the gesture input means to determine whether the first image resembles a diagram and, if so, for causing the display means to provide the first interactive item'"'"'s respective interactive presentation that includes the first image;

the diagram data being based on a gesture indicating the diagram;

the diagram data indicating the gesture'"'"'path of motion; and

second means for receiving a request signal requesting the first information item'"'"'s respective presentation and for responding to the request signal by causing the presenting means to provide the first information item'"'"'s respective presentation;

the request signal including a selection signal from the selection input means, the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area.

8. The system of claim 7 in which the first means is further for using the diagram data to obtain an identifier of the diagram, the system further comprising a data structure for providing data indicating the first interactive item when accessed with the diagram'"'"'s identifier;

the first means further being for accessing the data structure with the diagram'"'"'s identifier to obtain the data indicating the first interactive item.

9. The system of claim 7 in which the first information item is a video item.

10. The system of claim 9 in which the display means comprises first and second display screens, the first display screen being a workstation screen for providing the first interactive item'"'"'s respective interactive presentation and the second display screen being a video screen for providing the first information item'"'"'s respective presentation.

11. The system of claim 9 in which memory includes a video storage medium storing the first information item, the second means comprising a video player connected for accessing the video storage medium, the video player being for accessing the first information item on the video storage medium in response to the request signal requesting the first information item'"'"'s respective presentation.

12. The system of claim 7 in which the gesture input means comprises a mouse with which the user can perform the gesture indicating the diagram, the mouse providing the visual feature data to the first means.

13. A method of operating an information retrieval system that includes:

presenting means for providing presentations to a user, the presenting means including display means for providing visual presentations to a user;

selection input means for providing signals indicating user selections within visual presentations provided by the display means;

gesture input means for providing data based on gestures of the user;

memory for storing a plurality of interactive items, the display means being capable of providing a respective interactive presentation based on each of the interactive items;

the respective interactive presentations of a first one of the interactive items including a first image, the first image including a plurality of parts including a first part;

the memory further being for storing, for each of the parts of the first image, a respective set of related information items based on each of which the presenting means can provide a respective presentation for providing information to the user;

the respective set of related information items of the first part including a first information item;

the first image further including, for each of the parts, a respective selectable bounded area that is selectable by the user with the selection input means in requesting presentations based on information items from the respective set of related information items;

the method comprising steps of;

receiving visual feature data from the gesture input means, the visual feature data being based on a gesture indicating a desired set of visual features, the visual feature data indicating the gesture'"'"'s path of motion;

using the visual feature data to determine whether the first image includes features resembling the desired set of visual features and, if so, causing the display means to provide the first interactive item'"'"'s respective interactive presentation that includes the first image;

receiving a request signal requesting the first information item'"'"'s respective presentation;

the request signal including a selection signal from the selection input means, the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area; and

in response to the request signal, causing the presenting means to provide the first information item'"'"'s respective presentation for providing information to the user.

14. The method of claim 13 in which the step of receiving the visual feature data further comprises a substep of causing the display means to present a preliminary set of visual features;

the gesture indicating a change in the preliminary set of visual features that would produce the desired set of visual features.

15. The method of claim 14 in which the preliminary set of visual features is a preliminary diagram and the desired set of visual features is a modified diagram.

16. The method of claim 13 in which the step of causing the presenting means to provide the first information item'"'"'s respective presentation further comprises substeps of:

in response to the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area, presenting a selectable menu item, the selectable menu item including a description of the first information item'"'"'s respective presentation;

receiving a menu item selection signal from the selection input means, the menu item selection signal indicating user selection of the selectable menu item; and

in response to the menu item selection signal, causing the presenting means to provide the first information item'"'"'s respective presentation.

17. A method of operating a system that includes:

presenting means for providing presentations to a user, the presenting means including display means for providing visual presentations to the user;

selection input means for providing signals indicating user selections within visual presentations provided by the display means;

gesture input means for providing data based on gestures of the user;

memory for storing a plurality of interactive items, the display means being capable of providing a respective interactive presentation based on each of the interactive items;

the respective interactive presentation of a first one of the interactive items including a first image, the first image including a plurality of parts including a first part;

the memory further being for storing, for each of the parts of the first image, a respective set of related information items based on each of which the presenting means can provide a respective presentation;

the respective set of related information items of the first part including a first information item;

the first image further including, for each of the parts, a respective selectable bounded area that is selectable by the user with the selection input mens in requesting presentations based on information items from the respective set of related information items;

the method comprising steps of;

receiving diagram data from the gesture input means, the diagram data being based on a gesture indicating a diagram, the diagram data indicating the gesture'"'"'s path of motion;

using the diagram data to determine whether the first image resembles the diagram and, if so, causing the display means to provide the first interactive item'"'"'s respective interactive presentation that includes the first image;

receiving a request signal requesting the first information item'"'"'s respective presentation;

the request signal including a selection signal from the selection input means, the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area; and

in response to the request signal, causing the presenting means to provide the first information item'"'"'s respective presentation.

18. The method of claim 17 in which the step of receiving the diagram data further comprises a substep of causing the display means to present a preliminary diagram;

the gesture indicating a change in the preliminary diagram that would produce a modified diagram.

19. The method of claim 17 in which the step of causing the user output means to provide the first information item'"'"'s respective presentation further comprises substeps of:

in response to the selection signal indicating user selection of the first part'"'"'s respective selectable bounded area, presenting a selectable menu item, the selectable menu item including a description of the first information item'"'"'s respective presentation;

receiving a menu item selection signal from the selection input means, the menu item selection signal indicating user selection of the selectable menu item; and

in response to the menu item selection signal, causing the user output means to provide the first information item'"'"'s respective presentation.

20. A method of operating a system that includes:

display means for providing visual presentations to a user;

memory for storing a plurality of visual items, the display means being capable of providing a respective visual presentation based on each of the visual items;

the visual items including a first visual item; and

gesture input means for providing data based on gestures of the user;

the method comprising steps of;

causing the display means to provide a preliminary diagram presentation that includes a preliminary diagram;

receiving diagram change data from the gesture input means, the diagram change data being based on a gesture of the user indicating a change in the preliminary diagram that would produce a modified diagram, the diagram change data indicating the gesture'"'"'s path of motion; and

using the diagram change data to determine whether the first visual item'"'"'s respective visual presentation includes an image resembling the modified diagram and, if so, causing the display means to provide the first visual item'"'"'s respective visual presentation.

21. The method of claim 20 in which the system further includes a data structure for providing, when accessed with an identifier of the modified diagram, data indicating the first visual item;

the step of using the diagram change data comprising substeps of;

obtaining the identifier of the modified diagram based on the diagram change data; and

accessing the data structure with the modified diagram'"'"'s identifier to obtain the data indicating the first visual item.

22. The method of claim 20 in which the system further includes selection input means for providing signals indicating user selections within visual presentations provided by the display means;

the first visual item'"'"'s respective visual presentation including a selectable bounded area, the memory further being for storing a first information item related to the first visual item;

the system further including presenting means for providing a presentation based on the first information item;

the method further comprising steps of;

receiving a selection signal from the selection input means indicating user election of the selectable bounded area; and

in response to the selection signal, presenting a selectable menu item to the user, the selectable menu item being selectable to request the first information item'"'"'s respective presentation by the presenting means.

23. The method of claim 22 in which the first information item is a video item.

24. The method of claim 20, further comprising a step of responding to the diagram change data by causing the display means to provide a modified diagram presentation that includes the modified diagram.