Coordinating haptics with visual images in a human-computer interface

• US 7,225,115 B2
• Filed: 10/04/2001
• Issued: 05/29/2007
• Est. Priority Date: 10/04/2001
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Novint Technologies, Inc.
• Inventors: Jones, Jake S.
• Primary Examiner(s): Rodriguez, Paul L.
• Assistant Examiner(s): Sharon, Ayal
• Application Number:
  US09/971,379
• Publication Number:
  US 20030100969A1
• Time in Patent Office: 2,063 Days
• Field of Search: 703/2, 703/7, 715/702, 715/821, 710/72, 710/73, 708/130, 708/131, 345/619, 345/156, 345/157, 341/20, 341/22, 382/190, 382/199
• US Class Current: 703/7
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The present invention provides a method of generating haptic interface experiences that is capable of generating haptic companions to existing two-dimensional images. The method exploits the dominance of visual over haptic in humans: human tactile, proprioceptive, and kinesthetic senses do not supply as much fine detail as human vision, and consequently haptic representations do not have to be as detailed as the corresponding visual images to be believable to a human user. An existing two-dimensional image can be used as the visual component of a human-computer interface. A three-dimensional haptic model that is topologically similar to the image can be selected, where topological similarity means that the haptic model provides an acceptable approximation to important characteristics of the haptic experience desired to accompany the two-dimensional image. A relationship can then be established between the image in the visual component of the interface and the haptic model in the haptic component of the interface such that interaction with the image in the visual component is linked with interaction with the haptic model in the haptic component. As an example, a two-dimensional image of a ball can have a haptic model of a sphere associated therewith, and the center and radius established as a relationship such that visual cursor motion at the boundary of the image of the two-dimensional image of the ball is linked with haptic force feedback as the cursor encounters the surface of the haptic model.

Claims

1. A method of generating a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image of an object, comprising:

a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;

b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;

c) Generating a final haptic model by modifying the initial haptic model so that the two-dimensional projection of the correspondence points in the final haptic model are substantially aligned with the correspondence points in the two-dimensional image of the object;

wherein modifying the initial haptic model comprises modifying the initial haptic model according to a set of parameterized operations, and determining parameters for the operations that produce a final haptic model whose correspondence points in a two-dimensional projection are collectively within an error metric of the correspondence points in the image of the object; and

wherein the set of parameterized operations include one or more of the following;

scale along an axis, and relative scale between parts of the model.

2. The method of claim 1, wherein selecting from a library comprises using automated image recognition to select three-dimensional models having two-dimensional projections similar to the object.

3. The method of claim 1, wherein establishing correspondence comprises presenting a representation of the initial haptic model to a user, and accepting from the user specification of points in the initial haptic model to place in correspondence with points in the display of the object.

4. The method of claim 3, wherein accepting from the user specification of points in the initial haptic model to place in correspondence with points in the display of the object comprises monitoring the position of a cursor, and identifying the position of the cursor in the haptic model as a correspondence point responsive to an indication from the user.

5. The method of claim 1, wherein the image of the object is one of a sequence of images, and further comprising determining correspondence points in the image of the object in other images in the sequence, and further comprising generating haptic models for the object in other images in the sequence according to the haptic model correspondence points and the correspondence points determined for the object in each of the other images in the sequence.

6. A method of providing a human-computer interface comprising a two-dimensional visual space and a generated three-dimensional haptic space;

comprising;

a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;

b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;

c) Generating a final haptic model by modifying the initial haptic model so that the two-dimensional projection of the correspondence points in the final haptic model are substantially aligned with the correspondence points in the two-dimensional image of the object.

d) Establishing a correspondence between a haptic cursor in the haptic space and a visual cursor in the visual space;

e) changing the haptic cursor position and visual cursor position responsive to user input;

f) Providing force feedback to the user according to the haptic model and the position of the haptic cursor relative to the haptic model; and

further comprising representing the position of the visual cursor to the user by a cursor graphic overlayed on the two-dimensional image, and further comprising changing the cursor graphic responsive to the position of the haptic cursor relative to the haptic model.

7. The method of claim 6, wherein providing force feedback comprises determining the position of the haptic cursor in the two-dimensional projection of the haptic model corresponding to the two-dimensional image, and establishing the depth relative to that projection of the haptic cursor to be the depth in the haptic space of the part of the haptic model that the haptic cursor first contacts.

8. A method of generating a haptic model corresponding to an object in a visual image, comprising:

a) Selecting a haptic model from a set of haptic models, wherein the selected haptic model approximates desired haptic aspects of the object;

b) Establishing a correspondence between features of the selected haptic model and the representation of the object in the image; and

c) Generating a haptic model from the selected haptic model and the correspondence;

wherein haptic models in the set of haptic models comprise handles, and wherein establishing a correspondence comprises identifying index points in the representation of the object in the image that correspond to the handles of the haptic model.

9. The method of claim 8, wherein the set of haptic models comprises a plurality of haptic models, each associated with a class of objects and each representing a simplified haptic model adaptable to generate haptic models within the haptic perception range of objects within the associated class.

10. The method of claim 8, wherein the image comprises a two-dimensional image.

11. The method of claim 8, wherein the image comprises a two-dimensional image within a sequence of two-dimensional images.

12. The method of claim 8, wherein the handles comprise points of discontinuity in the haptic model.

13. The method of claim 8, wherein the handles comprise points identifying scale of the haptic model.

14. The method of claim 8, wherein the handles comprise points identifying rotation of the haptic model.

15. The method of claim 8, wherein generating a haptic model comprises:

a) Establishing the location in haptic space of the handles of the selected haptic model from the location in the image of the index points in the representation of the object;

b) Generating a haptic model from the location in haptic space of the handles and from the selected haptic model.

16. The method of claim 8, wherein generating a haptic model comprises generating a haptic model by adjusting the selected haptic model so that the locations of the handles of the generated haptic model in haptic space correspond to locations of the index points in the representation of the object in the image.

17. The method of claim 16 wherein adjusting the selected haptic model comprises at least one of:

scaling the haptic model, positioning the haptic model, rotating the haptic model, stretching the haptic model, twisting the haptic model, and morphing the haptic model.

18. A method of generating a haptic model corresponding to an object in a sequence of visual images, where the images in the sequence comprise images of the object at different sequential times, comprising:

a) Selecting a haptic model from a set of haptic models, wherein the selected haptic model approximates desired haptic aspects of the object;

b) Establishing a correspondence between features of the selected haptic model and the representation of the object in a first image in the sequence;

c) Generating a haptic model for the first image from the selected haptic model and the correspondence; and

d) Determining the change in the representation of the object in a second image in the sequence; and

e) Generating a haptic model for the second image from the selected haptic model, the correspondence, and the change.

19. A method of generating a haptic model corresponding to a plurality of objects in a visual image, comprising, for each object in the visual image:

a) Selecting a haptic model from a set of haptic models, wherein the selected haptic model approximates desired haptic aspects of the object;

b) Establishing a correspondence between features of the selected haptic model and the representation of the object in the image; and

c) Generating a haptic model from the selected haptic model and the correspondence;

wherein generating a haptic model comprises modifying the selected haptic model according to a set of parameterized operations, which parameterized operations include one or more of the following;

scale along an axis, and relative scale between parts of the model.

20. A method of generating a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image of an object, comprising:

a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;

b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;

c) Determining an error between the two dimensional projection of a haptic model and the two-dimensional image of the object, where the haptic model is determined by applying a set of parameterized operations to the initial haptic model, which parameterized operations include one or more of the following;

scale along an axis, and relative scale between parts of the model.

d) Determining parameters for the parameterized operations that produce a haptic model corresponding to an error within an error bound.

21. A method of providing for interaction with a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image, comprising:

a) Selecting from a library of haptic models a three-dimensional haptic model that is topologically similar to the object;

b) Establishing a relationship between the two-dimensional image and the three-dimensional haptic model such that interaction with the two-dimensional image is linked to interaction with the three-dimensional haptic model;

c) Establishing a correspondence between a haptic cursor in a haptic space and a visual cursor in a visual space;

d) changing the haptic cursor position and visual cursor position responsive to user input;

e) Providing force feedback to the user according to the haptic model and the position of the haptic cursor relative to the haptic model; and

further comprising representing the position of the visual cursor to the user by a cursor graphic overlayed on the two-dimensional image, and further comprising changing the cursor graphic responsive to the position of the haptic cursor relative to the haptic model.

22. The method of claim 21 wherein interaction with the two-dimensional image comprises motion of a cursor in a visual representation of the image responsive to user manipulation of an input device, and wherein interaction with the three-dimensional haptic model comprises force fed back to the input device responsive to user manipulation thereof.

23. The method of claim 21 wherein interactions are linked such that forces applied by the user to the three-dimensional haptic model affect the display of the two-dimensional image.

24. The method of claim 21 wherein establishing a relationship comprises establishing a correspondence between the three-dimensional haptic model and the two-dimensional image such that perceptive discrepancy therebetween is minimized.

25. The method of claim 21 wherein establishing a relationship comprises establishing a correspondence between the three-dimensional haptic model and the two-dimensional image such that an error representing the difference between the two-dimensional projection of the haptic model into to plane of the image and the two-dimensional image is minimized.