Automatic haptic generation based on visual odometry

• US 10,600,290 B2
• Filed: 12/14/2016
• Issued: 03/24/2020
• Est. Priority Date: 12/14/2016
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Wu, Liwen, Saboune, Jamal, Raynes, Paige
• Primary Examiner(s): Maung, Nay A
• Assistant Examiner(s): Torres, Jose
• Application Number:
  US15/378,686
• Publication Number:
  US 20180165926A1
• Time in Patent Office: 1,196 Days
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Examples of devices, systems, and methods to automatically generate haptics based on visual odometry are disclosed. In one example, a video having a plurality of frames is received and an optical flow estimate between a first frame from the plurality of frames and a second frame from the plurality of frames is created. In this example, the second frame is subsequent to the first frame. An apparent movement of a stationary object between the first frame and the second frame is detected based at least in part on the optical flow estimate in this example and at least one haptic effect corresponding to the apparent movement of the stationary object is generated based at least in part on the optical flow estimate. The generated haptic effect(s) may be output to a haptic file or a haptic output device, or both.

Claims

1. A method of producing a haptic effect comprising:

receiving a video having a plurality of frames;

determining a partial area of a video frame;

detecting an abrupt camera movement, comprising;

determining an optical flow estimate of one or more stationary objects within the partial area between a first frame and a second frame, the second frame subsequent to the first frame;

detecting an apparent abrupt movement of the one or more stationary objects within the partial area between the first frame and the second frame based at least in part on the optical flow estimate and a magnitude threshold;

determining the abrupt camera movement based on the apparent abrupt movement of the one or more stationary objects;

generating a haptic effect corresponding to the abrupt camera movement; and

producing the haptic effect via a haptic output device.

2. The method of claim 1, wherein the video comprises at least one of a video file, a real-time video, or a point-of-view video.

3. The method of claim 1, wherein the first frame and the second frame are consecutive frames in the plurality of frames.

4. The method of claim 1, wherein the second frame is separated from the first frame by at least one frame in the plurality of frames and wherein the second frame is near the first frame.

5. The method of claim 1, wherein the second frame is selected from the plurality of frames based at least in part on a frame rate of the video.

6. The method of claim 1, wherein determining the optical flow estimate comprises:

determining a plurality of feature pixels within the partial area of the first frame; and

generating a plurality of motion vectors corresponding to the plurality of feature pixels within the partial area of the first frame and a corresponding plurality of feature pixels within partial area of the second frame that indicates the apparent abrupt movement of the one or more stationary objects between the first frame and the second frame.

7. The method of claim 6, wherein detecting the apparent abrupt movement of the one or more stationary objects comprises detecting a same direction or a substantially same direction of the plurality of motion vectors.

8. The method of claim 6, wherein the plurality of feature pixels within the partial area is determined based at least in part on a detection of the one or more stationary objects and wherein the partial area surrounds the one or more stationary objects.

9. The method of claim 6, wherein the plurality of feature pixels within the partial area correspond with a plurality of feature pixels from a prior optical flow estimate corresponding to a frame other than the first frame in the plurality of frames.

10. The method of claim 6, wherein the partial area comprises a top-center portion of the first frame and a corresponding top-center portion of the second frame.

11. The method of claim 6, wherein a size of the partial area is based at least in part on a number of detected stationary objects.

12. The method of claim 1, wherein the apparent abrupt movement of the one or more stationary objects is detected by determining a cluster of motion vectors within the optical flow estimate having a same direction or a substantially same direction and having a magnitude exceeding the magnitude threshold.

13. The method of claim 12, wherein the generated haptic effect comprises an intensity corresponding with a magnitude of at least one motion vector within the cluster of motion vectors.

14. The method of claim 1, wherein the haptic effect is output to a haptic track.

15. A non-transitory computer-readable medium comprising processor-executable instructions configured to cause a processor to:

receive a video having a plurality of frames;

determine a partial area of a video frame;

detect an abrupt camera movement based on processor-executable instructions configured to cause a processor to;

determine an optical flow estimate of one or more stationary objects within the partial area between a first frame and a second frame, the second frame subsequent to the first frame;

detect an apparent abrupt movement of the one or more stationary objects within the partial area between the first frame and the second frame based at least in part on the optical flow estimate and a magnitude threshold;

determine the abrupt camera movement based on the apparent abrupt movement of the one or more stationary objects;

generate a haptic effect corresponding to the abrupt camera movement; and

produce the haptic effect via a haptic output device.

16. The non-transitory computer-readable medium of claim 15, wherein the optical flow estimate comprises a plurality of motion vectors and the apparent abrupt movement of the stationary object is detected based on a cluster of the motion vectors pointing in a same or a substantially same direction and having a magnitude exceeding the magnitude threshold.

17. The non-transitory computer-readable medium of claim 16, wherein the haptic effect comprises an intensity parameter based at least in part on a magnitude of a motion vector from the cluster of the motion vectors that corresponds to the abrupt camera movement.

18. A device configured to produce a haptic effect, comprising:

a camera;

a display;

a haptic output device;

a non-transitory computer-readable medium; and

a processor in communication with the camera, the display, and the haptic output device, wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to;

receive a video having a plurality of frames from the camera;

determine a partial area of a video frame;

detect an abrupt camera movement based on processor-executable instructions configured to cause the processor to;

determine an optical flow estimate of one or more stationary objects within the partial area between a first frame and a second frame, the second frame subsequent to the first frame;

detect an apparent abrupt movement of the one or more stationary objects within the partial area between the first frame and the second frame based at least in part on the optical flow estimate and a magnitude threshold;

determine the abrupt camera movement based on the apparent abrupt movement of the one or more stationary objects;

generate a haptic effect corresponding to the abrupt camera movement; and

output the first frame and the second frame to the display; and

cause the haptic output device to produce the haptic effect as the second frame is displayed on the display.

19. The device of claim 18, wherein the apparent abrupt movement of the stationary object is detected by determining a cluster of motion vectors within the optical flow estimate having a same direction or a substantially same direction and having a magnitude exceeding the magnitude threshold.