System and method for providing guidance or feedback to a user

• US 10,521,669 B2
• Filed: 11/14/2016
• Issued: 12/31/2019
• Est. Priority Date: 11/14/2016
• Status: Active Grant

• Current Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Dayal, Rajiv, Chen, Tiffany L., Ching, Patrick K., Northcutt, Brandon D.
• Primary Examiner(s): Perlman, David
• Application Number:
  US15/351,321
• Publication Number:
  US 20180137359A1
• Time in Patent Office: 1,142 Days
• Field of Search: None
• CPC Class Codes: A63B 24/0003 Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/4451 User profiles; Roaming
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G09B 19/0069 Engineering, e.g. mechanical, electrical design computer engineering G09B19/0053
• CPC Class Codes: G09B 19/0092 Nutrition
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B

Abstract

A device for providing guidance or feedback to a user. The device includes a camera configured to detect image data indicating a user performance of an activity. The system includes a guidance unit connected to the camera. The guidance unit is configured to identify the activity based on image processing of the image data or an identification of the activity from the user. The guidance unit is also configured to determine a criteria associated with the activity. The guidance unit is also configured to determine a user performance of the activity based on the image data. The guidance unit is also configured to determine feedback based on a comparison of the criteria and the user performance of the activity, the feedback indicating an improvement or suggestion for the user. The system also includes an output unit connected to the guidance unit, the output unit configured to output the feedback.

Claims

1. A wearable neck device for providing guidance or feedback to a user, the wearable neck device comprising:

a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion configured to extend across a first shoulder of the user and rest on a front body portion of the user and the second side portion configured to extend across a second shoulder of the user and rest on the front body portion of the user;

a camera located on the first side portion or the second side portion and configured to detect image data of a first angle of a user performance of an activity;

a GPS unit located within the body and configured to detect location data;

a memory located within the body and configured to store a learned model associated with the activity;

an activity detection unit located in the body, connected to the camera and the GPS unit, and configured to automatically analyze the image data to identify a presence of one or more objects associated with the activity, compare the one or more objects associated with the activity to the learned model and identify the activity based on the location data and the comparison of the one or more objects associated with the activity to the learned model;

a guidance unit located in the body, connected to the camera and the IMU, and configured to;

obtain, from a second camera of another device, additional image data of a second angle of the user performance of the activity,

determine a series of instructions associated with the activity and including a plurality of steps to be performed by the user based on the learned model,

determine an action being performed within the detected image data,

determine a current step of the plurality of steps of the series of instructions associated with the activity that is being performed based on the action,

determine that the current step of the activity has been completed based on the image data of the first angle of the user performance of the activity and the additional image data of the second angle of the user performance of the activity, and

determine an instruction associated with a next step within the series of instructions in response to determining that the current step has been completed; and

an output unit located on the body, connected to the guidance unit, and configured to output the instruction to the user.

2. The wearable neck device of claim 1, further comprising an input unit configured to detect input data from the user identifying the activity.

3. The wearable neck device of claim 1, wherein the learned model stored in the memory is periodically updated.

4. The wearable neck device of claim 1, wherein the output unit includes a speaker configured to provide an audio output or a vibration unit configured to provide a tactile output.

5. The wearable neck device of claim 1, wherein the one or more objects identified by the activity detection unit are one or more stationary objects unconnected to the user.

6. A wearable neck device for providing guidance or feedback to a user, the wearable neck device comprising:

a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion and the second side portion configured to extend across a shoulder of the user and rest on a front body portion of the user;

a camera located on the first side portion or the second side portion and configured to detect image data of a first angle of a user performance of an activity;

an inertial measurement unit (IMU) located within the body and configured to detect movement data associated with the user during the user performance of the activity;

a GPS unit located within the body and configured to detect location data;

an activity detection unit connected to the camera and the GPS unit, and configured to;

automatically analyze the image data to identify the activity based on a presence of one or more objects associated with the activity, and

identify the activity based on the presence of the one or more objects associated with the activity and the location data;

a guidance unit connected to the camera and the IMU, and configured to;

obtain, from a second camera of another device, additional image data of a second angle of the user performance of the activity,

determine a series of instructions associated with the activity and including a plurality of steps to be performed by the user,

determine an action being performed within the image data,

determine a current step of the plurality of steps of the series of instructions associated with the activity that is being performed based on the image data, the additional image data and the movement data,

determine that the current step of the activity has been completed based on the image data of the first angle of the user performance of the activity and the additional image data of the second angle of the user performance of the activity, and

determine a instruction associated with a next step within the series of instructions; and

an output unit connected to the guidance unit, the output unit configured to output the instruction.

7. The wearable neck device of claim 6, further comprising an input unit configured to detect input data from the user indicating the activity.

8. The wearable neck device of claim 6, further comprising:

a memory configured to store a learned model;

wherein the guidance unit is configured to determine that the current step of the activity has been completed by comparing the image data to the learned model stored in the memory.

9. The wearable neck device of claim 6, wherein the output unit includes a speaker configured to provide an audio output or a vibration unit configured to provide a tactile output.

10. A method of providing guidance or feedback to a user of a wearable neck device, the method comprising:

providing, by the wearable neck device, a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion configured to extend across a first shoulder of the user and rest on a front body portion of the user and the second side portion configured to extend across a second shoulder of the user and rest on the front body portion of the user;

detecting, by a camera located on the first side portion or the second side portion of the body of the wearable neck device, image data of a first angle of a user performance of an activity;

detecting, by a GPS unit, location data associated with the wearable neck device;

storing, by a memory located within the body, a learned model associated with the activity;

automatically analyzing, by an activity detection unit, the image data to identify a presence of one or more objects;

identifying, by the activity detection unit, the presence of the one or more objects associated with the activity;

comparing, by the activity detection unit, the one or more objects associated with the activity to the learned model;

identifying, by the activity detection unit, the activity based on the location data and the comparison of the one or more objects associated with the activity to the learned model;

obtaining, by a guidance unit and from a second camera of another device, additional image data of a second angle of the user performance of the activity;

determining, by the guidance unit, a series of instructions associated with the activity and including a plurality of steps to be performed by the user based on the learned model;

determining, by the guidance unit, an action being performed within the image data;

determining, by the guidance unit, a current step of the plurality of steps of the series of instructions associated with the activity that is being performed based on the action;

determining, by the guidance unit, that the current step of the activity has been completed based on the image data of the first angle of the user performance of the activity and the additional image data of the second angle of the user performance of the activity;

determining, by the guidance unit, an instruction associated with a next step within the series of instructions; and

outputting, by an output unit located on the body, the instruction.

11. The method of claim 10, wherein identifying the activity includes receiving, by an input unit, input data from the user indicating the activity.

12. The method of claim 10, wherein the method further comprises periodically updating the learned model stored in the memory.