Augmented reality operation system and augmented reality operation method

• US 9,628,646 B2
• Filed: 04/22/2016
• Issued: 04/18/2017
• Est. Priority Date: 04/25/2015
• Status: Active Grant

• Current Assignee: KYOCERA Document Solutions, Inc. (Kyocera Corporation)
• Original Assignee: KYOCERA Document Solutions, Inc. (Kyocera Corporation)
• Inventors: Matsuki, Yoshitaka, Kawasaki, Tomohiro, Shimamoto, Kunihiko
• Primary Examiner(s): HON, MING Y
• Application Number:
  US15/135,853
• Publication Number:
  US 20160316081A1
• Time in Patent Office: 361 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/12 Digital output to print unit , e.g. line printer, chain printer
• CPC Class Codes: G06F 3/1205 resulting in increased flexibility in print job configuration, e.g. job settings, print requirements, job tickets
• CPC Class Codes: G06F 3/1253 Configuration of print job parameters, e.g. using UI at the client
• CPC Class Codes: H04N 1/00413 using menus, i.e. presenting the user with a plurality of selectable options H04N1/00464 takes precedence
• CPC Class Codes: H04N 1/00493 Particular location of the interface or console

Abstract

An augmented reality operation system includes: an image-taking device, a display device, an MFP, an augmented reality processing section displaying, on the display device, a synthetic image obtained by synthesizing an operation image illustrating an operation section for operating the MFP with a taken image taken by the image-taking device, and an operation reception section receiving operation on the operation section in the operation image. The operation reception section, based on a position (on the synthetic image) of a hand taken by the image-taking device and a position of the operation section on the synthetic image, receives the operation on the operation section.

Claims

1. An augmented reality operation system comprising:

an image-taking device;

a display device;

a multifunction peripheral (MFP); and

an augmented reality (AR) server including a storage device having an AR operation program stored thereon, a processor, and an audio output device,

wherein the processor executes the AR operation program to thereby;

generate a synthetic image and start to display the synthetic image on the display device, the synthetic image being obtained by synthesizing an operation image illustrating an operation section for operating the MFP with a taken image taken by the image-taking device, the operation image on the synthetic image being synthesized with the taken image in a manner such that the operation image appears, for a user viewing a display screen of the display device, to be arranged in a space between the MFP and the display device;

judge, after the start of the display of the synthetic image, whether or not an image illustrating the user'"'"'s hand is present on the taken image taken by the image-taking device;

upon judgment that the image illustrating the user'"'"'s hand is present on the taken image, (i) delete the image illustrating the user'"'"'s hand included in the taken image from the synthetic image through image processing, (ii) by processing the image illustrating the user'"'"'s hand through image processing to clarify contrast with respect to surrounding images, generate an image of a virtual hand in which at least part of the image illustrating the hand is used, and (iii) further generate, instead of the image illustrating the hand on the synthetic image, the synthetic image in which the generated virtual hand is arranged at the position of the image illustrating the hand, and display the synthetic image on the display device, and

when a position of a fingertip of the image of the virtual hand overlaps with a position of a setting reception image in the operation image included in the synthetic image and also when a size of an image illustrating the fingertip decreases while the position of the fingertip does not change, judge that the setting reception image has been pressed, and make the audio output device stereophonically output operation sound for the operation section in a manner such that the operation sound is outputted at a position in a real space corresponding to a position of the operation section illustrated by the setting reception image on the synthetic image to thereby receive the operation for the MFP performed on the operation section illustrated by the operation image.

2. An augmented reality operation method by an augmented reality (AR) system, the AR system comprising an image-taking device, a display device, an multifunction peripheral (MFP), and an AR server including a storage device having an AR operation program stored thereon, a processor, and an audio output device, the method comprising:

via the AR server by executing the AR operation program by the processor, generating a synthetic image and starting to display the synthetic image on the display device, the synthetic image being obtained by synthesizing an operation image illustrating an operation section for operating the MFP with a taken image taken by the image-taking device, the operation image on the synthetic image being synthesized with the taken image in a manner such that appears, for a user viewing a display screen of the display device, to be arranged in a space between the MFP and the display device;

judging, after the start of the display of the synthetic image, whether or not an image illustrating the user'"'"'s hand is present on the taken image taken by the image-taking device;

upon judgment that the image illustrating the user'"'"'s hand is present on the taken image, (i) deleting the image illustrating the user'"'"'s hand included in the taken image from the synthetic image through image processing, (ii) by processing the image illustrating the user'"'"'s hand through image processing to clarify contrast with respect to surrounding images, generating an image of a virtual hand in which at least part of the image illustrating the hand is used, and (iii) further generating, instead of the image illustrating the hand on the synthetic image, the synthetic image in which the generated virtual hand is arranged at the position of the image illustrating the hand, and displaying the synthetic image on the display device, and when a position of a fingertip of the image of the virtual hand overlaps with a position of a setting reception image in the operation image included in the synthetic image and also when a size of an image illustrating the fingertip decreases while the position of the fingertip does not change, judging that the setting reception image has been pressed, and making the audio output device stereophonically output operation sound for the operation section in a manner such that the operation sound is outputted at a position in a real space corresponding to a position of the operation section illustrated by the setting reception image on the synthetic image to thereby receive the operation for the MIT performed on the operation section illustrated by the operation image.