Improving the Realism of Haptic Interaction for Teaching of Sensorimotor Skills

Investigators: Maxim Kolesnikov and Miloš Žefran

Support: NSF grant CMS-0600658 and UIC College of Dentistry


Recently, haptic simulators have shown great potential in teaching sensorimotor skills. This is especially true for areas where the traditional training technique is expensive, such as medical and dental training. The goal of this work is to address several key areas where improvement is needed to increase the realism of haptic interaction for teaching of sensorimotor skills. These key areas are:

1. Haptic rendering algorithms. Modern penalty-based haptic rendering algorithms do not produce realistic forces in certain situations. A new method for haptic rendering needs to be implemented based on physics rather than on heuristics. At the same time the new method has to be fast enough to be executed in the allowed time frame.

2. Recording haptically augmented training video aids. This would provide a way to convey the information necessary to master a new skill from a teacher to a trainee. Several haptic playback schemes have been proposed. Evaluation of the effectiveness of different schemes for learning the task is required.

3. Collaborative haptic environments. In these interactive environments the information to master a sensorimotor skill could be conveyed from a teacher to several trainees simultaneously. For this idea to work, methods for running haptically augmented online demonstrations have to be developed. Other issues, such as switching between user behaviors and arbitration mechanisms need to also be addressed.

Work in the first two areas is described below.

Energy Based Haptic Rendering Algorithm

Existing penalty-based 6-DOF haptic rendering algorithms have a built-in deficiency: they do not properly account for the orientation. Consider two different scenarios:

Whether a pure force or torque is applied to a cube resting on a plane, existing penalty-based methods would yield roughly the same contact force – the force perpendicular to the surface of the plane and a function of the penetration depth defined in strictly translational sense. In reality, pure instantaneous torque applied to the object should correspond to the contact wrench that has only torque components.

Instead of computing translation-based penetration depth, our proposed haptic rendering algorithm takes the full rigid-body configuration (translation and rotation) into account. We look at the set of all possible configurations that correspond to the situation when there is no collision, and find the closest-point projection of the initial inadmissible configuration onto a set of admissible configurations. For this approach to work a suitable distance metric function in configuration space SE(3) has to be found. This function is required to be invariant to the choice of reference frames, to allow to obtain scale-invariant solutions and to be based on physics rather than heuristics and be theoretically justified. We propose to use the kinetic energy function as the distance metric function.

Once the optimal admissible configuration is found, generalized penetration depth is computed and feedback wrench is determined using a standard penalty-based approach.

If the objects are convex, then in the planar 2D case the optimal closest-point configuration can be found analytically as a solution of a constrained optimization problem. In the 3D case we can use bisecting planes to find cross-sections of the object, and then use the exact solution method to find the optimal configuration for each 2D cross-section. Then these solutions can be combined to get the solution in 3D.

Haptically Augmented Video Training Aids

Haptic applications can be used in two phases of teaching of a motor task: in the initial expert-influenced phase and in the component strengthening phase which is a sole responsibility of the trainee. It is known that the haptic applications are very useful during the second phase, but their use is limited in the first phase. The challenge is to find a suitable way to convey the information necessary to master a new skill from an expert to a trainee. Haptically augmented video training aids are a solution. First, such video is recorded by the teacher on a simulator. Then it can be played back either with haptic augmentation or as a regular video.

Each sensorimotor skill has two components: position and force. To acquire the skill is to be able to reproduce both components with a certain degree of accuracy. Both components can be recorded by using a force sensor and tracking devices. But force and position cannot be both displayed simultaneously. Any form of haptic augmentation should pursue two goals: convey both the position and force data to the possible extent and achieve educational objectives, i.e. it should be an aid in sensorimotor skill acquisition, not a distraction.

One possible method of haptic augmentation is haptic playback. Visual display is used to produce visual cues (Corno, 2005). As long as the user follows the target with a reasonable degree of accuracy, he will feel the same force that was recorded and will reproduce the same spatial trajectory. For this approach to work, the user is assumed to be cooperative and active participant. It should be noted that the trajectory of the visual target and the recorded trajectory may differ considerably. An important question that we intend to investigate is whether this has any significant impact on the achievement of educational objectives.

A simpler method of haptically augmenting training videos is penalty-based haptic pseudo-playback (haptic virtual fixtures). In this case the priority is given to following the correct trajectory. Position and orientation of the user controlled object is recorded along with some auxilliary data. When played back, the recorded trajectory is displayed and haptic device renders a guiding force to drag the tool towards the desired recorded position. Here the user may not be cooperative. A force sensor is not needed in this case.

For the purpose of testing various haptic augmentation schemes we use the PerioSim which is a fully functional haptic application intended to teach dental students to learn the correct use of dental tools in various dental procedures. It has been developed jointly by the UIC College of Dentistry and the UIC College of Engineering.

In addition to haptic virtual fixtures to control the correct trajectory, virtual fixtures in other modalities may be used, i.e. auditory virtual fixtures to control the orientation on 3-DOF haptic devices.

Movie 1: one user ("an expert") records a probing procedure on PerioSim
Movie 2: another user ("a student") loads the recorded procedure, then plays it back in the observation mode, learning mode and testing mode. At the end the performance index is displayed.

Future Work

Several issues need to be addressed in future. For the energy based haptic rendering algorithm the feasibility of obtaining a closed form solution directly for general three-dimensional case without resorting to plane bisection technique needs to be investigated. If found, such a solution would allow for even faster and computationally less complex haptic rendering. Also, the performance of the algorithm needs to be evaluated from the perceptual point of view. It is necessary to study how humans respond to various haptic rendering schemes and to learn more about the users’ psycho-haptic experiences.

For producing haptically augmented training video aids it needs to be investigated whether the haptic playback algorithm is effective in achieving educational objectives. The main question here is how closely the target tracking performance correlates with learning performance. Human studies will be performed and quantitative metrics will be used.

An extensive area that is also planned to be addressed is the extension of results in haptic playback to collaborative haptic environments. This is a potentially promising area. Having several users interact with a haptic environment at the same time would be useful in situations when an instructor demonstrates a sensorimotor skill to students. Results in haptic playback and/or penalty-based haptic pseudo-playback can be applied with the main difference being the ability to convey haptic information at the same time instance. However, other issues need also be addressed. They include switching between user behaviors (active and passive), dealing with a wide range of dynamic behaviors that users can display when they interact, mechanisms for access control and arbitration between the intensions of the users.

Please, have a look at the slides of a more technical presentation of this project.