HapLearn — Iteration 2
Objective
Our goal is to build a prototype for multimodal data visualization and exploration which utilizes haptic representation of data to improve user experience.
In this iteration, we will focus on scatter plots with the goal of improving navigation, analysis, and rate of information transfer.
Motivation
Based on feedback from Iteration 1, and multiple revisions of our project goal (due to technical issues), we narrowed down the scope of our project for this iteration. In Iteration 1, we were aiming to achieve our goal only through haptic feedback (for example for visually impaired users), however, we realized that this is much more challenging. As noted by Bovard et al. (2018), users found that the haptic modality is easy to detect, however, it is more difficult to use to convey rich information than the visual modality. Keeping this in mind, we used a scatter plot graph that had been famously optimized by Hans Rosling to show a high and rich amount of information visually. Then, we tried to find some user experience challenges for this type of bubble chart that could perhaps be improved using haptics (ie. fill in the gaps). The following points are the key features that could be improved as well as break down of tasks between the members:
- Exploration of force-feedback based haptic rendering of texture for overlapping data points using Fisica (Kattie)
- Exploration of force-feedback based haptic rendering of data navigation using PID + GUI development (Rayan)
- Exploration of force-feedback haptics using Fisica to encode a data dimension (population) via damping (Ahmed)
- Exploration of vibrotactile haptic feedback for encoding a 4th data dimension (literacy rate) as well as for guiding data analysis (Karthik)
In this Iteration, I worked on various things for example enhancing some of the GUI features, merging the Fisica library with the GUI code, and texture rendering for overlapping data points. Since overlapping data points in bubble graphs are difficult to distinguish visually (even when the points are “highlighted” in some way) I wanted to see if I can represent a “crowded” area of the graph haptically. I also wanted to see if it is possible to create a fine enough sensation to feel a “bump” for each overlapping point.
Overview
I started with working on the main GUI and tried to implement the Fisica library to implement a “texture” while moving over the “bubbles”, however, this was not successful (explained in the section below). I was able to see the FBodies, however, they did not generate any force feedback. I was envisioning a feeling similar to moving your finger over the edge of sheets of paper laying on top of each other.
I decided to go back to the code from Iteration 1 and implement the textures in the same manner. My goal was to implement a slight “bumb” or “vibration” when Haply moves over a bubble edge since it was noted in Edson (2013) that vibration, stickiness, and viscosity had higher perceivability in 2D scatter plot visualization.
To achieve this, I overlayed the bubble graph with FCircles that were set to be sensors, and if the Haply end effector touched the sensor region, the position of the Haply would be slightly adjusted. This created some vibrations when Haply was going over the regions with a lot of overlapping points, however, I noticed that the vibrations were not consistent. It was also very difficult to get a consistent “bump” feeling from Haply, as different FCircle sizes would produce a stronger/weaker force even if the position was being adjusted in the same direction and magnitude. I am not sure what causes this, however, sometimes the force would either be too strong or be negligible. In addition, sometimes Haply may also produce random and uncontrolled vibrations when going over these regions.
if (s.h_avatar.isTouchingBody(c2)){
s.h_avatar.adjustPosition(-0.08,0);
}
The regions with more overlap definitely produced more vibrations, however, there was no guarantee that a force is felt if there are only two bubbles overlapping. My conclusion is that this form of representation is perhaps no better than the visual representation, and it doesn’t add anything valuable to the user experience. However, it may be possible to attain higher precision feedback using another device.
Since I was not satisfied, I also wanted to try to simulate “stickiness” in the bubbles that have more overlaps (ie. high overlap = stickier). The code was not too different from before, however now I am taking the direction of contact with the bubble into account. So the position will be adjusted accordingly, creating a “pulling in” sensation on the end effector.
if (s.h_avatar.isTouchingBody(c7) && s.h_avatar.getY() > 4){
s.h_avatar.adjustPosition(0.1,-0.17);
}else if (s.h_avatar.isTouchingBody(c7) && s.h_avatar.getY() <= 4){
s.h_avatar.adjustPosition(0.1,0.17);
}else if (s.h_avatar.isTouchingBody(c7) && s.h_avatar.getX() <= 16.9){
s.h_avatar.adjustPosition(0.5,0);
}else if (s.h_avatar.isTouchingBody(c7) && s.h_avatar.getX() > 16.9){
s.h_avatar.adjustPosition(-0.5,0);
}
This form of the simulation was much better than I expected, and the bubbles that had higher overlap had a significantly different feeling and were stickier. There were also some uncontrolled and random vibrations for the bubbles with a lot of overlap ( for example 0.2,0.4), but the bubbles with fewer overlaps worked better (0.64, 0.58). Even using stickiness, I am not convinced that Haptics in this form can add anything valuable where the visual modality may be failing. This is because it is still very hard to tell whether, for example, there are 2 overlapping bubbles vs. 3 overlapping bubbles just based on the sensation.
For a future improvement, it might be worthwhile to experiment with a “grainy” texture as well, where the higher overlapping bubbles feel grainers than the lower overlapping bubbles.
Challenges with mering the code
My work on the GUI features was limited to adding buttons to navigate the user to the MIN Y and MAX Y values. Our goal was to have all of the functionality working in one GUI. However, we soon started facing the challenges that some of the other teams had faced earlier as we realized that some of us are using the PID loop and some are using Fisica. Sadly, we failed to realize all the functionality in a single GUI after spending numerous. The main issue is that the Haply end effector is represented differently in each system. However, Rayan has managed to add damping in a different mode (asynchronous) in the main GUI.
Reflexions
Overall, our team was very satisfied with the haptic navigation where the name of the country is entered and the haply will snap to that location. This can be very useful for crowded graphs where names are hard to see and difficult to find specific points.
We also think that representing the 4th dimension (literacy rate) with vibrotactile worked very well and it was very obvious which countries had increasing/decreasing literacy rates.
Using damping to represent the size of the circles (bigger size = more damping) worked well, however, the haptics is only reinforcing the visual modality in this case and is not really “filling in a gap”. We could perhaps map damping to an invisible dimension in the subsequent iteration.
Next iteration plans
For the next iteration, we need to work on merging and combining the codes and creating a single GUI. We also need to work on representing the fourth dimension of our data using Haply as well as improving the vibrotactile representation. We also would like to obtain some feedback from our classmates to improve the prototype!
References
Bovard, Pooja & Sprehn, Kelly & Cunha, Meredith & Chun, Jaemin & Kim, Seungjun & Schwartz, Jana & Garver, Sara & Dey, Anind. 2018. Multi-modal Interruptions on Primary Task Performance. 10.1007/978–3–319–91467–1_1.
Edson A. G. Coutinho and Selan R. dos Santos. 2013. Enhancing 2D scatter plot visualization of multivariate data with haptic effects. In Proceedings of the 12th Brazilian Symposium on Human Factors in Computing Systems (IHC ‘13). Brazilian Computer Society, Porto Alegre, BRA, 256–259.
