What are haptics and its significance?
Pedro Lopes: Yeah, it’s one of those beautiful terms that sounds a little jargony at the beginning and sort of scientific, but it’s very, very helpful. So I hope everybody starts using it more in everyday life. It’s a term that we can use when we want to talk about a sensation that it’s neither auditory. We’re not listening to it, we’re not seeing it, but it’s something that we feel with our skin. And it turns out we feel a lot of things with our skin. It’s actually not just whether we’re touching something or not. We feel forces. We feel pressure, textures. So all those things that we couldn’t live without, right? You couldn’t even grab a pen if you didn’t have a sense of touch in the sense of forces. That’s what we mean when we say haptics, but it is something that is a little bit absent in technology, right?
Computers can make sounds and they can shine light through their screens to tell us there’s a new thing happening, but they don’t talk back to our skin except when they vibrate inside of our pockets, which is already helpful, but it’s not as powerful as if they could move our body and show us that new skill. And so, one way that we can get inspired by inventions in medicine is by looking at other ways to move the body. And electrical muscle stimulation is one such way. This is a technology that is extremely old. It looks very sci-fi and new. It uses the little electrodes, little patches that you can put on top of your muscles that look like this and you pass a little electrical current through the inside of your body, a very, very tiny amount of current. And because your muscles are little electrical machines, once they feel the presence of the electrical energy, they will actually contract.
That is how your brain controls your muscles by sending those impulses. So we’re just mimicking those impulses and letting the muscles contract. This technology is old, more than a hundred years old, and in medical rehabilitation is very often used to sort of bring back your muscles. If you had some muscle tissue loss, maybe you broke your leg and you had your leg in a cast for months and you lost, your muscles have atrophied, one way to bring you back to speed is to put the electrodes inside and kind of exercise your muscle safely while you’re still inside of the cast. Now what we are doing is sort of asking the same questions that roboticists have asked with the motors, but we’re asking with the muscle stimulation.
Roboticists said, “Can we help people move with these exoskeletons by sending the computer control to the motors that help the body move and maybe pick up a box that otherwise you couldn’t or balance in a situation where you would have a hard time balancing.” Now we’re doing this directly to your muscles rather than indirectly through this exoskeleton. So we’re sending these computerized electrical impulses at the right time, at the right moment to rebalance your body, to move your body in ways that give you a sense that you’re being controlled, but also a sense that you’re doing the movement. And this is where it gets really different from the exoskeleton. This is really interesting, which is if you’re wearing this exoskeleton, you have to almost imagine Pinocchio’s father puppeteering you. And You’re being moved by it. Whereas if your body is being controlled by the muscle stimulation to some degree, you’re being puppeteered by this computer. And again, we’ll talk about the dystopian angles in just a second, but it is your body that is moving.
You actually feel your muscles contracting. You immediately know which muscle to move with, how much force, because in the end of the day is exactly what you do when you contract your muscles.
Paul Rand: What is happening between your brain and the muscles at this point?
What is mind to muscle connection with haptics?
Pedro Lopes: We move your muscles through these electrical impulses and your brain now has to wrestle with this command that was never initiated by your own body. So it creates a sense of conflict that you feel the muscle is contracting. Your brain is confused by that. Sometimes people call that a sensory conflict similar to how maybe you felt, Paul, if you are in the car and you start to feel nauseated because your eyes see movement, but your body is not moving. That’s typically how people explain this sense of dizziness with car sickness, right?
This is a type of conflict too where your body is moving, but your brain didn’t instruct it to move. So one thing that we do at an engineering level is can we fix that conflict, right? Can we, for example, wait a little bit and instead of moving you very rapidly and very early, almost like you would be surprising, we actually wait for your brain to generate the wish of moving and now you have intended to move. We can still help you move, but we waited for you to move so you no longer have this sensory conflict of I wanted to, I never wanted to move and I’m moving now. You wanted to move and you were moving, but we are able to help you move maybe with the right force, with the right timing, and with the right balance.
Paul Rand: If you’re getting a lot out of the important research that’s shared on Big Brains. There’s another University of Chicago podcast network show that you should check out. It’s called Capitalisn’t. Capitalisn’t used as the latest economic thinking to zero in on the ways that capitalism is and more often isn’t working today. From the debate over how to distribute a vaccine to the morality of a wealth tax. Capitalisn’t clearly explains how capitalism can go wrong and what we can do about it. Listen to Capitalisn’t part of the University of Chicago podcast network.
Paul Rand: Tell me what reactions you’ve seen for users, and how do they experience what you’re talking about?
What are commons reactions for the first use of haptics?
Pedro Lopes: I must say that the most common reaction at the very first time you try this is laughter. And I’ve asked the neuroscientists and I’ve asked the psychologists, and they also don’t know why this happens. It’s very, very interesting. People look at their hands and especially we do a lot of stimulation on the hands, helping people, for example, learn how to operate tools they’ve never seen before. Imagining you’re a carpenter first day of the job, and they tell you, please use that chisel, and you’ve never even heard a word chisel before, and you grab the tool and automatically the muscle stimulation starts to perform that smooth movement-
Paul Rand: Holy cow.
Pedro Lopes: And you go like, “Whoa.”
Paul Rand: Wow.
Pedro Lopes: “That’s what a chisel do.” The first thing they do is look at their hands, very surprised, and they laugh because it’s haptics. It doesn’t need to be explained to people. We actually do very little talking in these experiments because people very rapidly understand, “Oh, this is doing X, this is my body doing.” We’ve actually put people through experiments where we ask them to manipulate tools they’ve never seen before. Really strange tools like a magnetic sweeper. It’s something in factories you use. If you drop a thousand nails on the floor and you go like, “Who’s going to pick those up?” You have a little sweeper to pick them up that has a magnet inside. But if you’ve never worked in a factory, you have no idea what a magnetic sweeper is.
We give this thing to people. We throw a thousand nails on the floor, they have no idea how to use a magnetic sweeper, but we stimulate their bodies to start performing the right movements in the very rapidly. Within seconds after the laughter ceases, they really rapidly understand not only what the tool does, but even what the mechanisms of magnetic sweeper has a little latch that you have to disengage for the magnet to move. They very rapidly understand my hand is doing the right movements. I now understand how this tool works and sometimes even infer the mechanism, and I should really emphasize that we’re not exploring this as an opposition to watching videos and learning from mentors. The idea is you could do all of them now, right? You could have also a video telling you this is a magnetic sweeper and it works like that, but imagine having all this extremely rich information entering your body at the moment you’re learning. And so the other reaction that we’ve had is when we do experiments that endow people with an ability to do something they’ve never done before.
For example, we had a long project that just finished where we were helping people learn musical skills. We put their hands on a piano and we’re going to move their fingers independently to play a melody they’ve never played before.
Paul Rand: Unbelievable.
Pedro Lopes: People love learning a melody like that, right? It’s not the same as watching a video. We’ve actually compared it to watching a video, and this is potentially a little bit faster. It’s not the same as copying a teacher. Copying a teacher is very good. We as humans have evolved by copying each other very well. So we have a big brain infrastructure for copying movements. But there are things that are difficult to copy. I don’t know how much force you put in each key of the piano, the intensity of a note, but if you feel it, you immediately know it. So people have a kick out of feeling those sensations. How much force to put in each melody, you have to play the melody again, unassisted muscle stimulation is out the window.
The device isn’t even there. Do you still remember? In the group that trained with muscle stimulation remembers it better? And in fact, we even found the same thing that you find in education studies, which is not only the muscle stimulation should constantly tell you what the melody is, but it should actually help you where you make mistakes. So we had actually a group that experienced also the muscle stimulation, but it only intervened when you made a mistake. If you’re good, it just lets you play. And then the moment you make a mistake, it says it was actually not that finger. It’s not the index in this part, it’s the ring finger only kicks in there and corrects that mistake. That group remembers it even better than the other.
Paul Rand: Okay, extraordinary. Tell me about devices like Jump Mod or this whole idea of chemical haptics and how that impacts our understanding of perception.
How do chemical haptics impact our understanding of perception?
Pedro Lopes: So Jump Mod is a device that helps you feel a realistic sense of jumping. So if you’re practicing basketball throws in virtual reality, we can actually give you a sense that you’re jumping higher in virtual reality and your jumps are becoming more useful, and you can come out back to the outside world and feel like those jumps are meaningful. Chemical haptics, much like that is an attempt to make you feel like you feel physical things in the virtual reality world, touches there. Vibrations are there, even pain is there. We can simulate pain in a safe way so that if you’re practicing say, what do you do in an emergency situation in a laboratory, if you spill a chemical on you, it becomes really immersive to feel like a sensation on your skin. It’s not a real pain. We simulate pain, but now it’s much more serious than if you’re just watching a video on YouTube. One of those training videos that we watch every year at universities that says, what do you do in a case of a fire emergency?
And you’re just watching the video laid back. Yeah, sure, sure. But we’ve actually put people in these immersive situations, they feel the heat of the fire.
Paul Rand: I’m sure people are hearing about this, but I think the perception and the idea, and arguably the ethics of Neuralink is very different than what you’re talking about. What is Neuralink and how do you think about it?
What is Neuralink?
Pedro Lopes: Yeah, the number one distinction that the listeners should start to make as they think about all these new technologies that are in the body, is are they surface level? Are they outside of my body? Are they things, that I strap on like my smartwatch or maybe like a little EKG, that’s little leads to measure your heartbeats that your doctor might send you home with. Those are surface level devices or is it an implanted device? Does it live primarily inside of my body? Neuralink, unlike the things we investigate in my lab where we’re probing the future of these externalized devices that you can take on and off very rapidly, Neuralink investigates a different future and many other companies like Neuralink exists, I should note, of implanted devices of devices that exist inside of your body. So you can imagine immediately if you’re a person with accessibility needs, if you’re in a wheelchair, paraplegic, this is kind of game changing. And many companies are trying to do the same.
Now more recently. And Paul, maybe that’s why you’re mentioning Neuralink being so much on the news, is Neuralink is also on the news for trying to do, not just reading, but writing to your brain. So I describe when the devices are inside reading from your electrical activity, but neurons in most cells in your body go both ways. So if I can send a little current, just like we did for the electrical muscle stimulators and it moves your body, we can also send a little current to your brain and that maybe causes something to happen in your brain. And one very exciting area that these folks are looking at is can we put those implants close to the parts of your brain that deal with vision, that deal with images that come from your eyes and help maybe restore vision? Neuralink is not the only company trying to do that, but imagine just being able to see lines or a few features of the world if you at the moment cannot see.
So that’s very exciting, and I agree with brain computer interfaces for writing to the brain as being such a powerful tool. Other things are interesting in that realm too. In my lab, we do research using those same ideas, but for helping you learn to move faster or helping you learn new movements. We don’t do it internally like Neuralink do because we don’t want our participants to have to take surgeries to put the electrodes inside. We just do it externally. So you can use big magnets to stimulate the inside of your brain in a safe and external manner, but it all falls into this category of what could we do if we could interface with you by writing to your body, not just reading to your body from your body.
Paul Rand: You used the word a few times in our conversation about a potential dystopian future, and let’s break apart what that word potentially could mean with dystopian, and whether it’s eroding autonomy, giving a different sets of agency, messing with somebody’s head in a completely unexpected way. Talk to me when you think about this potential dystopian threat or barrier, how do you think about it?
What are the potential dystopian threats?
Pedro Lopes: Yeah, there’s so many parallels to what we’ve been seeing in the robotic side that I think it’s a helpful analogy too, right? In self-driving cars, and now we are having this sort of trend of robotic humanoids or little robots that almost look like humans that are going to come into our homes and help us do the dishes. But what happens when they have a glitch, and technically something goes wrong and they, rather than putting the dish in the dishwasher, they slap us. We need to deal with the same type of threats in this human computer integration work, right? If we’re going to design a technology that can help a barista learn the correct amount of force or a golfer or a surgeon, we need to design it with all the fail-safes possible. So the word that you used Paul, agency and sense of control, it’s something that maybe in the last five years or so became actually the most prevalent research topic in my lab.
Most of the papers we publish these days, most of the experiments we do these days are not so much about how to control the muscles because we’ve done that about 10 years ago. But while the muscles are being controlled, how does the human override that control?
Paul Rand: Got it.
Pedro Lopes: What dials do we give to the human to automatically turn on the system or turn off the system, right? So for example, in the split body, how do I say stop stirring the caramel, I want to do it myself, or I’m the one now. I don’t want you to start that type of control. It’s something that folks in robotics are looking at as well. So we do it in many, many forms and shapes with some of our colleagues in neuroscience who will look at what does the sense of control actually means in our brains? What part of our brain might be responsible for the sense of control? Because if we can know where it is, we can also monitor that part. And anytime that part says I shouldn’t be in control, the system is always off. Sometimes we just have a control system inside of our muscle stimulation that if the text that you’re moving turns it off immediately, that’s very similar to what these robotics companies are going to try to put in your home. If the robot moves against you, it will automatically stop.
But again, if you, and I know you had many security people on this show as well, if you know a little bit about computer security, you know that no matter how many layers you put of these fail safes, there’s always one more hack and one more way to attack the system.
Paul Rand: Yes, of course.
Pedro Lopes: So ultimately, there’s also a societal type of discussion that is to be had about these technologies. Where should they exist? Who has access to them and how should they be regulated? Which agency will regulate them? Is it the FDA? Is it some other agency that we need to create that doesn’t even exist, that fundamentally understands that these are not just drugs and food, but are new kinds of technologies, and we might have to need the FAI and a federal of AI regulation to regulate these kinds of things.
Paul Rand: The level of accomplishment that maybe a barista feels, certainly somebody playing the piano, and that human level of accomplishment is a pretty human condition. As you experiment and we think about how machines and bodies share control, how does that change or begin your thinking of an evolution of what being human actually means?
How does this technology change human capabilities?
Pedro Lopes: I think you just touched on the most important question of them all, and that has been always the most important question with technology, right? As a new technology innovates a sport, that sport goes into a little crisis and asks, “What is running, if we now have new running shoes? What is baseball, if we have carbon fiber, something?” So I find this a beautiful human moment, and you’re right, that body integrated technology can revolutionize this, in a little deeper level and make us question even a little bit deeper what means to be human. Something very funny, just to tell an anecdote of something that happened here in the lab in 2019 is that the Guinness Book of World Records, which is this association that sort of cracks crazy amounts of performance in games and in other activities of life, contacted us because of a work we did where we accelerated human reaction time. With the muscle stimulation. We can make a human move a little bit faster than they normally would.
They could maybe catch a falling object that they would normally not catch. And so they contacted us and they did some measurements, and they determined that we had enabled a human to be artificially assisted by 50 milliseconds. And that was, now, it’s still in the book, I believe, the fastest artificially assisted reaction time.
Paul Rand: Wow, cool.
Pedro Lopes: It’s very fun as a scientist to have a record like that. But what that did in my lab was to instigate the question that you said. We said, “Wait a second. We didn’t really help a human, did we? This is all artificial, or is it, or is it not?” And we started debating that ourselves. And that’s actually what led to us thinking about human values and agency and the sense of control was all because the Guinness World Book of Records came and asked us that question whether this was real or not. And you’re right. What happened then is that most of the work in our labs stopped being about helping people to do something when the electrical muscle stimulation is on.
But the question of when you remove it, did you learn something? So we became very interested in this as a learning technology that maybe we’ll supercharge people to be better at jumping, like with Jump Mod or better at running or starting the running a hundred-meter dash because they have muscle stimulation telling them when to move, but not necessarily at pushing them over the edge electronically, but biologically kind of pushing them over the edge, training them to be better humans. So this led us to question also the work we did with piano learning. I don’t want to go to a bar and see the pianist wearing an exoskeleton playing virtuoso. Isn’t that fun for anyone? I don’t think so. I think that’s not core human expression. That’s not art making meaning making.
I want that person to maybe have new ideas as they’re playing at home. What if I read the movement comes about and they have a new artistic intuition? And that could be interesting, but not to automate, not to replace humans. So I tend to think the human condition is augmenting ourselves with tools but not replacing ourselves. Right? We want to be in here making meaning and artistically expressing our thoughts, but not replacing us, just augmenting us to do more and do in new ways. I think that’s something that we enjoy doing as animals on this planet.
Lea Ceasrine: Big Brains is a production of the University of Chicago Podcast Network. We’re sponsored by the Graham School. Are you a lifelong learner with an insatiable curiosity? Access more than 50 open enrollment courses every quarter. Learn more at graham.uchicago.edu/bigbrains. And if you like what you heard in our podcast, please leave us a rating and review. The show is hosted by Paul M. Rand, and produced by me Lea Ceasrine, with help by Eric Fey. Thanks for listening.
