For my last interview at CSUN’s Accessible Tech conference this year, I spoke with Michael Karpelson of the Wyss Institute. The audio for this conversation will be in NosillaCast #830 which will be published on 4 April 2021. You can learn more about the Wyss Institute and Michael’s work by going to wyss.harvard.edu/… You can also follow them on Twitter @wyssinstitute
Below is a transcript of our conversation created by the service https://otter.ai and corrected by Steve Sheridan. The conversational style has been maintained with the transcript.
Allison Sheridan
Michael Carlson is a senior staff engineer at Harvard School of Engineering and Applied Sciences. At the CSUN assistive tech conference, he told me about a unique idea he’s been working on to help with age-related vision loss. Welcome to the show, Michael.
Michael Karpelson
Thank you. It’s great to be here.
Allison Sheridan
All right. So when we talked, you mentioned that the work you’re doing is part of something called the Wyss Institute. Can you tell us about that?
Michael Karpelson
Absolutely. So Wyss Institute is a multidisciplinary research institute that’s part of Harvard University in Boston. I work with the Harvard micro-robotics lab, which is part of the Wyss as well as the Harvard School of Engineering. And this lab is headed by Professor Rob Wood. On the research side, we’re mostly known for our work on insect scale robots, basically, tiny flying robots inspired by bees, crawling robots inspired by cockroaches and centipedes, and other insects people would normally find gross.
Allison Sheridan
Is it creepy walking around your lab, I mean, is there like little bugs flying all over the place?
Michael Karpelson
Fortunately, they are well confined to experimental areas, by you know, by the fact that the technology is still a few years away from being you know, autonomous and out in the world.
Allison Sheridan
Okay, so you have them in cages. Good, good.
Michael Karpelson
Yes. So that said, we’re always looking for ways to kind of take the technologies that we develop in the course of this work, and look for more near term applications. So for example, some of the micro manufacturing we develop could potentially be applied to new types of surgical tools. And over the past couple of years, we’ve been leaning on our expertise in tiny actuators and mechanisms to develop this new type of tactile display for the blind and visually impaired, especially those who as you mentioned, we’re visually or losing vision from age-related causes. And that’s our multisegment tactile display project.
Allison Sheridan
Okay, so I only figured out what you meant by multisegment tactile display when you held it up in front of me on screen. So we’re gonna need to break that down for the people who are listening. What do you mean by tactile multisegment display?
Michael Karpelson
Absolutely, I thought this, this question might come. So a multi segment display is something that’s familiar to virtually any sighted individual, picture the numbers on your microwave oven, or your digital wristwatch, which are all composed of these seven little sticks or segments. And by lighting up the different segments, you can get different numbers and letters to appear. But if you’re listening to this, and you’re not familiar with this type of display, either because you’re blind or due to some other issue, then imagine a number eight that’s kind of boxy and drawn using two squares instead of two circles. And now imagine that each side of those two squares, which there are seven because they share one side, imagine that each side is its own little segment, raise or lower. That will be the simplest kind of multi segment display the seven segment display, which can reproduce numbers and most letters in the English alphabet. And if you go to more segments, such as 14 or 16, then you can do the entire alphabet with pretty high fidelity.
Allison Sheridan
Oh, okay. Okay. Now, when you said raise or lower in the case of clock on a microwave, that’s light, but you’re actually raising and lowering those segments?
Michael Karpelson
Correct. Correct. And our technology such that you can both have the visual indication, so the segment’s would light up, and they would rise and lower out of the plane of the display, so you can pass your finger over them, and feel what the number or letter is.
Allison Sheridan
Okay, so you talked about going from seven to you said 14 or more?
Michael Karpelson
Usually, I think 16 segment is the maximum number of sub segments that have been realized.
Allison Sheridan
Okay, and that’s because and as you as you increase that number, it probably wouldn’t increase the fidelity to your fingers to be able to feel it.
Michael Karpelson
Right, at some point, you kind of you, you run out of ways of representing numbers and letters with just these these in all these line segments. And at some point, you would probably transition to display that’s just composed of dots called a dot matrix display. So 16 segments is kind of where that utility tops out for the multi segment displays.
Allison Sheridan
I see. I see. So now when I picture my microwave oven, saying, you know, blinking 12, you know, because it’s, it’s what they’re for, right? But I picture the the microwave display, the the letters are like maybe an inch tall. You’re going to be doing this in much, much smaller segments, some smaller size, right?
Michael Karpelson
Not that – well, yes, smaller than an inch. Our our current design point, the characters, I believe, about 16 millimeters tall. So that’s, you know, more than half an inch less than an inch. And that is based on some preliminary studies that we’ve done with, you know, people actually touching mock ups of these different versions of these characters and letting us know what was easiest to read what was most easy to perceive.
Allison Sheridan
Okay, so now what kind of devices would you…? Well, actually, let’s stay back with the problem to be solved. So you talk specifically about age-related vision impairments. Why specifically people with those conditions?
Michael Karpelson
Excellent question. So, like many researchers in the space, we initially thought about Braille when we thought about assistive tactile technology. But based on our conversations within the visually impaired community, and also on our own kind of background research, we learned that the majority of visual impairment today is due to age-related causes. And many people in later life could find it difficult to suddenly learn this entirely new form of communication, Braille. The benefit of these multisegment alphanumeric characters is that they are immediately recognizable with little to no training. As I, as I mentioned, you know, our preliminary tests, people have either picked it up right away, or within a couple of minutes of practice. Another benefit that we also touched on, you can combine visual indication with tactile, in other words, to segment both light up and move and rise up. So our target user might be someone who’s beginning to experience age related vision loss, initially, probably relying more on the high contrast visual indication that is supplemented by tactile and then gradually transitioning to a greater reliance on tactile as their vision decreases. And a secondary question there wouldn’t maybe you just haven’t asked it yet, but I figured I’d touch on anyway, is why multisegment? Why not … we also touched on display composed of hundreds of tiny dots that can display any tactile pattern? And the answer there is cost. In any tactile device, the cost is driven by the number of actuators. Basically, each little thing that moves, whether it’s a braille.or, a segment in a display, or a dot in a dot-matrix display, it’s going to need its own separate actuator. And the more actuators you have the more to device ends up costing. And this is why so many braille displays still cost 1000s of dollars. And only now kind of I think after decades of development, we’re seeing the first Braille devices in the $500 price point.
Allison Sheridan
Right, right. So I’m going to show off my mechanical engineering prowess and explain to people what an actuator is. If you’ve ever taken apart a speaker or been interested in how they work, basically, you’ve got a coil of wire that goes with a magnet into the coil. And then as you apply electricity to the current to the to the coil that causes the magnet to move. And that’s what moves a speaker. That’s called an actuator. And that’s what you’re you’re building little teeny tiny ones of these, right.
Michael Karpelson
Something like that. More broadly, an actuator is kind of anything that brings a component into motion. The particular actuation technology, you mentioned is called the voice coil actuator, which yes is you know, big the mainstay of speakers. We do use an electromagnetic actuation mech method that also relies on coils and magnetic fields. It’s a little bit different than you know what you’ve seen a speaker, but many of the principles are the same.
Allison Sheridan
Oooh, I’m gonna need to see some, you know, patents on that or something. I want to I want to learn what that is. I told you, I’d want to
Michael Karpelson
Patents should be out there.
Allison Sheridan
Okay, so um, now, I wanted to dial back to when you were talking about people who’ve just lost their vision, or they’re just starting to learn their lose their vision, you’re saying they would work with high contrast visual elements of the segments moving and lighting up. And but then eventually, as the vision continues to decline, they could move into the tactile space. So the idea of age related is, I still remember what that microwave oven clock looked like. And that’s why it’s easier for me. If if I’m been blind since birth, I would naturally know what an eight looks like, there’s no reason for me to learn that method, because I can learn, you know, I can learn Braille, and that’s going to be more efficient. But if I already know what an eight looks like, and I know what a seven looks like in those segmented displays, it would be faster for me to learn to use your tool, is that right?
Michael Karpelson
Correct. That is exactly it. Basically, you have you know, the the benefit of a partially sighted life, where you are used to more conventional looking numbers and letters. And this is an approximation of that, rather than Braille, which is a whole different type of binary code, which you would need to learn to, you know, map to every letter and number.
Allison Sheridan
I wonder whether it could become the gateway drug, though, to learning to read Braille. So if I start realizing I can recognize letters with my fingers, I might be going okay, so what I want to do, I want to do more with my fingers now that my eyes aren’t aren’t behaving the way I want them to. Maybe I could do something more with my fingers.
Michael Karpelson
It’s certainly something that’s occurred to us. You know, we don’t know if that’s how it’ll play out in practice. But it certainly would be very interesting to see this as a gateway to, to Braille. Because in general, Braille literacy has been dropping. And partly this is a good thing because fewer people are blind from birth due to medical advances and kind of advances in prenatal care, and postnatal care. But yes, it would, it would actually be be a very attractive outcome, I think, if this opened a new world of tactile technologies to the users.
Allison Sheridan
Yeah, I don’t think you could make that happen, but it would be cool if it did.
Michael Karpelson
Exactly.
Allison Sheridan
So now talking about the size of these, what do you what are you picturing these, the segmented tactile displays, what what kind of devices could we hope to see them in?
Michael Karpelson
So this technology can be applied to broadly to any device with an alphanumeric readout, whether it’s an appliance, or wearable device, or medical device. We believe that the most promising applications might be for wearable and portable devices, basically devices that are carried outside the home. The reason for this is we we don’t view this technology as a replacement for something like audio feedback, or replacement for Braille, but rather, as a complimentary technology for certain use cases. In the case of something like a fitness tracker or Activity Monitor that you wear on your wrist, or a portable blood pressure monitor that you take with you, or a glucometer, for example, if you have diabetes, and have vision, loss associated with diabetes, these are devices that you’re going to take outside the home, where audio technology can sometimes introduce privacy concerns. By using audio technology, you’re kind of advertising to everyone around you if you’re visually impaired. And, you know, studies have shown that a lot of people prefer to not do that.
Allison Sheridan
You don’t want to go like, woah, your blood pressure’s high!
Michael Karpelson
Exactly. And, and also, of course, when you’re talking about age related vision loss, it is frequently associated with simultaneous hearing loss, which also complicates you know, the use of audio technology, especially outside the home, potentially, in noisy environments, you know. Imagine having to press that button many times and yelled information at you. And you know, a way to potentially avoid that and have a better user experience.
Allison Sheridan
So now if I picture this on my on my fitness device on my wrist, and I want to read, you know, how far have I walked? That that gets makes me think that you guys have probably studied how small it’s practical to be able to read with your fingers. These multi segmented displays?
Michael Karpelson
Yes, we’ve we’ve done some preliminary studies where we 3D printed many plastic mock ups of these segmented characters and we varied several parameters, the size of the character, obviously, the thickness of each segment, how far protrudes from the plane of the display. And we’ve we’ve had people kind of evaluate and tell us what they found to be easiest to read. And that is how we arrived at you know, our current design point that kind of 16 millimeter tall, eight millimeter wide character and sticks out about half a millimeter from the plane of the device.
Allison Sheridan
Ooh, I gotta write that down. You said eight millimeter wide and how how far do they stick out?
Michael Karpelson
We found that about half a millimeter is generally a nice sweet spot. Okay, it’s interesting we we came into with some assumptions such as larger is better. But that turns out not to be the case. Because frequently for a large character, people with smaller fingers need to move their finger over the character to really get all of it. And there seems to be really a sweet spot kind of around where where I mentioned, where you can quickly feel all of the segments with your finger tip, but also it’s not too small that you’re having trouble telling the different segments apart.
Allison Sheridan
Oh, interesting. So if you if you have to travel around to find it, you could mentally get lost right, you forget where was I before but if you can touch the whole thing?
Michael Karpelson
It’s I don’t think it’s so much about getting lost as just the time it takes for you to pass your finger over the entire thing and of course faster is better when you’re trying to get information out of the device.
Allison Sheridan
That’s really interesting. Sorry, I’m start writing stuff down because I’m really intrigued with this. This is this is pretty cool. So how far along in the development Are you are you years away, minutes away?
Michael Karpelson
Somewhere in between. Basically, for that for the past couple of years, we’ve we’ve been working on this project. We’ve been addressing kind of the key technical challenges of this display, of course that actuate our array which is you know the the driving force behind it, and all of the pieces and components that interface with that. We’ve been doing a lot of also manufacturing validation basically. Today using our fancy you know, lab equipment, we could make prototypes that work but they will be prohibitively expensive if you were to make them and put them into a product. So in some ways, this is not like a standard research project, and we’re trying to kind of bake the cost engineering into the design into the development process. We’re working with contract manufacturers to actually make key pieces of this device using the processes that would eventually be used in a manufacturing scenario, such that when, you know, when somebody, we or partners ready to put this into a product, there’s a clear path to manufacturing and also a clear path to kind of a reasonable cost of low to medium volumes. A lot of the times we hear the, you know, stories such as, Oh, this will be really cheap once we make a billion of them. But the reality is that this is an assistive technology. It’s not going to be an iPhone, and so it has to be practical when manufactured at that these low to medium volumes, otherwise, it becomes you know, an overpriced luxury item that, you know, that doesn’t really make a difference. Now, as far as where we’re now? Around, June of this year, we’re targeting we’re hoping to produce some fully integrated prototypes, what we call study grade prototypes. Basically, ones that are, you know, robust, robust enough to pass around and really get some more do a more rigorous user feedback study that what we did before with the static 3D printed samples, and of course capture, you know, things that cannot be captured with a static model such as you know, how, you know, are people comfortable with how fast the numbers change, either too slow, is it too fast? Of course, the the visual indicator and things like that. So, yes, we are we’ve, rather than kind of drive quickly towards a fully integrated demo, that’s possibly not, you know, realistically manufactured, we’ve been trying to do it all at once and have a realistic path to manufacturing, testing key sub components of this system, such that when we do have that fully integrated prototype, it’s also something reasonable with a path to to commercial manufacturing.
Allison Sheridan
I guess I should assume that people in this field at Harvard are smart. But that sounds really smart to do it that way. I’ve worked on projects where, you know, we made this, for me, what, for me was a very tiny voice coil actuator system, and, yep, completely impractical and make a second one. I mean, we never could, but we spent all this money making it once and it didn’t, it didn’t work too well.
Michael Karpelson
Yeah, it’s, it’s certainly, and when working in research, it’s kind of easy. We usually live in this world of one-off prototypes for a paper or for a big demo. But here where we’re targeting a real world application, we certainly need to consider some of these. And one of the main focuses of the Wyss Institute is is commercial translation, basically, what are the arguably the the primary mission of the Institute is to take kind of that lab grown technology and help get it to the real world faster.
Allison Sheridan
Oh that’s interesting. That’s what the institute is for.
Michael Karpelson
Yep.
Allison Sheridan
Okay. So if people want to learn more about this as questions get involved during offer to be a tester or anything like that, is there a path for that?
Michael Karpelson
Yes, probably the easiest way is to go to the Wyss Institute website. It’s Wyss.harvard.edu. It’s confusing, because it’s pronounced VESE, but it’s a W. And then once you’re there, just search for tactile display. It will take you to a page that has my contact information, the contact information of other people that are involved in the project. Feel free to reach out at any time, we’re happy to answer questions. We’re happy to talk about partnership opportunities. We’re happy to discuss, you know, user feedback studies when we’re closer to that point. And yes, we generally just love to hear from you.
Allison Sheridan
Very cool. Very cool. Well, thank you very much, Michael, for coming on. This is this is a whole different angle to take on this and I’m excited to see people working on it. Like I like to say a lot of people say about disabilities is your if you live long enough, you will have one. So you know everything you’re doing is going to benefit benefit all of us probably at some point, if we’re lucky enough to live that long.
Michael Karpelson
We certainly hope so. And thank you very much for having me. And hopefully we’ll be chatting again in a year when this is in a bunch of assistive products.
Allison Sheridan
That would be awesome. Thanks a lot.