GLEN GORDON: On FSCast 202 we’ll get to know financial educator Chris Peterson and learn about some of the advantages of opening an ABLE account. Then, to celebrate World Emoji Day, I’ll be joined by Garreth Tigwell and Benjamin Gorman, two authors of the paper “Emoji Accessibility for Visually Impaired People.” All upcoming on our podcast for July of 2021.
Hello, everybody. Glen Gordon welcoming you to this month’s edition of the podcast. It’s hard to believe that summer is well underway, soon will be over. And when summer ends, you know what happens; right? We do our first public beta of our new software versions. Namely, JAWS, Fusion, and ZoomText 2022 will be in public beta right around Labor Day. We have lots of new exciting features, some of which I’ll be demonstrating on the podcast in the next couple of months. And I would guess, if I were a betting man, that Eric Damery might make an appearance next month to talk about some of what’s coming up.
It’s very seldom that I’m interviewed on other podcasts, but it just so happens I was on two of them within a week of one another, and both were related to Ireland, at least produced by people who live in Ireland. One of them was Blind Guys Chat. That’s Oran O’Neill, Stuart Lawler, and Jan Bloem. It was kind of fun talking with those guys. They have a great sense of humor and chat about all sorts of things with a variety of people. So if you’re looking for podcasts to check out, that would be a good one, Blind Guys Chat.
And then I was also on the NCBI podcast, which is the podcast produced by the National Council for the Blind in Ireland. That’s a little more Irish-specific. But if you live in that area, that might be a good podcast for you to check out, as well.
As always, it’s great to hear from you. Feel free to write to me at fscast@vispero.com, fscast@vispero.com. I hear from more and more folks as the months go by, and it’s great to hear your successes with our software. And even though it’s sometimes hard to hear, it’s good to hear if you’re having problems, and you’re frustrated you’ve not been able to get the support you need. I can’t necessarily provide it directly, but I’ll do my best to try to get you to the right people, if going through official support channels seems to have stalled for you. That’s fscast@vispero.com.
GERRY CHEVALIER: Hello. This is Gerry Chevalier with a JAWS Tip that many of
you are likely aware of, but maybe don’t use as often as you might. And that
is the FSReader app that is installed on your desktop with every installation
of JAWS. FSReader has three useful functions. It allows you to read the DAISY
formatted training books from Freedom Scientific. Secondly, it acts as a
portal to the online JAWS training resources. And thirdly, it allows you to
read DAISY books from other providers such as Bookshare or DAISY libraries in
other countries. I’m currently on my desktop, and I’m going to use first-letter
navigation F to find the FSReader icon.
JAWS VOICE: FSReader 3.0, three of 52.
GERRY: There it is. I’ll tap ENTER to open it.
JAWS VOICE: ENTER. FSReader – Untitled.
GERRY: FSReader is opened. It says “untitled” because we haven’t got a book open at the moment. I’m going to go now to the file menu, ALT+F.
JAWS VOICE: ALT+F. Menu bar. Menu. File menu. Open... CTRL+O.
GERRY: And here’s the standard CTRL Open item on the file menu. We’ll use that to open other DAISY books. I’m going to DOWN ARROW.
JAWS VOICE: Open JAWS Training Table of Contents, CTRL+J.
GERRY: The Training Table of Contents is the heart of FSReader. It has a dedicated shortcut, CTRL+J. I’ll tap ENTER.
JAWS VOICE: ENTER. Leaving menus. JAWS Training Contents.
GERRY: Press H to jump through the headings.
JAWS VOICE: JAWS Training Contents heading level one. JAWS Basic Training DAISY Books, heading level two. Book one, Introduction and Overview of the JAWS Basic Training. Book two, Getting Started with JAWS, heading level three. Book three, Working With Menus, Dialog Boxes. Book four, the JAWS User Interface and Utilities, heading level. Book five, Introduction to Microsoft Windows, heading level three.
GERRY: Let’s press ENTER on this DAISY training book.
JAWS VOICE: ENTER. Book five, Introduction to Microsoft Windows, heading level three.
GERRY: I’m going to press CTRL+P, the pause and resume hotkey.
JAWS VOICE: Book five, Introduction to Microsoft Windows. In this book you learn about the Windows desktop.
GERRY: And I tapped CTRL+P again to stop playback. The interesting thing about CTRL+P is you don’t have to be in the FSReader app window to use it. For example, if you’re in the various windows following the instructions that are playing in the background training book, you can press CTRL+P to pause and resume playback without having to ALT+TAB back into the FSReader app. That’s a very cool feature. I should also point out that these DAISY training books need to be resident on your local computer. However, that’s not an issue because, if you select a training book from the Table of Contents, and it’s not installed, the FSReader will offer to download it for you. I’m going to back to the file menu now, ALT+F.
JAWS VOICE: ALT+F. Menu bar. Menu. File menu. Open... CTRL+O.
GERRY: And I’m going to press ENTER to open a DAISY book from another producer.
JAWS VOICE: ENTER. Leaving menus. Open dialog. Filename: edit combo.
GERRY: SHIFT+TAB to my Downloads folder, where I have downloaded a couple of DAISY books.
JAWS VOICE: Name, ENTER, bar. Folder view, list view. “Harry Potter and the Chamber of Secrets” 2113506.
GERRY: That folder contains a DAISY book that I downloaded from Bookshare. I’m going to press ENTER to open it.
JAWS VOICE: ENTER. Folder view, list view. Not selected Harry_Potter_and_the_Chamber...
GERRY: It conveniently positions at the OPF, which is like the root of the DAISY book. I will select it and then tap ENTER to open it.
JAWS VOICE: Selected. ENTER. “Harry Potter and the Chamber of Secrets,” 25 headings and two links.
GERRY: Tap H to move through the headings.
JAWS VOICE: Chapter 2, Dobby’s Warning, heading level two. Chapter 3, The Burrow, heading level two.
GERRY: And I’ll do a Say All.
JAWS VOICE: Heading level two, Chapter 3, The Burrow. “Ron!” breathed Harry, creeping to the window and pushing it up so they could talk through the bars.
GERRY: So there you have it. FSReader, the desktop app, waiting for you to explore. You can use it to access the online training resources from Freedom Scientific, or read their DAISY formatted training books, or read other non-protected DAISY books from other producers.
GLEN: Well, thank you, Gerry. And you taught me something I didn’t know. I didn’t realize CTRL+P for starting and stopping audiobooks works regardless of whether or not FSReader is in focus. So a very good tip.
Additionally, my favorite FSReader feature is the Table of Contents, probably because of the nature of the stuff I tend to read, which is programming and other reference books where the contents means everything. So when a book is open, and you press F6, you’ll go to a tree view that has the Table of Contents displayed. You can find the section you care about. And then when you press F6 a second time, you’ll go back into the main document in sync to where you just moved in the Table of Contents. So it’s a very quick way to move through elements of a DAISY book.
For sending in his Power Tip, Gerry gets a year added onto his JAWS license. If you’d like to do the same, write to us with your Power Tip, fscast@vispero.com.
GLEN: Chris Peterson is a long-time JAWS user, software developer by day and
financial educator by night, weekends, and whenever else he can spare the
time. Back in 2020 he founded Penny Forward as a vehicle to help blind people
learn more about finances and help gain financial independence. Chris, welcome
to the podcast.
CHRIS PETERSON: Glen, thanks for having me.
GLEN: I’m interested a little bit in your origin story. You’re a software developer. How did that all come about?
CHRIS: Well, when I was a young person, I wanted to be a radio broadcaster, like many of us do. And I found out two things about that. I was actually enrolled in broadcasting school. I didn’t realize quite at the time that broadcasting didn’t pay very well, and it was going to require me to move more than I probably wanted to. And so I got cold feet, and I quit before I started my first semester. And I took a couple of years off and decided at some point that I would like to be programmer, and ended up following a girlfriend to college and enrolling in a computer science program there and doing very well at it. I graduated with honors.
Then I happened to land a job with IBM in Rochester, Minnesota. And part of that was my own tenacity, and part of that again was this girlfriend that I was dating who also happened to land a job there. And Rochester, Minnesota only has a couple of employers. IBM is one of them. But it was a good place to start. And I worked there for 14 years, until our division was sold to a company called Lenovo. And I worked for Lenovo for five years. And I frankly was starting to feel burned out on it. I had been putting in a lot of long, long hours. I had worked my way up to becoming a team leader, which I was really proud of. I had a person on my team who had retinitis pigmentosa and was gradually losing his vision. And that was something that we really bonded over, and I felt extra good that I was able to help him through that very gradual process for him, just sort of by happenstance.
And then Lenovo laid off my entire team in the United States except me. And I started to wonder if I was kind of stuck. And so I thought about what I might do next, and what I could do next was look for more programming jobs. Or I thought I could do something entirely different. And over the course of a few years, my financial advisor and I had discussed the idea of me becoming a financial advisor because I had taken an uncommon interest in some of the stuff that he was talking with me about. And he thought I would do a good job at it. So I went to put in an application with Thrivent Financial, which is where my financial advisor worked.
And then my wife came to me, and she said, “We are going to have another baby.” And one of the things about working as a financial advisor is that you work on 100% commission. So you really don’t make a lot of money until you start to bring on a fairly decent client base. And I had trouble figuring out how I could learn the reins of a new job and earn 100% commission and have a brand new baby in the house, all at the same time.
And I got an email out of the blue through LinkedIn from a recruiter saying, “Our client, Thrivent Financial, is looking for a lead software developer in Minneapolis. Are you interested?” And I said, “Yes, I am interested.” And I got on the phone with him and told him this entire story. And he said, “Well, this is great because most of the people I talk to have never heard of Thrivent Financial before.” So he got me an interview with the hiring manager that turned into an in-person interview with her and her team. When I walked out of that in-person interview with her and her team, she made it pretty clear that she was planning to offer me a job, and that it would just take a little bit of time.
So I waited and waited and waited. And I also talked to the other people I had been interviewing with at Thrivent and said, you know, “I’m going to put this on hold for now because I’ve got this potential other opportunity that’s a better fit for me right now. But I’m still interested.” And they said, “That’s okay. Maybe we can figure out how to use you as a software developer and still get you an education in finance in case you want to change roles in the future.” And that was a pretty neat offer. So I took them up on that.
So over the next couple of months I got hired. I was a contractor for three months, then was hired as a full-time employee and started to really work on studying personal finance as an area. And in 2020 I launched Penny Forward, and it was a blog.
GLEN: So you’ve got to tell me, did Penny Forward come because the web domain was available?
CHRIS: No, it did not. It actually came because I thought long and hard about how to come up with a name that would mean something to blind people and also be reflective of what I’m trying to promote. And what I’m trying to promote is blind people building bright futures, one penny at a time. And those pennies may come from ourselves, from jobs. But like so many things in life, they may also come from our families, friends, other people that find reasons to support us because they like us, because we have talents, because we’ve given something back to them in some way.
And I’m also a guide dog user. And I thought about, well, what about Penny Forward, like a command to your guide dog. And you’ll notice that I use a lot of analogies to orientation and mobility in some of the things that I write and some of the things that I say with regard to money because money is complicated for a lot of people. There’s a lot of unknowns, and that can make people afraid, just like entering an unfamiliar place can have a lot of unknowns and can create a lot of anxiety for people. And sometimes it’s nice to have somebody that can help you a little bit to figure out where to go. And a guide dog does that for some of us, for orientation and mobility. And a financial educator can help us to figure out where to go to make our finances work harder for us in the financial world. The domain name also happened to be available.
GLEN: I’ve listened to several of your podcast episodes now, and you’re very soft sell and very nonjudgmental, which are two great things if you’re an educator and someone is going to come talk to you about perhaps the things they did wrong.
CHRIS: It is very important to be nonjudgmental because everybody’s got a different path in life. And what you or I may consider to be an unsatisfying path may be perfectly satisfying and may feel like success to somebody else.
GLEN: Do you think blind people have unique financial challenges, other than perhaps not having enough money? Or do you think it’s just the nature of people in the world generally aren’t very educated on these things?
CHRIS: It’s a combination of both. There is an epidemic of lack of financial knowledge in this country particularly. I don’t really know about the rest of the world. But blind people also have some unique challenges. Social Security is one of them because a lot of us receive it at some point in our lives. And sometimes that can be a really good thing, and sometimes it can create a lot of extra stress and headache, even after we’re working. I just received a letter from Social Security this past year, and I haven’t been receiving SSI or SSDI for 25 years. And they told me they overpaid me by $600, and I needed to write them a check to pay it back. It’s weird that I have paid into the Social Security system because of my wages for the last 25 years, and yet they’re still looking at what they paid me in SSI and SSDI, which wasn’t a lot because I was only getting it for about four years.
GLEN: Did you resolve it?
CHRIS: I have not resolved it yet. I’m still trying to figure it out. And that’s part of the problem with the system.
GLEN: It’s hard to get to the right person, to the person who actually understands the issue and can solve it for you.
CHRIS: It really is. And oftentimes, if you ask questions, it can make it worse because then someone looks harder. And they might decide – this happened to me. They may decide that they underpaid you by $8,000. And one day $8,000 just lands in your checking account for no reason. And then they might decide later that they made a mistake and yank it back. And of course when you’re on SSI you’re not allowed to have $8,000 in your checking account. You can only have $2,000 at a time in your checking account. So you have to spend that money right away, if they put it in there. You don’t have a choice, or you’re going to lose your monthly benefit check.
GLEN: The fact that you said you can’t have more than $2,000 in your bank account if you’re on SSI is probably a good segue into something that I had no idea existed. They’re called ABLE accounts. And you’ve been one of the guys who’s trying to publicize those.
CHRIS: Yes. ABLE accounts are a fairly new thing. They’ve only been around since 2015. They’re a tremendous opportunity for people who are blind because they allow you to sock away money in an investment account. If you’re receiving SSI, that money does not contribute towards that $2,000 limit that I just talked about. But even if you’re not receiving SSI, the growth of that account is tax free, if you take it out and spend it on qualified disability expenses. And there’s a broad range of qualified disability expenses that are categorized roughly under housing, education, transportation, employment expenses, financial management expenses, legal expenses. I could go on and on. There’s a lot of them. So it would be pretty hard to spend the money in your ABLE account in a non-qualified way. And this is something that only people with a qualifying disability, which blindness is one of them, are able to do.
GLEN: It almost feels like a Roth IRA, where you put money in post-tax, but yet it grows tax-free.
CHRIS: It’s a wonderful comparison because in that respect the Roth IRA is very similar to an ABLE account. There are a few differences. The qualified disability expenses rules is one of them. Whereas a Roth IRA you can’t take money out of the account until you’re 59.5. A Roth IA only allows you to contribute six or $7,000, depending on your age, per year. An ABLE account allows you to contribute up to $15,000 per year. And even more than that, if you’re working.
GLEN: And you can take the money out before you’re 59.5; right?
CHRIS: That’s correct. You could take it out next week, if you needed to use it for a qualified disability expense like paying your mortgage or rent.
GLEN: Let’s say you want to buy a family car that your child or your wife can drive. Does that qualify?
CHRIS: It does, if they are going to drive you around in it.
GLEN: At least part of the time.
CHRIS: Right. It doesn’t have to be all the time. Just part of the time. It has to benefit you as a member of the family.
GLEN: Educational expenses.
CHRIS: If you want to pay for yourself to go to college or to go to a trade school or to do continuing education to keep your credentials up to date if you’re maintaining a certification or something that is required for your job, those are all things that would qualify.
GLEN: And I even read somewhere that food can quality, assuming it’s food for yourself.
CHRIS: Yeah. Food for yourself can quality because the whole idea of an ABLE account is to allow people with disabilities to support themselves more securely, and so basic living expenses such as food and housing and clothing are all qualified, if they benefit you.
GLEN: Do you set this up and essentially say this is an ABLE account? Or are they a thing in various states that you have to apply to the appropriate parties?
CHRIS: So most states have an ABLE account program. ABLE accounts are very similar to college savings plans in the way that they operate. So the program may exist in a state, but the state may contract with a company that actually administers the plan. And in most of those cases you can fill out an application online to get the account set up. It’ll ask you a few questions about yourself, if you have a disability. They don’t even care what your disability is most of the time. You just need to keep documentation around for yourself in case the IRS comes looking for it. And once the account is set up, you can put money into it manually by sending them a check. You can set up direct deposits that transfer out of your bank account into your ABLE account. And many ABLE accounts also come with a built-in checking account as an option. So you could even have your paycheck from your employer or from the Social Security Administration go directly into your ABLE account, if that made the most sense for you.
GLEN: The one detail that I was surprised by, actually, is you need to be blind relatively early to qualify. Do you have any idea why?
CHRIS: The reason is that the law was written that way because people who are older may have found other ways to save. In other words, Congress would prefer that you used savings from your employment income, and that you put that away in a Roth IRA, some kind of an emergency fund or whatever it may be, to support yourself when you become blind later in life. There’s actually a group of lobbyists that are trying to change that now. There’s a bill that’s working its way through Congress, but I have no idea when it’ll pass, that would increase that age to 43.
GLEN: And the current age is like 26 or something?
CHRIS: Yes, it’s 26. And incidentally, that means that the reason for your blindness had to occur before age 26. In other words, if you had glaucoma, but it didn’t affect your vision until you were 50, but you were diagnosed with it at age 24, the thinking is that you would still qualify.
GLEN: Interesting. You created Penny Forward, the podcast and the website. It feels like it’s the start of an empire. Not literally an empire, but do you have bigger plans than what’s going on currently?
CHRIS: I do have bigger plans, and there’s other things going on besides the podcast and the website and the Facebook presence. I intend to turn Penny Forward into a nonprofit organization. And its mission is to provide blind people with the knowledge and resources to navigate a complicated financial landscape and target their goals. And we envision a world where people are confident and are able to weather hard times, take advantage of rare opportunities, and more powerfully support causes that we deeply care about. We’re doing that through the podcast and the website. We’re also doing that by collaborating with other organizations such as World Services for the Blind, several affiliates of the American Council of the Blind.
I most recently participated in a panel about ABLE accounts at the NFB’s national convention, and a panel about podcast content creation at the ACB national convention. And along with ACB Next Generation, one of the ACB affiliates, we’re offering monthly financial education classes through Zoom community calls that people can take advantage of. And some of those are streamed on ACB Radio, as well. We’ve so far covered taking your financial temperature, which is figuring out where you’re spending your money right now, figuring out where your credit score is and calculating your net worth. We have covered building credit, either from the ground up or reviving your credit, if for some reason you’ve made mistakes, and your credit score is very low. We are about to cover investing and building up your net worth, which is really the way to achieve financial security because that acts like a battery that you can draw from when your income might be low or nonexistent.
GLEN: It does seem like a little bit of a Catch-22. To gain financial independence, you need enough of an income to stash enough away. Which means it feels like in many ways it all comes back to finding a job.
CHRIS: Absolutely. Finding a job is really important. And I am going to be participating in August in a webinar with World Services for the Blind focused on employment barriers and how people overcome them. And it’s something that I hope to do more on in the future because I feel like there are a lot of blind people who have talents and the right education to be working, but just haven’t found the right employer. We need to figure out how to match those up.
GLEN: It’s scary now because so much of the interviewing process is delayed until you pass the automated screening. And I think it’s harder and harder to pass automated screening unless you really know how to work the system.
CHRIS: Yeah, it’s unfortunately true, and there are whole companies that have been built up around that. And there’s a lot of good and some bad advice on the Internet. But there’s also other ways that you can find jobs. And one of the most effective ways is by using your own personal network, you know, figuring out who you know that might know enough about you and feel good about you to recommend you for a position that you might be able to do. And sometimes that might not be exactly what your dream job was, but you’ve got to start somewhere. And by putting yourself in the position to be present in the areas where people that are working in your dream field are, and getting to know them, and showing them that you have talents, and that you have motivation and a work ethic, and are capable, there are going to be people that are going to be asking you if you would like to interview for their jobs. You don’t even necessarily need to apply.
GLEN: Yeah, I mean, this is a tangent, but the way that I learned about you for the podcast is essentially networking that you did. You came to our Freedom Scientific Happy Hour at the NFB convention, started talking about ABLE accounts and that you were doing this podcast. And one thing led to the next, and we invited you to be on. Now, was that planned? Or did that just happen organically?
CHRIS: It kind of happened organically. I joined the Happy Hour because earlier that day I had been on this other NFB panel. And I was kind of curious to know if anybody had attended it and what they thought of it. So I asked the question. And nobody had attended the panel, but they had lots of questions about ABLE accounts, and Matt was very kind and kind of let me monopolize the room for 20 or 30 minutes, answering their questions, if I could. And from there he asked me to connect up with him, and then connected me with you.
GLEN: But it is a comment on how much networking really gets stuff done.
CHRIS: It really is. And it’s also a comment on how to do networking right. In other words, it’s not necessarily going to be effective to just jump into, say, the Happy Hour as an example, and say, “Hey, I’m the Penny Forward guy. I’ve got a podcast. Everybody pay attention to me.” But it is effective...
GLEN: Go ahead, sorry.
CHRIS: But it is effective to jump into those rooms and ask for help; or, even better yet, provide help if somebody asks for it. And get to know people and develop a rapport with them. And if they ask you what you do, or ask you to tell them more about yourself, then you can bring up Penny Forward, or you can bring up that you’re looking for a job in psychotherapy or whatever it might be. And if you’re around the right people, their ears might just perk up, and they might say, “Hey, I’m interested in that. I want to know more about that, and I want to know it from you because you seem like a good person that I already like.”
GLEN: This is really great. I’m surprised it’s taken so long for this to happen.
CHRIS: I appreciate you saying so. I did feel, when I started this, like this had to be done. Some people when they go into the ministry will say that they have a calling. I do kind of feel like I have a calling to make this happen and put a lot of time and effort into it. And I don’t feel like I can go back or stop at this point.
GLEN: You definitely should keep doing the podcast, too, because it’s really well done. I’ve listened to several episodes now. They’re well-produced, good questions. You get out of the way and let your guests talk, and so we really get to learn a lot about them. And it’s always in 20 or 30 minutes, on average. So I commend you for doing the podcast, and I commend you for having such a good producer/editor in the form of Byron Lee, who’s really good.
CHRIS: Well, thank you for saying so. Byron is a very talented audio editor. And he was able to start his small business, SuperBlink.org, by bringing me on as one of his customers. And he’s done a wonderful job, and I hope that people look him up, if they’re looking for something similar.
GLEN: Excellent. Well, Chris, thanks for being with me on the podcast.
CHRIS: Glen, thanks for having me. It’s been really fun.
GLEN: The website is PennyForward.com. And I assume people can write to you at pennyforward@pennyforward.com.
CHRIS: Absolutely.
GLEN: If Chris got you thinking about an ABLE account, you can find more information about those in a series of articles on his website.
GLEN: If you’re a little unclear about the meaning of the word “emoji,” the
way I interpret it is some special characters that to those of us who use JAWS
or other screen readers get expressed as two- or three-word descriptions. But
to sighted people, these single characters actually represent little pictures
that you can easily insert into things – email, tweets, et cetera.
I started thinking about emojis recently when I was talking to a colleague who began to use them, and he inserted what he thought would be a really innocuous emoji, which was a pensive face, sort of a thoughtful considering face. And he got a note immediately back from someone saying, are you okay? Come to find out that although the short description says “pensive face,” the longer description, and the way it’s actually used, which we found on Emojipedia.com – a very useful reference, by the way – is “a pensive, remorseful face, saddened by life, quietly considering where things all went wrong.” So clearly the longer description is much clearer than “pensive face.” It got me to thinking about all the issues that those of us who can’t see the images might have when trying to use emojis properly.
So I started looking, and I found a paper written by a couple of HCI researchers. HCI stands for Human-Computer Interaction. And although HCI is far broader than accessibility, these are folks who got their PhDs at the University of Dundee in Scotland. And that HCI program really stresses accessibility. So a couple of sighted guys who care about this stuff, which is I think to our advantage when we’re discussing something as highly visual as emojis.
I speak more specifically of Garreth Tigwell from the Rochester Institute of Technology; Benjamin Gorman from Bournemouth University in the U.K. They are two of the three authors of the paper “Emoji Accessibility for Visually Impaired People.” Their co-author, Rachel Menzies from the University of Dundee in Scotland, wasn’t able to join us. But we have two out of three. Thank you both for being here.
BENJAMIN GORMAN: Yeah, thanks for having us, yeah. Good to be here.
GARRETH TIGWELL: Yeah, thank you.
GLEN: So the epiphany that I had after reading your paper is that emojis got introduced into the language to allow people to be more emotionally expressive. But for those of us who are blind, they get turned into three- or four-word descriptions, which kind of defeats the purpose.
BEN: Yeah, exactly. I think and that was probably what kind of spurred us on to kind of probably do this work because we thought, how does that translate for people who are using screen readers or any other kind of assistive technology? And that was kind of our starting point, I think, Garreth; right?
GARRETH: Yeah, definitely. And I think especially because it’s a visual medium. And so to you and I, Ben, we look at emoji, and we look at the images and stuff. And so, yeah, it comes a natural question because we do accessibility research that’s like, well, hold on a second, how does this translate for people who are blind or are using screen readers? And what’s actually going on behind the scenes there?
GLEN: When you went in, beginning to do this paper, did you have some conclusions that you were thinking you were working towards? I realize that that’s not how research is done, but we all have preconceived notions.
GARRETH: So I have a paper that’s years older than this one that looked at emoji misunderstandings. And my intent was to continue that research forward into emoji accessibility. But my focus of my PhD research changed direction. And then, so coming back to this paper a few years later, it was kind of surprising to see that no academic publication had come out in all of that time that had focused on the angle of accessibility. So we really had a lot of freedom to explore this.
Now, what we did find was a lot of anecdotal evidence, blog posts and things like that, some tweets, as well, which we reference in the paper, that kind of indicated that, yeah, there’s something worth exploring here. But there was so little more systematic academic kind of styled work that we didn’t really want to constrain ourselves too much, which is why we’ve kind of gone this very qualitative, exploratory approach to this research.
GLEN: Does your research show anything about how do people know what the proper use of emojis are? In other words, are you likely to encounter them a lot before you’d use a particular emoji?
BEN: I don’t know if anybody really knows how to use emojis well, I would say.
GARRETH: Yeah.
BEN: I would think, and this kind of goes with other research, once you’ve kind of offered a close kind of group of friends – so for instance, Garreth and I have a group chat with one of our other good friends. And we use emoji quite a lot. And we kind of understand when Garreth uses an emoji, I kind of understand what he means by that emoji because there’s kind of that knowing, that knowledge of bond between us as a friend group. And I think if you start chatting with somebody else, you probably are less likely to start using emoji with them until you become more familiar with them. And then you can have learned their kind of intricacy. So I think it’s an interesting question. But I think it’s different depending on who you’re chatting to, really.
GARRETH: Especially during the pandemic, though, we’re starting to use things like Slack. So people are – I think people are starting to get a bit more comfortable with introducing it in that space. And something that we also found that kind of relates to the problem that we discuss in the paper, as well, is companies have started to use them to kind of appeal to people. So you might find there’s emoji in the subject line of an email and things like that, where in the past that probably might have seemed a little unprofessional. But there’s slowly this shift happening where it is starting to get a little bit mixed in with work context.
BEN: Yes, I think definitely for brands; right? So, you know, I think one of the interesting examples we found where we were kind of just – I guess for a while Garreth and I kind of held like a folder of screenshots where anytime we found an interesting tweet that was using emoji where they probably shouldn’t have, we kind of screen shot and sent to each other. There was a lot of ones from a particular rail operator, so like a train operator in the U.K. And when they were posting, like, updates about trains that were going to be late, or delays and road works and all that kind of stuff, they kind of littered this really important tweet with kind of emoji. And it could potentially cause issues. And I think that was another reason why we were, oh, we should really look into this before this becomes a wider issue.
GLEN: And screen readers like JAWS take those emojis and use the description from the Unicode Consortium, which gets put in line, which can make the message potentially confusing.
GARRETH: Yeah. It definitely disrupted that flow of communication. And you mentioned two- or three-word descriptions. Some emoji now have five or could be even more because new emoji get added, and they’re also allowing people to change skin color and mixing gender within like the Greek people type emoji. Which is great, but then it also means the description behind that becomes much more complex and much longer. And I think in certain contexts that’s fine, to capture that. But then the real issue comes with how people start using those emoji and then how that translates to a screen reader. And then that’s where kind of things break down a little bit.
BEN: The other interesting thing that a lot of our participants spoke about was that the descriptor doesn’t actually help somebody with a visual impairment to actually understand what is that, that thing, because it maps to gestures, perhaps, facial expressions that they may not have any kind of visual mapping of; right? Especially if they’ve been blind from birth, I think. And I think that came up a lot in the responses, that the descriptors don’t actually help anyway.
GARRETH: Yeah, we had a participant that talked about emotions, or that emoji are supposed to be like an emotional representation. And I think it was – they would end at the word “happy,” and then it suggests a whole bunch of different emoji to represent happy. But for them, because they were born without any sight, it’s just kind of meaningless. And then they don’t know which is the most appropriate of these different happier faces for that situation that they’re typing.
BEN: The interesting thing that still exists is that on different platforms there’s these subtle differences which, on the face of it, might not seem that big of a deal. But they can be quite stark, still. And I think an interesting one we found just the other day was there’s an emoji with like a moon face on it. And the moon face on Android kind of looks kind of cheeky, kind of looks like he’s smirking. And when you see that on iOS, it’s a completely different – it’s a completely different expression. And that still causes confusion, I think, as well.
GLEN: We’ve talked thus far about totally blind folks. The other group of people that listen to this podcast and who our products serve are those with low vision. That’s a whole different kettle of fish; isn’t it.
BEN: Yeah, and I think that was kind of one of our other big things, as well, because if you actually are not relying on the kind of descriptors that you talked about, those three little words, then actually the kind of subtle differences between just the visual representations, because they are quite small on the screen, they’re probably only, I don’t know, maybe 30x30 pixels unless you’ve got some kind of magnification on. And I think the differences between one emoji to another can sometimes be very, very subtle; very, very small.
GARRETH: That’s why in our paper for this survey we kind of focused more broadly on anyone that has a vision impairment. So for even people that are color blind, for example, before people had the option for kind of changing the skin color of emoji, usually yellow. And then so, you know, you might have a face, and then they’re blushing, so then the cheeks are supposed to be redder. If someone has red/green color blindness, they can’t often distinguish easily between colors that go from green through yellow to red. So that’s why the kind of first half of our paper we kind of focused more generally and just anyone that has a vision impairment, tell us your experience with emoji. And then for the latter part of the paper, we focused more specifically on screen reader users because we found the challenges there were – there were more. And there was more kind of technological stuff, as well, that we could kind of explore.
GLEN: I think screen reader users have a distinct advantage when it comes to selecting emojis, as compared to someone who’s using magnification and trying to look at a large group of emojis, you know, very close to one another.
GARRETH: Yeah, we actually had some interesting insights about searching from our participants. And some people, you know, once they’ve kind of used a few emoji, depending on your device and how it’s set up and I guess the keyboard that you use, you can have like a frequently used panel. And people seem to rely on that a lot because emoji are continually being added. And, you know, at the last count it was over 3,000. And I don’t know where we’re at now. I think even trying to just search through that, participants who are screen reader users that said that they found searching could be very tricky, as well. So they kind of use these strategies to kind of minimize having to do that.
GLEN: I realize this was not covered in your paper, but do you have any idea how many emojis people tend to use?
GARRETH: I probably only use about 10. Unless there’s a very specific context and, you know, you need to go and search for like a particular animal emoji or a flag or something, there’s not many that I would often use.
BEN: But I think it depends on the context, as well; right? So I think when I’m chatting to Garreth, I use different emojis than I’d chat with my friends, other friends, and I’d chat with my parents. Or, you know, if I might happen to be on a dating app or something, I’m going to be using a different set of emoji; right? And so I think collectively, especially like what context, as well. I think across all of those contexts, I’d probably use quite a few, I would say. It’s not the whole gamut of, yeah, 3,000 emoji, but it’s probably maybe up to 30, I would say, I regularly reach for.
GLEN: I was surprised just how much emoji sometimes take on special meanings. I think there was a reference in your paper that talked about someone using the pizza emoji to represent love.
GARRETH: I can’t remember the specific details of that paper. But I think what they were getting at with that was that there was a particular participant that said between them and their partner, that was kind of what was used. And this kind of goes back to what Ben was saying about kind of group dynamics and I guess sort of cultures and things like that. Sometimes you get like an in-joke among friends. And I found this with research, the other study that I did, as well. Sometimes something just happens, or – yeah. You might then use an emoji for a particular joke. So it’s not necessarily that the pizza universally means love. But, like, for some people, they might kind of repurpose it in that way. Which is kind of the power of emoji.
And that also then just makes the whole situation of understanding and teaching people about emoji very complex because no matter how hard you try, there’s always – there’s going to be situations or contexts that you just don’t know about. So, like, if you add a new friend to a group chat, and they use an emoji, maybe using it for its intended purpose, not knowing that there’s an in-joke among those group of friends about that emoji. So, yeah, it’s a very interesting thing to kind of deconstruct and think about.
GLEN: So you take the...
BEN: But even...
GLEN: Go ahead, Ben, and then I’ll pick it up.
BEN: So sorry, I was going to give an example of another emoji which is commonly misused. So there’s an emoji which has got two hands kind of clasped together. Now, it’s not obvious that the hands are the same person’s hands, but it’s actually meant to be a praying emoji. But it also kind of looks like a high-five. So a lot of people use the praying emoji for a high-five. But it’s not, you know, displayed in different ways to people who use a screen reader. It’s going to be using the same kind of emoji classifier depending on where it’s used.
GLEN: What are your general recommendations that you discussed in the paper?
BEN: The first thing was that, if you are using emoji, just think about the number of emoji. So as we said when a screen reader reads out, you know, “crying tears of joy” as an emoji, and you’ve used it maybe three times in a row, that gets read out three times, all of the words. And that’s a pretty common thing to have kind of repetitive emoji online because the more “crying with laughing face” or “crying tears of joy,” the kind of more you’re laughing, essentially. But that’s problematic because, yeah, that gets read out to the person with the screen reader.
So I think, yeah, think about how many emoji you need to use was the first kind of guideline. The second one was think about the placement of emoji in your message. So it’s probably better to have your text first, and then the emoji at the end, just so it’s not getting read in place of words, or it might get misconstrued as being part of the sentence, rather than being an emoji within the sentence. And I think that the kind of final thing was think about your reach. So if you are, as I said, like an actual train operator, perhaps you need to think about you’re not just – your reach is a lot larger. So if I’m sending an emoji to one person, I probably don’t need to worry about my usage as much, as long as I know that that person can understand that message quite clearly.
If I’m a brand, or like a national government, for instance, so even during the pandemic the National Health Service was using emoji in kind of coronavirus tweets, you know, about restrictions and when these restrictions would start and that kind of thing. And that’s like a public health message. And putting emoji in that tweet, that’s probably not a good idea; right? Because that can lead to a lot more challenges perhaps, especially on a public health message. So it’s think about your reach, and think about the impact of your one tweet and who might find it inaccessible, essentially.
GLEN: Have you given any thought to what we as screen reader vendors should do to improve things?
BEN: We did talk briefly, and I can’t remember if we talked about this in the paper or not. It could have just been when we were kind of talking about future research ideas. Giving the user some control over when this emoji appears, I want this particular descriptor read out. I think that might be really useful, to kind of give a little more agency into what the screen reader user hears, so they can kind of customize a little bit of that kind of presentation, essentially, of emoji. I think that would make a lot of sense. And I’m not sure how that would be easy to facilitate, but I think it could be useful.
GLEN: We actually have it. It’s called the Dictionary Manager, and you can pick a character, and that gets priority over our built-in symbol descriptions. So you can pick an emoji of your choice and give it a name, and that’s what’ll get read out.
GARRETH: I was actually going to just add to that and say we did have a participant that mentioned that they could also – they allowed their screen reader to skip over emoji. And I think also maybe just reading it once, if there was a string. But we also found others that suggested that as design solutions. So I think what that says to me is that the options might be there within the screen readers, but then maybe the controls aren’t clear for the user. And there might need to be some redevelopment there to kind of say, hey, by the way, we have this feature, if you want to use it. That might be part of the issue that’s going on is that some users just aren’t aware that they have that to use.
GLEN: I am absolutely sure you’re right. Screen readers, you know, end up having lots of features, and people know the things that they use. That’s the case with all of us.
GARRETH: Just to add one other quick thing to your question, I would maybe say that all of the screen reader kind of companies, software creators, they should try and create their own little consortium of sorts, a bit like the Unicode. But even if it’s just to kind of maybe identify where the differences are within how JAWS might describe a particular emoji versus voiceover, and try and aim for some consistency there, might kind of help reduce certain misunderstandings, as well.
GLEN: Yeah, I think it does make good sense. I was just coming back to your idea of someone turning off speaking of emojis. That just seems like you could completely miss intent and not know you were missing intent because you didn’t want to hear the emojis.
GARRETH: Yeah. And actually the other problem is that people these days have a habit of replacing words with emojis, which is a whole other issue. We have an example in the paper, actually, where someone replaces the word “sunny” with the sun emoji. Now, the main challenge there is that the sun emoji is just sun, and not sunny. So when it reads the sentence with a screen reader, it’s not really grammatically correct, and it’s a little bit confusing. But then I guess if you turned it off, or you had it so it skips the emoji, then you don’t even get any context that they’re talking about the sun. So, yeah, it’s not the way to kind of solve this, for sure.
GLEN: We’re trying to figure out what to do with emojis in braille. At the moment we’re not showing the long description because that typically has been a speech function. But that’s suboptimal. But so is potentially putting in long strings of words between text on a scarce resource, which is a braille display that only shows you 40 or 20 characters at a time.
GARRETH: Yeah, we had a participant that mentioned that they use braille, and they said exactly what you’re saying. You know, when you have emoji that have four- or five-words descriptions, it just, yeah, it’s not a good experience.
BEN: And jumping off that, I think the other one is very – you have these kind of combined emoji. So when you have, you know, if you’re using the skin tone modifier and maybe the gender modifier to an emoji, that’s actually three descriptors that get read out as a combined thing perhaps in certain systems. And some systems might read it as one unique character representation, but others might read out as kind of the separated emojis. Well, and that’s, yeah, even longer; right?
GLEN: As a blind person reading the descriptors, does a combined emoji really mean the three descriptors strung together? Or does it mean something completely different because they’ve been combined?
GARRETH: Yeah, that’s a good question. And I think the idea of, like, the combined emoji is great, and they’ve definitely taken off, like it really appeals to a lot of people. They’re able to represent themselves better within emoji, which is awesome. But yeah, I think it’s difficult. And part of it is, well, for screen reader users, if it is just repeating the individual descriptions, maybe that’s not the best way. So someone needs to look into whether there’s an alternative way to kind of do that. So I think you make a really good point there.
GLEN: Well, I thank you very much for both joining me on the podcast. This was lots of fun. Garreth, I know you have a YouTube presentation about a lot of the paper results, if people want more Garreth. The paper is called “Emoji Accessibility for Visually Impaired Persons.” Garreth and Ben, thank you very much for being our guests.
BEN: Yeah, thanks for having us. It’s been great.
GARRETH: Well, thank you.
GLEN: Cool. If you want to play a little with the emoji panel, you bring it up by hitting Windows Key and period. By default you’re on the screen that shows most recent emojis. And if you haven’t used them, it turns out being common ones like smiling face with smiling eyes. If you know the approximate name, you can simply start typing, and you’ll then hear JAWS announce “pizza,” for instance, if you enter “pizza” as your search key word. Pressing ENTER will insert that emoji, but leave you in the panel. So if you hit ENTER three times, what you’re going to get is three slices of pizza. Hitting ESCAPE will get you back to the document you’re working in. So a little bit of an emoji primer. I will try it this month, just because I’ve been emoji-missing for a long time. And I will try to remedy that. Perhaps you will, too.
GLEN: See you all next month for FSCast 203. I’m Glen Gordon. Have a great
August.