Robotic Reaction – Giuseppe Loianno, Electrical and Computer Engineering & Mechanical and Aerospace Engineering, Tandon School of Engineering, NYU – The Development of Advanced Tracking and Control for Robotics Technologies

Richard Jacobs: Hello. This is Richard Jacobs, I have Giuseppe Loianno Assistant Professor in the electrical engineering department and in mechanical and aerospace engineering, the Tandon School of Engineering at NYU, New York University. He runs the agile robotics and perception lab (ARPL). They perform fundamental and applied research in the area of robotics autonomy. So thanks for coming. How are you doing today?

Giuseppe Loianno: Thanks a lot for the invitation. I’m doing very well. Thanks.

Richard Jacobs: Yeah, specifically in the realm of robotics agility, we are going to talk about using eye tracking or eye control drones. Is that when you like to focus on today?

Giuseppe Loianno: Yes. So that’s the main topic, we’ve been developing this kind of technology for the past year and so we got some really exciting results.

Richard Jacobs: Well, why control the drone with your eyes? Why not a little remote control that you hold by the RC cars, what’s the tradeoffs in doing both?

Giuseppe Loianno: That’s actually an excellent question. I mean, we talked to that many people are suffering about the ability to really control these machines with the external devices especially with remote controllers and basically this used today by expert pilots or it’s generally very difficult that Naive users start to use and they are very good at using a remote controller so that’s why we told to use, basically saw my tracking glasses that gives the ability to the user directly to control the drone using gauge.

So what is looking and where his or her eyes are pointing in the direction that the drone is going to go. Basically, the second main reason is also that we really are trying to make these devices and especially these drones accessible to everyone. So it’s sort of this kind of eye tracking glasses gives really the idea that you ever sort of drone companion and not anymore a machine that is absolutely a fire from your understanding.

Richard Jacobs: So you know what, I noticed that I’ve tried to use drones a few times and it’s very hard to control them. And the sensitivity of the joystick that you’re using to control it is usually, you know, it takes a while to get used to the drones fly over the place and crash and they start to get out of control. People tend to push hard on the joystick, then they crashed and the other way. But with the eye tracking, I could see that it would follow the natural motion of your head and you wouldn’t with your head to one side and then centrally to make a correction. So maybe it’ll work better. I don’t know.

Giuseppe Loianno: Yeah, so that’s the old band. We also try to be coupled basically the orientation with respect to real gaze that many people can see. They’re the same thing, but it’s actually not for better controllability of the platform and the other really attractive a part of these devices that it’s really non-invasive. So it’s like a normal pair of glasses that people wear basically every day and so we should really imagine that in the future all the time, most of the glasses would be probably in bed with the sensor suites that these new glasses provide.

Richard Jacobs: Okay, you know what, I noticed that I’m just looking around as I talk to you and it’s not comfortable for me to keep my eyes fixed and move my head. Nor is it comfortable for me to keep my head and just move my eyes. Well, that sounds like you know, doing eye exercises so it seems like you need the combination of the two head movement and eye positioned or where the gaze is, because you know let’s say I’m staring in a lamp and I moved my head to the left. It feels unnatural for me to keep my gaze on the lamp, so it seems like the most natural thing is to remind those too, maybe I’m wrong.

Giuseppe Loianno: You are perfectly right, but the main problem is that gaze moves away faster sometimes than your head. Especially if you point to lateral objects compared to where you are. You basically notice if you take your attention that basically it’s your eyes are moving first and then your head is moving after you basically. So they have a sort of Anti fee, but even formation that we wanted to capture in our setup and so we are able to do that basically.

Richard Jacobs: So what have you noticed are you at the stage where you’ve made prototypes and people are using it and you’ve observed in the lab?

Giuseppe Loianno: So basically now we are still at an early stage, so these have been just published in a new paper it’s working pretty well. We plan to expand to multiple users and now we are able to let them interact with each other.

The other problem, we are also trying to obtain the full 3D information from the gaze, which is actually quite hard, so we can basically send the drone in the direction in space, but we cannot really specify how far, so this information is still missing and we hope to collect it through new computer vision, algorithm based.

Richard Jacobs: Yeah! So do you have to combine days and head motion? Well, maybe even other signals you can get full control of the drone.

Giuseppe Loianno: The idea is to combine, first of all, to combine computer vision technique. Like how, for example, you do object detection in the environment and use this detection to basically get a better estimate of the 3D games. The other thing we would like to do is a sort of multi-modeling directions. So people, of course, don’t only use the gazing formation, but they use other types of information as a gesture or like voice. So we would like to combine all these types of interaction together in a unique framework for better controllability of the drone and for better, let’s say human-robot interaction,

Richard Jacobs: And what other kinds of drones that this would be more suited for than ones that it’s not suited for it?

Giuseppe Loianno: I would say that the approach that you are using is sort of quite general. I mean it just depends on the user and it can be extended to multiple types of platform, so the algorithm that we have developed on the user side is quite general.

What we would like to do is to make some kind of a case study to see how people react. And how people feel using these type of glasses. Right now we have tried on a couple of users, let’s say around five, about to like to extend these approaches to also different types of people with different ages to see the difference in the perception of the system.

Richard Jacobs: What kind of tasks seem to lend themselves to eye tracking and which ones don’t seem to be good for this.

Giuseppe Loianno: It seems to be very good to point to a specific location and do understand the human behavior basically without using an explicit comment like let’s say the voice or the gesture. So it’s sort of hidden command that’s coming directly from the eyes. So it’s very difficult to perceive from another external user. I would say it’s sort of complementary to the other, let’s say, ways of interaction, It’s sort of less expressive, but at the same time it’s probably more powerful to position the drone in space.

Richard Jacobs: Is this literally only a one person control or does this open up the ability for multiple people to control different parts of the drone so that people fly cooperatively? Or is that not even unintended on your radar?

Giuseppe Loianno: So yeah, we envision to basically have a complete framework that incorporates multiple robots and multiple humans in the loop. The key idea is to have human collaborate with each other, but at the same time can interact with those machines to solve a complex task. So this is just the first building block of these entire structures. So we really hope to extend, basically the approach to, multiple robots first with the single user to control this swarm from the single user perspective and then to extend the single user to multiple users. Basically, the drones and the different machines are assigned to the user that is part of the task.

Richard Jacobs: Well, imagine if someone controlling a swarm and then a second person comes in and the part of the swarm breaks off and the other person who controls it, I guess there are all kinds of interesting things you could do.

Giuseppe Loianno: Yeah, it’s sort of interesting because at that point you have multiple agents that they have to collaborate together. The drone is the one that you can literally control the human can also run, can also walk or like perform some kind of task by themselves and then the drone can compensate for what the human is not able to do as a sort of a companion.

The main thing is that these machines can accomplish with the human competitive task, let’s say mapping and environment so that can be accomplished in a faster way, both from the human perspective and from the drone perspective. And also the drone can be sent in space or in the region that the human doesn’t know to give a first overview. And so that there are safe to identify humans to enter, for example, in a building or after a natural disaster, we should really imagine these drones as helping ourselves during normal life and not anymore as enemies.

Richard Jacobs: Wait, I just realized something obvious if I’m controlling the drone by looking at it, that’s totally different from me seeing through the drone’s camera and controlling the drone is if my eyes are in the drone looking out. How do you reconcile that or what you found is best?

Giuseppe Loianno: Yeah that’s a good question. Actually, this is looking to the camera of the drone and controlling, it is becoming a sort of sports. So that’s like FPV pilots. I wouldn’t say that for now this can be for sure reconciled. I mean if you think about way to augmented reality device the table also gaze ability, you can basically think about reconstructing the environment using for example, the using the camera from the drone so the user can basically have a perspective, a new perspective in his or her new word that is a combination of images and the maps that comes from his own exploration and that also comes from the drone exploration so these can increase actually the human capabilities.

Richard Jacobs: Well, what happens if we controlling a drone and it goes out of the line of sight, you switched to onboard drone camera mode, Can you have both of those in your view at the same time? Would that confuse you? It just seems like a difficult problem.

Giuseppe Loianno: Oh yeah, so that’s certainly a difficult problem out of the line of sight. They should certainly switch to the field to the camera view. But within the line of sight, you can probably also use a combination of both but not showing the image to the user but showing a new word that is a sort of a combined the map combined 3D reconstruction drone from the user perspective but also from the drone perspective. So the human basically as an open word in front of him that shows a much larger area, reconstruct it to increase the situational awareness.

Well, what if you pair to drone with a follower? They kept it a certain distance and its only job is to mimic the first drone, stay a certain distance away and habits camera trained on it. So when you go out of the line of sight, that camera takes over and you still are getting the pseudo-line of sight, you know the view of the drone. But then you could still switch to that drone’s particular camera, the main drone, and she can close up anyone.

Giuseppe Loianno: Yeah, that’s definitely an option but the main problem is to use the camera of the drone many people need lots of hour of training and this is still like some kind of concern for the naive user. So using the drone is certainly an option, but it’s not something that people feel comfortable or ready to use it as a first experience.

Richard Jacobs:  Well, that’s what I’m saying, maybe you need a follower drone that people you know at a much lower threshold for training versus using the drugs hammer.

Giuseppe Loianno: That’s definitely an option.

Richard Jacobs:  So what’s the Holy Grail for you? What kind of experience do you want to create for users and how long do you think it’s going to be able to get there?

Giuseppe Loianno: Okay, so I think that the experience we want to create is that first of all, the drone has to be a sort of a companion for the human so I would like that drone, for example, is able to enter into a building with the human and the human receives direct feedback. Meanwhile, he’s pointing to different areas, received direct feedback about the structure. For example, if there are some critical parts that need to be replaced and that the human intervention as such, to facilitate the inspection and basically mapping task that right now are basically just executed by an expert, people by working in these areas or by people that know how to control manually a drone.

This is for example, so the case of bridge inspection, so many people send an expert user to look at the bridge manually or there was some kind of pilots that send a drone directly the pilot, the drone around the bridge but a third case would be to have a user that uses the gaze to drive the robot around the bridge. And that same time received feedback online about the current status of the bridge and what needs to be replaced or maintained in the near future.

Richard Jacobs:  Well, very good, so what’s the best way for people to find out more about what you’re doing and maybe watching videos on it or when can they get the device itself to do the eye tracking?

Giuseppe Loianno: So the device, I believe it’s already available online. We post mostly the algorithm, probably within the next couple of years. This is like a long process in the research that just started the latest update. We certainly are found on my website. That’s the robotics and perception lab and on the DC’s a website, that’s the new program that the sponsoring actually this type of research.

Richard Jacobs: Well very good, is there any other way to get in touch with you or in your lab and departments to ask questions or give ideas?

Giuseppe Loianno: Yeah, so my personal email Loianno @nyu.edu. So people are really welcome to email me directly. There is a newspaper that actually on my website that includes also the latest video, the latest release in terms of algorithms and also the latest collaboration that we have in addition to the main research areas, that we focus on. So gaze driving is just one of the multiple works that you are carrying on at Nyu basically and we have also affiliated to the lab, a YouTube channel that you can find actually on the lab website.

Richard Jacobs: That was great! We’ll thank you for coming on the podcast and it’s interesting to see all the implications of what you’re working on. So keep up the good work.

Giuseppe Loianno: Thank you for being here. Thanks a lot for your time and thanks a lot to everyone for listening.