HANS PETTER HOLEN: Welcome to RIPE 71. My name is Hans Petter Holen, I am the Chair of RIPE. Have been so for a year or so now. Great pleasure I welcome you here to Romania. I think this is our first meeting in this country and it's always interesting to come and see a new part of our service region.
Since the last meeting, there has been a lot of activities going on in Internet governance, in the IANA transition, you will see later in the agenda that we have several talks about that during the week, both in the Plenary and in the Cooperation Working Group, it's a great focus on this community from the outside, people with getting increasingly interested in Internet, so it's very important that those of us that are here is building and strengthening this community in working together in an open and transparent manner.
It's really difficult not to look outside this room and all the things that are happening in our service regions. I won't mention any events in particular, but there are major terrorist actions in several of the countries in our regions that makes me really sad. On the other hand, our work here to build a better Internet Internet in this region, to pull our strength together to make sure we can get the best possible Internet to help our community, help our part of the world, the whole world to work better together is perhaps something that can be our contribution to the world.
It's also a great pleasure to say that IETF, during the summer, Rob Blokzijl the former Chair of RIPE was awarded the John Postell award for his service to the community over many years. I am sad to say that he is not here today with us, he was not well enough to travel, so this is actually the first RIPE meeting ever that he didn't attend.
So at this meeting, if you go to my first slide, we now have 525 registered participants at this meeting, and 336 of you have already arrived and I expect this number to be increasing. So, this is probably going to be yet another record breaker. So, if I even take this one and click, we have a couple of elections coming up, we'll talk about more about that later but we have a PC election, the Programme Committee, the whole programme for this week is put together by the Programme Committee for the Plenary Session and then the Working Group chairs forth Working Group. So. The other election that we have is north NRO NC, the Number Research Organisation, the number council is there to look after global policies and elect two of the ICANN board members, and there is one seat up for election and we have three candidates. They will get the chance to introduce themselves later today. And we will do the election on Friday for this.
There is an experimental network here, how many of you have used IPv6? Okay, so, the rest of you, you can actually try using IPv6 during this week meeting, and there is an experimental, our no longer that experimental, in NAT 64 network so you get a v6 only so ‑‑ if you can't reach your service over v6, so please test that, and give feedback to the IPv6 Working Group and show up there if you are interested in this.
And Jen is waiving here so she really wants a lot of people in the IPv6 Working Group.
We are getting such a big meeting now, we have a large community spanning Europe, the Middle East and central Asia. We have put in place two trusted contacts for this meeting, Nick and Miriam, are you here? So if you are experience something during the meeting that you don't really think is appropriate, you can talk to them or me or anybody else about this. We have now in place a code of conduct and it's printed here and you can find it on the website. Of course this is completely not necessary we are because we are a good, open be transparent community and we work together but it's been put in place by a task force put together by the PC and Working Group Chairs, so we have this in place in case it's needed.
If you walk to the microphone, because there are some microphones here, five microphones, please state your name clearly into the microphone and your affiliation, and we have stenographers here, so, if you just mumble, they will not be transcribe what you are saying.
So, this week is not only about meetings.
How many came just for the socials?
Well, there you can see that there is a hard start today with meeting the executive board, welcome drinkses, then there is party on Tuesday. On Wednesday you actually have to do it yourself, we think that there should be some personal initiative here, so then you can figure out who you want to go out and party with or just have an eye quiet night and then there is the RIPE dinner on Thursday.
And by this, I want to, of course say thank you to all our sponsors and then hand over the microphone so a representative from our local host, Eric, are you here?
ERIC BALEANU: Hi everybody. My name is Eric Baleanu, I present Interlan, the host of this meeting, welcome to Romania. First I would like to thank RIPE NCC for choosing Bucharest as the location for the RIPE 71 meeting. And for choosing us as the host.
It's a great honour for us and we hope that we meet your expectations. Also, I would like to thank the sponsors who made this meeting possible. Especially to RIPE NCC and Internet Society. I would like to tell you a few words about Interlan. Interlan was founded in 2006, it's an association of telecom operators, generally small or medium‑sized company and business to business services, local retail. Internet association has now 25 members from Bucharest...
Interlan operates the largest internet exchange in Romania, open to all national and international operators, content providers, educational produceers and so on. Interlan internet exchange has 60 connected entities, and daily traffic of over 70 gigabits per second. The main POPs are located in the largest neutral datacentre in Romania. We hope you enjoy attending this meeting and I would like to remind you to attend the RIPE 71 party tomorrow night which is also sponsored by Interlan.
Thank you.
(Applause)
HANS PETTER HOLEN: If I had known there would be dancers here I would have brought my violin so I could play to this. Now I will hand over to the Chair of the Programme Committee, Benno, welcome to the stage and you will introduce the Programme Committee and take us through the rest of the programme.
CHAIR: It's a great pleasure here to open the Plenary programme: The PC, maybe we can stand up, my fellow companions. So these are the PC members, they do all the hard work. They put a lot of time in getting a programme together. We had quite a number ‑‑ a good system of submissions, 27, and we had quite tough discussions what to select and not to select. We had very good submissions which didn't make the Plenary, but they were for the Working Groups and some of them were adopted in the Working Groups, so keep on submitting good sessions and presentations to the Plenary. We are more than happy to receive, to help you ‑‑ well, target the right audience for the Plenary presentations or help you to forward your presentation to one of the Working Groups.
So, besides that, we are also very curious how well we did, so we spend quite some time, many hours in getting this Plenary programme together. And we asked you, a couple of minutes, to rate our ‑‑ to rate the presentations we have selected for you, so please, get on the RIPE NCC access login, rate the presentations, it's really helpful for us. And also for the presenters because they get good feedback how to improve their presentations, but also contact us, one of the faces you have just seen, asking things, is this maybe an interesting presentation for the Plenary? We can share presentations, give us feedback, things with ‑‑ well, you really miss in the programme, we are ‑‑ well, they tell us we are quite friendly people, I think we are.
And finally, I want to, again, Hans Petter already mentioned the PC elections. There are two seats for election, so, please nominate yourself, the nomination closes at three o'clock, Tuesday, tomorrow. If you want to be part of this nice bunch of people and help us getting a good Plenary programme, please join us, contact me or one of my fellow PC members in discussing what's expected if you are a PC member and we are looking forward to your nomination. Okay. That's it. I think we can go to the first Plenary presentation, so...
NIELS RAIJER: Good afternoon everybody. I am very honoured to be giving the first really big presentation here, it feels like a keynote speech and up to a few minutes ago I thought this was because I submitted the most interesting presentation but obviously nothing competes with Eric from Interlan just showed us, so I am humbled. Anyhow I am here to tell you something about the state of the mobile satellite Internet. I do this in my capacity as the owner of a modest‑sized consulting ISP called Fusix. We build BGP networks and we run BGP networks for those companies who cannot do it themselves. And a couple of our customers include customers who have mobile satellite Internet as their primary business. Personally, I am also the vice‑president of the NLNOG foundation and I am one of the founders of Coaclue.
What am I going to try and tell you today? I'm going to show you what mobile, mobile satellite Internet looks like today and what it will look like in the near future. And my purpose of this talk is to show you what some networks are there, including my customers, are doing to your beautiful content and your amazingly high speed uplinks, and whatever, because if we look at your world, then you live in a world where people's mothers have 40‑gig Internet at home. Your routers get bigger and bigger and bigger and your graphs bandwidth graphs only go up and up and go up. It's 100 gigs, sorry, I will correct that. However, my world, this is our AMS‑IX port. Okay, it was three months ago, that is grown a little bit by now. And if you are not shocked enough yet, then this is ping output of very happily paying customer. You didn't know it was possible, right. I have seen a ping output with 30,000 milliseconds RTT in some cases: So what am I going to try and tell you here, my world is a mobile satellite world. You may know a little bit about satellite stuff with dishes and everything and VSAT channels where you rent a transponder on the satellite. Well, this is not it. It's totally different. We don't do dishes. We don't do transponders, we don't do lease lines out of the laner station, our customers are typically Inmarsat distribution partners, you don't want to use this, you only want to use this if you have absolutely nothing else. For two reasons, well, firstly, this horrendous ping output that you just saw, latency is obviously just enormous, but also the cost, you need to think multiple dollars per megabyte transferred. Obviously it depends a little bit on the package that you have and the contract that you get.
The negotiating that this kind of thing runs on is currently, mostly used negotiating is Inmarsat BGAN, because this stands for broadband, up to 492 kilobit and GAN stands for Global Area Negotiating, it's also a nice term. Beginning when I heard, it I thought they must be joking. They were not. It is a 3G network, it is actually simple to operate, because it is exactly the same as you have in mobile phone with an APN and everything and Inmarsat, being the satellite operator of this network is actually doing all those difficult 3G things like HLR, VLR, DPS oh ‑‑ did you get that? Yeah... all the difficult 3G stuff is done by Inmarsat. That's the trick. So the distribution partner, just gets an IP Sec tunnel from the IT GSN in which the traffic is delivered and the distribution needs to have some radio servers to hand out the IP addresses. And that's it.
BGAN uses L‑band frequents, 1.2 gigahertz region, and there is some talk of IPv6 because I know that otherwise Natalie would ask about it, but she doesn't have, to I'm already saying. There is a BGAN set‑up on Inmarsat BGAN where there is actually working IPv6, but it is still not there in the field. Hopefully, one day.
How does this look if you look at what the end user is actually seeing? I included here a picture of three land‑based terminals. Actually, my two customers on this serves, one is specialising in Maritime and one is specialising in Aero, so we don't do much with the land terminals; it's just for lab setup. And then the smallest one in the picture is like a pocket book and the biggest one is, I don't know, like a laptop from the 1990s, I have a picture of somebody who you may know later on so you see the size.
Of course, the speed that you get depends on the size of the terminal. Because the antennae is actually matching the size of the terminal, and we don't use external dishes or something, so it is really important to have a good‑sized terminal.
In land it looks like this, you just put it up on the ground, you aim is at the satellite and you have your one second ping. In Maritime it looks like a dome antennae which has a gyroscope built in and it automatically finds the satellite and keeps tracking the satellite. And in Aero, which is called SVB swift broadband, it's an omnidirectional antennae which is screwed on top of the aircraft that flies along with it and than means it will reach the satellite.
So, just for size perspective, there is somebody whom you may know and he was devoid of Internet in his summer house, so he came to me and begged, please help me, I need Internet, I need Internet. This is the guy. He is setting up his explorer 700, that's actually the biggest land‑based unit that there is for BGAN and Jap himself has grown a little bit since this picture was taken, but the terminal, I can assure you, is still the same size.
Now, this is all very nice, but the B for broadband is not really what we expect these days. So there will be a successor to the BGAN network and it's called Global Express. This works, although there are just like a handful of terminals already online anywhere in the world, and it gives you a speed up to, let's say, tens of megabits. Also, depending on the kind of packets that you buy, because if you want something affordable then your speed obviously goes down.
And actually this is not a 3G network any more; they built the whole thing with ethernet in mind and also with BGP in mind so there is really something nice if you connect a router to the like, they call it modem but it's also a router of course, if you connect a router to the equipment that you get from Inmarsat, and you run a BGP session between the two, then on your interconnect within Inmarsat in the date centre, you will automatically get whatever that router announces. So it is really nice tough being done there to interconnect the networks and make it work better than BGAN on an IP basis.
This network, the Global Express network uses the K A band to transmit ‑‑ I was going to say transmit or receive, but I'm not sure about the transmit actually but it uses KA band, 20 to 30 gigahertz. The downside of this is it is sensitive to rain fade, which means if it rains, there is no connection. So, for the Global Express land earth stations, they are looking, or they have selected locations where it does not rain a lot. Because if it does rain in the land/earth station, obviously everybody is without connection, I mean, if you have a Global Express terminal and it rains where you are, then you don't have a connection, but if it rains at the land/earth station, nobody has connection.
In that case, BGAN will be used as a backup, it's also built into the terminal. And when I asked in something like 2009 about plans for IPv6, they said yeah, of course, Global Express will support IPv6 from day one, no problem, you are right, we should include it. So, when I got the first provisioning form a couple of months ago, I clicked, there was a tab for terminal details, and IPv4, BGP, and... there were no other tabs so I asked where are the IPv6 tab? And it wasn't there yet, and they don't even have it in the lab yet. So, there is some work to be done there.
Now, the satellites that are used in this scenario, they are geostationary, it means that they don't move respective to the earth. It has advantages and disadvantages. One advantage is that you can always aim your terminal in roughly the same direction, and you will find a satellite. Disadvantages, the enormous distance between earth and the satellite itself, and its distance is about 30,000 km, so that explains most of the huge ping‑time that you saw in the output just now.
It also explains why there is no coverage on the poles, because obviously if the satellite is above the equator and the earth curves away from the satellite basically, the more you go to the north or the south, so, it is not just that the distance becomes larger but the area that the satellite transponder or pot beam has to cover is also becoming bigger, so, the power per square metre of either is going down, and that means that actually on the poles, cannot get coverage any more with this system. You could with another system from Iridium, for instance, who have satellites that are much closer to the earth and lots more of them.
Now, what do we see when we have customers like this? In general, I started doing things with satellite people in 2007 or thereabouts. These people don't have an IP background and, even today, they are still selling things like 64 kilobits transponders where the service requires an ISDN dial‑up to be installed in the land/earth station to terminate the 64 kilobits to some land base network. And I had almost ten years of experience running ISPs before I went into the satellite world, so, for me, in the beginning it was so hard to understand that people didn't know what a 247 knock would be or abuse handling or even anything about data retention laws, which is also a difficult subject in this, of course, because of the global nature of the whole system.
Apart from that, there is a whole ecosystem of companies and I have almost never spoken to any of those companies in a RIPE meeting such as this that offers service to improve the satellite experience. The latency cannot beat because it's still the speed of light, or a percentage of that. But, you can do TCP acceleration, automised VoIP even over this 800 to bill second ping time and there are lots of companies doing business in that regard and they seem to be doing it very well.
To look specifically at Maritime, you see this dome antennae with the modem router thingy that attaches to it and it will be installed in the vessel's machine room. The difficult things is that the vessel is usually away for a very long time so if there is something wrong with the hardware or software or anything that needs to be fixed, forget it cannot reach them. And to be honest, captains on board of the vessels who are usually in charge of the satellite connectivity, they don't usually want to cooperate in supporting the whole thing because their job is to sail this vessel to the destination and keep the crew happy, and that is maybe I think the only way to get them involved in the fixing the Internet, because, they call it crew welfare. I think you know what that means.
It is also very difficult to contact a vessel because telephone calls are difficult and expensive and the quality is very bad. So, forget about just calling a captain and saying what happens if you click on TCP IP? It can not be done. This is difficult.
In Aero, there are other difficulties, because my customer in this field specialises in the private aircraft segment, presidents, sheikhs, who are ‑‑ want to be able to use the Internet and make phone calls on board of their private aircraft and the problem there is that you never know when they are going to fly, because they don't even know when they are going to fly. So, forget about any maintenance Windows or something, it is just not possible, it always has to work.
Then if you look at typical things that are difficult in a satellite environment that you don't get on non‑usage‑based platforms. The biggest thing I think is the unwanted traffic. What happens? Obviously when you have a vessel with a customer on board who is trying to go to Skype or doing a Windows update or something, and this is blocked on the firewall land, then the segment that goes from the vessel via the satellite to land is really billable traffic, even it is blocked on land of course, there will be attempts sin packets to make the connection go up and these can amount to a lot of minute I can tell you. Customers say yeah, but I didn't do anything. I didn't go to Skype. I did not get the connection. So, why am I getting a bill? This can be solved by using an on‑board firewall. Unfortunately customers are moving in that direction more and more these days, although in the Maritime business it was frowned upon because it would cost money, so it saves a bit more money.
Another thing especially in the Maritime market that is difficult is the people not running their updates and ‑‑ of course you are not going to do a Windows update or download El Capitan while you are at sea because it would just be horrendously expensive. But the downside of that is that people don't keep their systems up to date at all which means they are vulnerable to all sorts of attacks, and you know, everybody in this room knows that those social engineering e‑mails become better and better everyday, and it is very simple to click on a link and get infected by something.
We try to do some stuff with, for instance, on the DNS recursors, here you see an example of a customer who are requesting a domain name that I cannot even pronounce and this name actually resolves to the good people of SIDN labs and they catch the traffic for us, so there's not any actual infection going on. But, we use these logs to prove to customers who plain about their bill that they had an infected system on board and that it is up to them to find the infected system, which is usually difficult because they are behind double NAT, the crew has taken their own equipment on board and everything, it's so difficult.
The on board firewall will more or less fix that, and more and more people are starting to use it. It also gives them the opportunity to sell vouchers for this crew welfare thing that I mentioned earlier.
But, what is, I think, in number 2 of the biggest problems is geolocation. You had not operated an IP network if you have not had a phone call from the American army saying, why am I going to Google.ru? This happens all the time. The network is actually set up in such a way that if a customer has a static IPv4 address then his /32 moves along with him globally, so then geolocation is totally out of the window. This is just useless in this scenario. Because you don't know where the customer is, we don't even know. We don't get location data from Inmarsat. So how can geolocation companies know? They don't. So some talks with Google periodically to make sure they show up in the correct language.
The service that is I spoke about earlier, acceleration and compression, this used to be something from the old days when we were still doing dial‑up, remember with squid proxies and things. Well, those things are still being used on a daily basis in satellite networks where people tweak TCP, use TCP accelerators to split the connection in two, land‑based section and a satellite‑based section and do funky stuff with the RTT ‑‑ with the TCP headers to make sure that the RTT is matched on both sides. There are also special webmail‑like services that allow you to just look at what they call the headers, but, to us, it would be the two address, from address and the subject of the mail. And then download only what you want to see.
And another nice one is the proxy which is squid‑based service that down samples the images, so when you look at the web page you think what happened here because you cannot make out what is on the images at all. This is to save traffic. If your crew member wants they can click on the minimumage and they will get a pop‑up that says this is cost you so much and then you can download the road image.
Now we go to political stuff.
Forced routing. Obviously, with this global network, there is some interesting being able to go to websites to see data that you are not supposed to see. So, countries where quite fast in making a presence in getting ‑‑ well getting their hands into the traffic, and it is done in various different ways, one of the ways is how the US does it, they just want that if there is somebody with a BGAN terminal on US ground, they want that the traffic of that terminal also lands on the US round station, which is... it's a nice place, I haven't been there, unfortunately ‑‑ so they can copy that traffic and inspect it. Other countries have built what they call a fourth routing gateway, and those countries include like Russia, China, Australia. I did look if all this info is available on Google before I started to tell you, so it is just on Google so I'm not telling you anything secret. And those countries want to have the traffic actually go through their country for inspection and maybe blocking, I don't know, and for instance with Australia, the thing is that what could happen is that traffic just goes from a terminal in Australia to the land/earth station in Amsterdam, ,their datacentre ‑‑ I can draw it for you if you are interested. It just adds another 300 milliseconds to the latency that is already huge. Why? Because the Government wants to to see the traffic. I can't blame them but it doesn't make our user experience any better.
Rounding up the presentation. I would like to show you a couple of future developments, and obviously ask you if there are things that we could implement to make life better. What we see a lot, the top question is: I want to block Skype. Well, I don't know if you ever tried to block Skype but it is difficult. Nearly impossible. If there are people from Skype here, I would like to know the secret switch that I need to push to may really block it. So far, we are just using contents‑based firewalls, what they call next generation firewalls, and they allow us to say that Skype text is allowed but Skype audio and video are not allowed. Obviously a content‑based firewall like that is far too expensive to put on board of a vessel so then you get the unwanted traffic problem again. So, we are not totally there yet.
There are more and more countries who want to have a forced routing facility, because they just want to know what people are doing in their country. So that is something that is also changing. Fortunately, a good development is that, with Global Express and this good routing of subnets and announcing via BGP, it will be much easier to do abuse handling because we will know who was behind a certain IP address. If there is a vessel with 200 crew on board we can give them a /24 of private space obviously but still we know who it was.
Speeds are increasing, Global Express is a very good development and I would like to know from you when I finish with the last slide just now, what further improvements we could make.
What I would like you to take along with you from this presentation is that the mobile Internet using Inmarsat actually sucks, but if it is all that you have, then it is what you want.
There are still lots of people in satellite ISPs who don't know how to spell, was it PI or IP? I do not remember. But they are really not into that world. And it is difficult to support end users with what they are doing basically because they are far away and very hard to reach.
And with that I would like to say thank you for listening and please microphones so I can get business developments, further improvement ideas from you, please...
(Applause)
CHAIR: Okay... questions...
AUDIENCE SPEAKER: Maxim Burtikov. When for example, I am Russian passport, I live in the USA and ask to deliver you a device in Australia and then go to China. To which country do you ‑‑
NIELS RAIJER: The country that receives your traffic depends on where the terminal is at the time. So when you log in, listen to that ‑‑ when you log in, your location is recorded and based on that your forced routing is applied. So if you are on a ship and you are outside of the territorial waters of Australia and you log in, you will remain without forced routing even if you sail into Australia. But keep this a secret, you didn't get it from me.
AUDIENCE SPEAKER: Geoff Huston... my question isn't about Australia. It's actually about, I believe your services using the hemispherical transporters on the Inmarsat space craft, is that correct?
NIELS RAIJER: I'm not the satellite guy...
AUDIENCE SPEAKER: Your coverage maps appear to point that your hemsipherical transponders, which is basically broad coverage, low power and bandwidth. My question is: Of course that's the only service out there and, as you said, there is a bunch of folks using Leos and a bunch of folk using spot beams. Using those, you can actually get more bandwidth, same latency, but normally at a different price point. To what extent is the market you're serving price sensitive?
NIELS RAIJER: Actually, Geoff, I must say this service also uses the spot beams. Spot beams cover a much smaller area of the land obviously, that's why they are called a spot beam, and the spot beams are usually turned off on these satellites and the satellites in question here are the Inmarsat 4 generation of satellites. So, as you call it the hemispherical thing is always on because they need to know where somebody logs in from but if you log in from a region where no spot beam has been activated yet the satellite will switch on the spot beam from that region and start giving you a higher beam service.
GEOFF HUSTON: For Angel user?
NIELS RAIJER: Yes.
GEOFF HUSTON: Christ, that must charge a lot.
NIELS RAIJER: By the way, in this picture there is something I forgot to add. You see three regions here. This is also not a secret any more but a fourth region will be added as we speak, because there is an important market, it's called India and Russia, that is now being covered half‑way with one satellite and half‑way with the other. So, they are actually adding a region between the dark blue one and the green one, and it should be online in a week or four or five...
AUDIENCE SPEAKER: You mentioned this TCP splitting. How you call it, are there any certificate...and by the way is the new service the global IP, would it also do that or is it also limited to this legacy Tor ‑‑
NIELS RAIJER: The TCP splitting service is something commercial from companies like Ziplink for instance, so they developed their own product. I don't think there is an RFC that that he made to do this.
AUDIENCE SPEAKER: I just kind of feel it would be incompletely incompatible with IPv6 and break many things.
NIELS RAIJER: I have a feeling you may be right.
CHAIR: Thank you very much.
(Applause)
JAN ZORZ: My name is Jan. I will take care of the housekeeping for the rest of this Plenary Session, and just on this note, for all our next speakers, when we are showing these cards with minutes, that means we are measuring the whole talk including the questions, so if you want to leave some time for questions, please take this into account.
Okay, welcome to Romania everybody. Nice seeing you all here. And our next speaker needs no special introduction, it's Randy Bush from IIJ, a big pleasure to have him here and he will talk about Let's Encrypt and about the automating and how to automate the certificates for the web sites. We heard that Richard Barnes, the original proposer of this talk could not be here, and we are happy that Randy managed to jump in.
RANDY BUSH: Hi, I am Richard Barnes, you will watch me grow 20 centimetres taller, 5 kilos lighter, 30 years younger and much better looking...
I'm sorry, I don't like to stand above everybody.
So, I'm giving Richard's talk because he had to do some things.
So, what problem are we trying to solve? The problem we're trying to solve is attacks, pervasive surveillance, all sorts of ugly things, okay. How do we deal with this? Encryption.
There is not enough being used. Here is some numbers, 40% Firefox page loads are in HTTP, that leaves 60% to go. You can read the numbers as well as I can. E‑mail, 40% to go. And what's going on between the routers, the home gateways etc.?
Okay. You are sitting there waiting to be collected by the Americans, the Brits, the Israelis, the Chinese, so on and so forth. Okay.
So, to get encryption on these things, the standard way we have to do it is TLS, but the problem is getting the certificate for TLS is an arcane art. It is go to some fancy CA, here is, you know, Richard quotes Cullen, who supposedly is very smart, I don't know. I have been an IETF area director and I don't think it indicates Smarts. But he couldn't ‑‑ he spent 40 minutes and he still has got a certificate. So not to pick on Jen, who is here, but he is also known as 'Furry'.
So, getting a certificate is actually a pain in the butt, okay, and it costs you money. There is ‑‑ I want to mention ‑‑ remind me towards the end to mention there is another way around this, but it is a good way. Which is let's make getting certificates as easy as DHCPing an IP address on your local LAN. So a bunch of people we know and trust, like Google and Amazon and Apple and blah‑blah‑blah have gotten together to deal with it, but here is some previous efforts: SSLMate was REST API to cover existing certificate authorities. And I don't know if it makes you happy to have those existing certificate authorities in your browser, they have 300 routes in my browser does not make me feel comfortable.
So, CertSimple was a way to get EV certificates, semi‑automated, send the validation and you'll see it in a few minutes. And now we have got let's encrypt. Which is a new CA, it's only accessible through REST API, that's new and sexy and you'll love it, it's like this year's cute way of having an API, I won't go into it. But, it uses a protocol called ACME, Automated Certificate Management Environment. One REST API, okay, and anybody could use it. Let's encrypt is just one instance, but theoretically all the CAs could move to this. It's currently implemented by Let's Encrypt. There is an IETF Working Group, I think this is supposed to be a positive statement. There is an Internet draft which I think is supposed to be another positive statement. And you can contribute on GitHub. I'm still stuck in SVN.
Getting the certificate with ACME is really pretty easy. I have actually done it. The client sucks. Let me tell you, but the way it does it is right. It's just this particular surface is a little hacked right now, which is you make an account, you prove that you own the domains and then you get certificates for those domains, okay. And the way it works is that you create the key pair, that's the password for the account, so that your messages to and from the serve are signed by the key pair. You register the key pair with the certificate authority. So the ACME client says, Hi, I'm Alice, I sign it with my secret key, my ACME server, my pubkey and the server acknowledges. Then we want to register some domains but I have to prove that I have control of that domain, or those domains that I am registering. And the way that's done is the client says to the server, hey, give me some hexadecimal garbage that I can install in that domain, right, I want to prove that I own example.com and the server gives me a challenge that says put this HEX decimal magic at example.com/whatever. So the challenge has two parts: the name to be provisioned and what is to be provisioned at that name, okay. And remember this came encrypted with my public key. I can decode it with my private key, provision it and then I tell the server ‑‑ first of all, I provision the name in the web server, in example.com, I tell the ACME server I'm cool, I did it just like you asked, the ACME server then issues a get, retrieves the data and looks good, your authorised to manage certs for that domain. So, this is the proof of control of the DNS domain. And then the client says, hey, I'd like that certificatem please. And the ACME server gives the client the certificate and the client installs it in the web server. The certificate ‑‑ this isn't in the slides ‑‑ the certificate that is currently being issued is restricted in the way 509 certs are for the use of a server for web. And when you read that detail carefully, it says that somebody else, a different service like mail could reject that certificate if it wanted to. In fact, nothing we know actually enforces that use clause. So, you could use the certificate for other things on that server such as SMTP, I IMAPS, etc. These are all done with http requests. There is the signing magic of authenticating with the CS in the first place, exchanging the token. Otherwise it's just JSON over HTTP, and that's supposed to be good. JSON is the new XML. If anything goes wrong, you want to revoke that certificate. So I already ‑‑ the server already knows me by this slide is bad ‑‑ that is the CA. So, the CA already knows me by my key, so I can issue a revoke request signed with my key, the server revokes the certificate and puts it in the server's certificate revocation list, the CRL here and tells the client that it was revoked.
So Let's Encrypt ‑‑ so that's ACME the protocol. And this could be used with any CA. Let's Encrypt is a service that is in beta today that is a new CA. It is free. It is sponsored by benevolent organisations with which you are familiar. The only way to get a certificate is through the ACME protocol. It's SHA‑2 from the start, ECDSA is coming. There are no humans in the loop. So, a reduction in engineering employment that I I'm not sure we'd approve of. And it's going to certificate transparency, so all certificates go to CT, all metrics are published, everything is open source. And the Let's Encrypt root versus authority cert, the root cert is already in all your browsers. Okay. They did the thing with the cab 4, I mean we're talking about Mozilla and Google and big monkeys like that so they can get it in your browser. You get one of these certificates, everyone is going to believe you.
How to get it? The general availability will be December 3rd. Until then you can sign up for the beta. I have done it. I succeeded, with some stubbornness. There is one former Gooey to use, this says this is the futuristic official client because this isn't what the client looks like today. This is what we wish it would look like, except the green should be a little more readable I would hope. So, it works okay, about 75% of the time. It requires root. It installs a container, it does ‑‑ this client is like too heavy. It could be done much more simply, and will be very soon.
You get the certificate on December 3rd. Until then, sign up for the beta. You can use the kinky client or there are other clients available. Use a provider like Akamai, who already uses it, or write your own.
This is the test, the beta certificates that have been issued. I'm somewhere in there on the green line. These are revocations down here. So, it's actually being used. For those that can't read, that's 10,000 up there. So that's a couple of certificates out there. There are some, by the way, who would say, hey, having half the world's certificates with one CA may or may not be the best security policy for the planet, but, hey, it's free and it's easy to provision and the CA game has been a rip‑off for too many years for us not to go to something that is free and open.
So, I think what we're talking about is security by default. Everything we do needs security. It needs to be automatic, okay. You can use Let's Encrypt on your web servers to get that stuff clean real quick. We need to apply the automation to the other CAs. And you should be looking at your network to see what else is unsecured, and doing something about it.
So, I'll add one more thing: There is a protocol resource recordTLSA resource record for the DNS called Dane, and what that does is trying to target the same problem as Let's Encrypt is. It let's you, say, put in your domain for, you know, myserver.com or whatever it is. I resource record that says this is the certificate for this domain, and therefore, you have a DNSSEC chain leading to what could be a self‑sign certificate, or you could get kinky like I did, is, I have my Let's Encrypt‑issued certificate with the Dane TLAS record pointing to it, okay. And that's the British for it is, I think, belts and braces, right, and I advocate it.
But this Let's Encrypt thing is going to, we hope, change the way we encrypt traffic for the web, and maybe mail, etc., significantly in the next couple of years, and significantly raise the encryption bargains passive monitoring and certain classes of attacks. So, you know, write to Richard, write to me, Google Let's Encrypt, or whatever you do, and come join the party on December 3rd when it's supposedly open, and can I answer any questions?
(Applause)
AUDIENCE SPEAKER: Dave Wilson, HHEAnet. Thanks, Randy. I found myself wondering, as I understand it, the purpose of these certificates is to, when you are encrypting in the first place, is to detect man in the middle, but it strikes me that the process where you prove that you are responsible forexample.com could itself be man in the middle. Is the thinking here it's better than the status quo?
RANDY BUSH: Here, it starts out with what we call a proof of possession, which is Alice sends her public key to the ACME server, encrypted with the ACME server's public key, there went your monkey in the middle, with the proof of possession which means her public key is kind with her private key. So therefore, if the server can decrypt that public key using that public key ‑‑ pardon me, validate the signature of the public key using that public key, then this server knows Alice must possess that private key.
AUDIENCE SPEAKER: Got you.
RANDY BUSH: Proof of possession. So this channel was encrypted. We have now done the key exchange, the server has Alice's key and knows that Alice ‑‑ public key, and knows that Alice has the private key. Does that help?
AUDIENCE SPEAKER: Benedikt Stockebrand. This is, in my opinion, still a problem. If I was your hosting provider, ISP, whatever, then I could actually send an e‑mail in your name sending some key I have generated, intercept the answer back and then move that to the web server. So ‑‑
RANDY BUSH: I'm not understanding if I'm your ‑‑ if I'm a hosting provider, what am I doing? What authority do I have?
BENEDIKT STOCKEBRAND: When I'm a hosting provider and I'm hosting your website, mail service and everything ‑‑
RANDY BUSH: And your domain. So therefore, I'm the one who has control over the domain.
BENEDIKT STOCKEBRAND: Okay. Yeah, but that ‑‑ it can also be somebody else in the path injecting these requests, and intercepting the answers from the ACME server. I think there is still some sort of problem how you can process somebody else if you have network access in the right spots. I have to think it through. It's just something ‑‑ I feel really nervous.
JAN ZORZ: There is Warren behind you.
RANDY BUSH: I don't believe that's correct because you prove you have control over the domain.
BENEDIKT STOCKEBRAND: The other issue about DANE and using the DNS, these are largely country‑based so just single point of attack for some national entities trying to mess around with it, so that's kind of scary, that's one of the reasons why we originally had these multiple CA approach, so it's easier to ‑‑ it's more difficult to subvert this ‑‑ it's kind of a national level.
AUDIENCE SPEAKER: Warren Kumari. So what Benedikt is saying is sort of kind of correct, except that, currently, CAs will already, the existing CAs will issue domain‑validated certificates, and so this is the sort of same set of proof. You prove that you control the domain, so if you can prove is in this, you would still be able to prove it in the original thing, you know, where CAs are currently issuing DVS it's exactly the I am type investigator recollect it's exactly the same vulnerability. If you have control of the domain, you have of the domain and that's all that CAs sort of care about.
RANDY BUSH: And we're lucky when they care about that. And ‑‑
BENEDIKT STOCKEBRAND: That's basically the point. The approach is ‑‑ let's say, the ACME approach is no more broken than TLS in general.
RANDY BUSH: Not TLS, than the domain‑based and CAs.
Secondly, as far as worrying about attacks on the DNS chain, paper in IMC last year, maybe it was the year before, 26% of websites are certified with certs issued by one secondary CA.
BENEDIKT STOCKEBRAND: Which is scary enough, but that's not the way it was designed, it's just what happened, which is bad enough but with the domain system it's just like that, but by design.
AUDIENCE SPEAKER: Just to know if there is any plan to automate this also with DANE DNS provisions to check that they have the real right to modify the zone files, because here it's the web ‑‑
RANDY BUSH: DANE is not going to tell you whether they have the right to modify the zone or not. DANE will tell you whether the certificate, okay, is down the DNSSEC chain.
AUDIENCE SPEAKER: But here there is only a challenge to make sure they can change the website. I mean, it's paged the http but that doesn't say anything they have the control of the zone file. Is there anything check one more check on that, additional check that they have the full control of the domain name on the zone file?
RANDY BUSH: No.
AUDIENCE SPEAKER: Is there any plan to implement it?
RANDY BUSH: The DNSSEC chain is the only thing you are going to get there. DNSSEC chain is kind of becoming assumed today.
AUDIENCE SPEAKER: So it would be do it yourself, not automated in this chain.
RANDY BUSH: Correct.
JAN ZORZ: All right. Thank you very much.
(Applause)
All right. We are now two minutes ahead of the schedule, that's good. So we have 11 minutes for each lightning talk, not ten. So, our next speaker is Thomas King from DE‑CIX, and he will talk about traffic volume dependencies of IXPs. I will measure the time, 11 minutes, including the questions.
THOMAS KING: So, I would start right away if I get the clicker.
So, welcome to my talk. My name is Thomas King, I am the head of research and development at DE‑CIX and I'm talking about traffic volume dependencies between IXPs. And I will go to the next slide.
That note is very important. So please take a few seconds to read it and remember it during the talk.
Please take a few seconds to read that slide, it's very important. This talk is about understanding what happened and learning from it; it's not about blaming anyone. And that's very important for me, that you guys keep it in mind please.
So, some motivation for this work is that I'm from the IXP community and we sometimes have discussions about how robust is the IXP inter‑connection system. And if you look on the map on the left‑hand side ‑‑ sorry, it's on the right‑hand side from your point of view ‑‑ you see that we have quite some IXPs in Europe and the question is: What happens if one of these IXP fails in terms of if there is an outage or something? And if that happens, what happens to the other IXPs around that IXP? And luckily, there was an incident actually during last RIPE in Amsterdam it was, I think, and we investigated that incident and also what it meant to DE‑CIX. As I said, this presentation is about results and about understanding what was going on.
So, let's directly look into the incident. What happened on May 13 that was during the last RIPE at AMS‑IX Amsterdam, there was an incident which resulted in more or less a complete traffic drop on their side, and that's already bad enough, but if you look on the impact that had on DE‑CIX Frankfurt, you see that at DE‑CIX Frankfurt we saw a traffic drop of 240 gigabits per second around that time, and this traffic drop happened quite fast, within five minutes, but it also recovered quite fast, the recovery started about ten minutes into the incident and we wanted to really understand what is going on because we actually expected a different behaviour, we were expecting that the traffic moved away from AMS‑IX to DE‑CIX so we were expecting a traffic volume increase instead of a drop.
So, if you look a little bit into the details and align the different views on AMS‑IX and DE‑CIX, it gets a little bit clearer. So, at 12:22 p.m., the AMS‑IX people created a loop with four times 100 gigabit per second interfaces, and the traffic was more or less blackholed at AMS‑IX Amsterdam and if you see, or if you compare that to the DE‑CIX traffic graph, you see at that time the traffic volumes start dropping, and after three minutes about 500 of the 600 BGP sessions at the route servers at AMS‑IX were already dropped and this is when we see the lowest of the traffic drop at that time DE‑CIX, so here we have the 240 gigabits of traffic drop already. So seven minutes into the incident, the NOC at AMS‑IX reacted, I think 7 minutes is pretty fast to identify what's wrong and to deactivate the parties that ‑‑ responsible for the loop, and at that time we see that the traffic volumes come back to the normal level we are expecting them to be.
And then, at 12:40, the BGP sessions to the route server were all back online at the AMS‑IX side and we see that the traffic is more or less on the same level as it usually is at DE‑CIX.
So, we wanted to understand what are the reasons for this traffic drop at DE‑CIX and what is the link between AMS‑IX and DE‑CIX here. So we looked into different potential reasons and we identified simply that are the most important ones. And I will quickly show you in the next few slides the reasons, the main reasons we identified.
The first one is remote peering routers were overloaded, so we, nowadays, have quite some customer at IXPs that are running remote peering, meaning they have one router somewhere and connect to many IXPs with the same router, and if there is an incident at one of these IXPs, for instance a loop or traffic storm or whatever, then the remote peering router might get overloaded by the number of packets. So it's usually just crashes or restarts or it shuts down its BGP session. In such a case, if it shuts down its BGP sessions or it restarts, all the other unaffected IXPs see some BGP sessions going down.
We looked in that and we found four customers at DE‑CIX Frankfurt which were affected by this, and they are, in total, responsible for traffic drop of 1 gigabits per second during the incident. So if you compare that to the 240 gigabits we saw in traffic drop in total, it's quite a number already, or only 1 gigabit, so it's not too big, but there is some...
There is some contribution.
So we looked into the ‑‑ into the second main reason we found. It is asymmetric routing paths and we design that in such a way that the upstream contains one IXP and the downstream of the routing path contains another IXP, and we looked into how common is that, especially in regard to DE‑CIX and AMS‑IX. And we carried out a RIPE Atlas measurement study and I will quickly go into the details because it's just a lightning talk so we have to be quick today. We found that 30% of the AS to AS paths we could identify and measure giving the highest coverage, whereas RIPE Atlas ‑ where this is on the upstream path.
Interestingly, 8% of the AS paths did not show us any IXPs, even that the ASs were connected to both AMS‑IX and DE‑CIX. That's a second reason.
Looking into the third reason is actually quite easy but hard to measure. If the Internet breaks, use especially not geeks, move away from using the Internet, they get a coffee, whatever, and so less users mean less traffic. This brings me to the end of my presentation. So what I showed you already is that there's traffic volume dependencies between IXPs, and we identified three reasons.
The first is overloaded remote peering routers; the second is symmetric routing paths; and the third is annoyed users which switch activities. We would like to extend the research activities we have done here to not only covering AMS‑IX and DE‑CIX but all the other large IXPs, to get an understanding if this is a common behaviour we see here, and we would like to come up with recommendations so that people running remote peering routers, for instance, know how to make sure that the systems are stable and run even if such an incident happens.
So that's pretty much it. Thank you for your time and I'm open for questions ‑‑ I already see Kurtis is lining up.
AUDIENCE SPEAKER: Kurtis Lindqvist. So, Internet traffic volumes at ISPs is a really, really poor correlation for end‑user experience, but that aside, I think the interesting thing would be actually compare the Mac increases at link and AMS‑IX because that would tell you something about the risk you are running, not something about the end user performance but a potential risk, and not just AMS‑IX and DE‑CIX.
THOMAS KING: That's a good point, we didn't know the numbers but we looked into that as well because we have this nice information already available.
AUDIENCE SPEAKER: Daniel Karrenberg, RIPE NCC. Just a suggestion for further research. Actually, in RIPE Atlas, all the measurements historic are stored, and I myself looked at some of the measurements during the time of this outage, so if you want to gain further insights on what I would call the behaviour of the exterior routing system, which look at where the packets actually went, it might be a nice idea to actually look at the measurements that were taken at the time. There's a treasure‑trove there. Just a suggestion.
THOMAS KING: Thanks for the hint, and there is also an article Emilia wrote for the Labs blog which is not published yet, but I think it's worth publishing because it looked into also where the traffic moved, and it's not only covering the AMS‑IX incident but also other IP hijacking incidents, but it might be good to get him to publish that article.
JAN ZORZ: Are there any more questions? I see Rudiger standing now.
AUDIENCE SPEAKER: Rudiger Volk, Deutsche Telekom.
I would like to point out and remind, that focusing traffic onto single points of failure kind of makes things more fragile; distributing more kind of, usually, makes things more resilient. For some of the focus points of traffic, be it interchange or exchange fabrics or be it remote peering routers, kind of creates more parties to share fate and usually the analysis of how that single point of failure inter works with other single points of failure that are around is hard to do, as you are trying right now, and for users, usually they don't think about it. But, kind of relying ever more on tighter focus, interconnects is kind of making things more fragile and that should be kept in consideration, and well, okay, there are points for having traffic in a very distributed fashion.
THOMAS KING: I completely agree. One last statement. If you want to see how asymmetric routing paths look like there was a project during the last RIPE Atlas who exactly did that, so go there ‑‑ there will be a presentation I think during the MAT Working Group which exactly shows that, so if you want to see things about that, just go there.
JAN ZORZ: Thank you very much.
(Applause)
Marco d'Itri, the next speaker from Italy, and he will talk about BGP security at the internet exchanges, and practical experiments.
MARCO d'ITRI: I work for Seeweb, an Italian infrastructure provider.
My little test. As the goal of finding out how well hijacking somebody else's network at an internet exchange works. It's something that we would like to not happen. We know that it's possible, and but I wanted to have some numbers about that.
So what I did. I asked a friend to lend me a /24 out of one of his own networks he was not using that IP space, so I could announce it and see what will happen. But nobody else knew about that. I had his permission to use his IP addresses. But from outside, this looks like an hijacking because I did not register the network in the IRR and did not tell anybody else in any other way about that. So I got, from my peering routers, a dump of the networks that I received at each internet exchange, and then I looked in each one of these neighbour autonomous systems for an IP address that would answer a ping.
This is the part, because it's more complex than it seems to get ‑‑ to find out at least one single pingable address at the ‑‑ in around the network. So, I tried to use the ping attribute in the route objects because it's really cool and it helps people who want to do measurements.
What I did was finding an address that would normally be pingable, then I would announce the network that I borrowed from the friend, and then try ‑‑ I tried again pinging this test IP address in each of my neighbour autonomous systems from the test IP. So, I would only receive traffic back if the other network accepted my announcement.
More details. I used the Quagga to receive the routes, damp the local routing table, and I used my little old programme, zebra‑dump‑parser to get a list of all the networks.
Lots and lots of scans of other people's networks, and you will find an address that will answer a ping.
I tried to exclude the dynamically‑assigned addresses because they may come and go at every time, so I wanted as stable addresses as possible to reduce false negatives. And then I used, again, Quagga, to select evenly announcing the test network from each one of the internet exchange one at the time. More Perl, so ping the targets and check the results. These are the results.
As you can see, a lot of my neighbours accepted the hijacked route. Most of the ones that did not accept the hijacked route, they did not because I was peering with them only using the route servers of the internet exchange. And some of of these route servers actually filter announces using the IRR.
Note so many of my own neighbours actually filtered my announcements. But, they really should, because I follow all the prescriptions in the routing resilience manifesto, which you should know about. Because I want to make simple for other networks to be able to filter my own announcements. So, they should try to do that.
Are there any questions?
JAN ZORZ: That was quick.
AUDIENCE SPEAKER: Rob Seastrom, ARIN AC and Time Warner cable, I have two questions. One: Did you try pinging from address inside the /24 while not announcing it to make sure that your peers are not pointing default at you.
MARCO d'ITRI: Yes.
AUDIENCE SPEAKER: Number two, did you try announcing prefixes longer than /24s?
MARCO d'ITRI: Networks I wanted do that another time but I remember somebody doing this kind of test without the hijacking part so this kind of test has been done already.
AUDIENCE SPEAKER: I'm interesting in seeing other people do the same test. Thank you.
MARCO d'ITRI: By the way, my code for doing this is really, really ugly, but I don't mind sharing it with anybody who wants to reproduce this test at other internet exchanges.
AUDIENCE SPEAKER: Jano, from Hungary. I find your test very interesting and thank you for sharing the results, they quite scary actually. I was just wondering whether you could use, instead of ping, simple mail or http for example at well known addresses?
MARCO d'ITRI: Yes, I could do that. The problem is again finding some test target. I used pings because most networks do not filter it completely at the border, so, I was able to find it except for maybe 10, 15 networks over all the internet exchanges at least the one pingable IP address.
AUDIENCE SPEAKER: Martin Levy, CloudFlare. Was the /24 that you used part of a larger allocation? And did it have an IRR entry?
MARCO d'ITRI: Yes, no, the larger allocation had one. Mine, the one I hijacked did not because the point was doing an hijacking.
MARTIN LEVY: Then the question would be, did the larger allocation have an RPKI entry, a ROA, which it probably didn't, but, if it did, would that have caused announcing it from your AS a different set of results if anybody is filtering at this point on an invalid ROA as a secondary test for you in the future?
MARCO d'ITRI: This is an interesting point. I really doubt that a sizable number of networks are doing this check right now.
AUDIENCE SPEAKER: It only takes one network to give you a bad day, so, it would be worth a test.
JAN ZORZ: Okay, I am closing the lines...
AUDIENCE SPEAKER: Erik Bais. Interesting results. The thing that I find specifically interesting is on the results side. I think one or two slides back, you say this is inexcusable, and I do agree with you on some parts here. However, some of this may also be technology‑related. Not every equipment out there can actually do proper peering filters and traffic filtering like this, not everybody has Google budgets, and if you actually want to do this, we actually have set up the system, we have the software in place to actually do the filtering like this, but if you start peering with, you know, companies like RETN, Hurricane, those kind of guys, they are pretty large peerings prefix filters and doing that kind of stuff and keeping it updated ‑‑
MARCO d'ITRI: I agree, I see your point.
AUDIENCE SPEAKER: That actually has a real impact on the hardware you're using and the kit you're using.
MARCO d'ITRI: I am from a small network so filtering me is easy, filtering one of these networks is hard.
AUDIENCE SPEAKER: Yeah, but, typically, those are the networks where you might actually expect this kind of behaviour from.
MARCO d'ITRI: Everything counts.
AUDIENCE SPEAKER: Rudiger Volk. I'm not really surprised about the results. Yes, I think there are too many parties out there that take the infeasibility of filtering some of their partners as an excuse of not filtering anybody, which is quite clearly a bad idea. Making the choice in all the cases where it is feasible and where it's not feasible, unfortunately is additional work which some people are not doing, and, yes, the requirements on the equipment used should take into account actually doing as much filtering as possible. So, saying I have a small box with outdated software is not an excuse to participate in unsafe peerings.
MARCO d'ITRI: I agree.
JAN ZORZ: Okay. Thank you. Thomas.
AUDIENCE SPEAKER: Thomas King from DE‑CIX. I'm looking at the results. I am wondering at DE‑CIX, we see a couple of unavailable peering sessions. The question is, did you peer with the route server?
MARCO d'ITRI: Yes.
AUDIENCE SPEAKER: Then it's even more interesting because we do IRR filtering, so, it might be interesting why this number shows up still. And I think we should take off‑line that and discuss it later because it's really interesting work, so thanks for that but I want to get a better understanding of what's going on.
JAN ZORZ: Marco, thank you very much.
(Applause)
All right. So, a few announcements. Please rate the talks. Please rate the talks, there are buttons on the website in the agenda to rate the talks. We'll be back at 4 I think with more very good agenda, and at 6 p.m., we have the best operational precise Task Force meeting in this room. You are all welcome. Thank you.
(Applause)
Coffee break.
