JEN LINKOVA: Welcome to IPv6 Working Group. I think it's a group you should be, right. If you want to hear about measurements you probably need to be in another room. I know you still ‑‑ I am pretty sure you are going to read your e‑mails while we are talking? So I would appreciate if you could read your e‑mail using NAT 64 network. And let us know if you couldn't read your e‑mails. Or if you could. I will actually going to talk about it at the end of the session. So, now, it's wrong slide. It's going to be last talk of today. Let's see. I want to ‑‑ first of all, thank you for scribes, thanks everyone who is helping us to have ‑‑ to run this session today. We start with I think agenda bashing. How can I switch slides or shall I ask you to? Agenda bashing. So ‑‑ thank you very much. So this is what we have for today's sessions. If you have any comments, suggestions? No. So, just for your information, this is the agenda for tomorrow, please come back. Even if you don't enjoy today's sessions too much. And by the way, who has read meeting notes from the last meeting? No objections. Approved. Great. And now, by the way, I was asked ‑‑ I was advised to mention that we have code of conduct on this meeting so please be nice to each other, don't kill kittens or any other animals. And besides that, please rate the talks, because I really want to know as Chair as people to try and build agenda, really want to know what kind of talks you like and what kind you don't like and what you want to hear about next time. So if you don't like the agenda, it's not actually my fault, not only my fault at least. Please give us feedback on the agenda. Let's start, Benedikt.
BENEDIKT STOCKEBRAND: So, what I want to talk about is some weird things that have happened to me and some of my victims deploying IPv6, not at the ISP level or similar, but at the enterprise level.
So, a couple of things first. This is not for you, this is for people finding the slides on Google but it just doesn't make sense to take a look at the slides by themselves. Second thing, I am dreadfully sorry, no kittens. Jen actually came up with some nice pictures of kittens in blenders but I decided no, they might be copyrighted. Yeah, the problem about IPv6 at the enterprise level, do I have to explain this? TCP IPsec, somebody ‑‑ IP stack. As you know all, IPv6 is a network layer protocol and as such, IPv6 is the network topic, right? You get your network, do IPv6 and that's the end of the story. Well, that's what some people think. When you do that at an ISP level, this is actually quite a bit of work; it's also quite a bit of work at an enterprise level but only once you are through with that the problem really starts. As you all know, we have a layered protocol stacks so every layer is only talking to the adjacent protocol layers and because everybody is running ‑‑ nobody is talking to the network layer directly, everybody is talking to the transport layer so applications run with IPv6 unchanged. Some people claim. Well, that's correct, except for the odd issue, and that leads to a huge problem in enterprise environments where there is slightly different sort of things going on than elsewhere, and I actually got this next slide ‑‑ let me get my notes ‑‑ I got this next slide on IPv6 for managers, I even do that, I even put if a time and charge extra. That is what the TCP/IP stack looks like in an enterprise. It's not to scale, by the way, but the video don't have the resolution necessary for this to do proper scaling. So the network layer is a tiny little thing, you can even tell reasonable managers that this is at a kind of throttle point is everything has to go through there. We are going to mess around with that, panic. But once you sort this out, the trouble here starts. The big problem is applications. And by the way, the reason why you can't look inside from up here is because you have just eaten. This is where real trouble starts and they are enterprises that are older than the average ISP, and they have a huge, huge zoo of applications that are real pain to deal with. You think IPv6 is a problem at the network level, it kind of is and yes, you are the first one everybody is waiting for and there is nobody around you can actually ask what are these funny letters doing in the addresses, what are they supposed to do there, they are supposed to be numbers and that sort of stuff, but the networking isn't really the issue; the real problem are applications. A couple of things that have happened to me, really, seriously:
We have some software developed in‑house, OK? But we still have the the developers, that is a very good thing, it doesn't have to be like that. We have the developers so what language are they using then, Java. You have to touch Java to work with IPv6, right. Yeah, but we are using some sort of library for all the network stuff so we don't actually touch sockets and IP dresses and the resolver and whatever, so what about that? Well, I don't know, what library are you using? So‑and‑so and so‑and‑so. Never heard of that. It's an OpenSource, OpenSource is good, because things get really ugly you can still fix the source. But it's been discontinued a couple of years ago. We need to take a closer look. We took a closer look and found out that the second‑last release, the release of this library actually supported IPv6. Great relief, only two questions:
Number one, are really all the in‑house developments using this library, are there some that don't? I don't know with this particular customer, we will see what happens there at some point. And the other problem ‑‑ the other question is, is it actually stable enough to be usable?
As far as we tested it, it works, but the big thing with this customer is they are not yet at the point that they can actually deal with the applications.
So, that can happen. Another thing that can happen is, oh, yes, just got a few Linux machines here, they are just servers, we need to check the software installed there, like 5,000 Linux packages to see if they are IPv6‑capable or not. Now, the vast majority of them, like 4 how is easy, that is an excellent font, doesn't matter, some you end up checking the sources because you don't get any other kind of information and that takes quite a bit of time. It can be a huge pain especially if you find it difficult to find people who know what is going on and you have that lot there because they have got a huge range of features, products, services, whatever, to deal with, and then of course, then there is the other languages starting with C, unlike |Java like Cobel, clipper and D Delphi. What are you going to do with these? How do you find these? You can spend weeks trying to sort that out and get a basic idea what the situation is like. You only find out in detail when things blow up in your face trying to deploy these or deal with these. And that's not the worst part. What is even worse, yeah, we have got this special industry‑specific software bought from obscure manufacturer specialising in the legal specialities with cross‑border tax things between, let's say, Austria and Lichtenstein, which is so obscure that there is only one manufacturer and they sell you whatever wrap they have and you are going to buy it because you don't have another choice. That is when things get scary. Literally, I had one customer, they asked one of these people, so what about IPv6? Well you don't need that. Well yeah sure but we plan to ‑‑ believe me, you don't need that. But what do you mean, we have ‑‑ you don't need that. End of discussion. And then you have a problem. You hardly find that in broadly ISP or similar context at the enterprise level, that's a huge pain.
And then of course, there is the occasional things I have seen, not with IPv6 or IPv6 deployments but elsewhere, where a customer had ordered a custom made software and tried to save some money by not buying the sources as well. Absolutely crazy but these things actually happen. And then you are sitting there, we need to fix this thing, OK, the one thing ‑‑ the first situation I have seen that was year 2000, but in this respect it's a very similar problem. However, this is a big problem by itself, but not in comparison, because then there is layer 8, the problem in front of the keyboard, and there are a couple of things I have noticed in there which are really, really, really scary. So first off, you are dealing with, I am not going to say second grade techies because that is unfair but what you frequently find in enterprises are people are pretty good in their very specific stuff but they don't know anything left or right, like, and lastly talk yesterday said, OK, you have network people who are actually dealing with ‑‑ on the command line all the time. And you show them cluster SSH which is also useful for Ciscos because get things done on all sorts of machines at the same time and the only response you get is cool, do you have that for Windows? What do you mean? People who don't know how to write a one‑line Shell script. That sort of people.
And that makes it difficult because when it comes to IPv6, you have to deal with all these other things, you need people who actually are capable to make use of these things, it's not if you want to go net Neff Ops but IPv6 as well, these people what they want, they have this cookbook memory; when this happens, I do so‑and‑so and so‑and‑so, and I have heard that from first level support at ISP actually, on IPv4, so you need to fix the Net mask, 2552552550 on IPv6. What is that for? To make it work. What does it do? It makes it work. You have these sorts of people, sometimes. And in an ISP con particulars frequently have people with a broader background don't expect that enterprise. These people are frequently rather unhappy with the things seriously changing, because they are losing control and they are afraid of losing control, it can be a huge, and I mean huge, problem.
Another thing, and I have actually done IPv6 for managers and I told them all about this, you have these people ‑‑ I would actually need a chair and something like this ‑‑ I have got eight more years to retirement. I don't need any of this. Unfortunately, those are the people who frequently get asked about is this important or not? Which can be a huge problem. You just need one of those to really give you a lot of problems. We have seen quite a few of these in training, it's not so much with the project apparently because they manage to make sure this project never even got started.
Similar issue, if you have people with like 10 or 20 or 30 years of experience with IPv4, and this cookbook recipe level of knowledge and you tell them, OK, there is something new, so they are at some point at the same technical level like the youngsters, possibly even less so because the youngsters have already taken a look at this IPv6 stuff out of curiosity, you might expect, it doesn't happen always but it can happen, there is some really serious trouble people not getting along in a network or whatever team, any longer, so this is where things really get ugly and it doesn't help to be a technician at this point, you want a psychiatrist. But you really want to expect that. And there is ‑‑ the resistance towards introducing IPv6 at layer 8, these people, is not something you want to think about. Unless you actually want to do this, then you actually should. It can be a really, really scary, and troublesome. However, of course, if there is a layer 8 there must be a layer 9, non‑technical management. Now, I don't have much of a problem with these, mostly because I have very few customers doing IPv6 deployments, those are the places where management realise we should spend money on that. There are not that many of these. So every once in a while I wind up in a situation where I have to explain managers, you want to deal with this. Why should we spend money on this? Everything is working fine. My usual answer is: Yeah, you want it to stay like that two years from now. Oh, by that time I have got a different job. I have heard that live. But then why should we spend money on that now if it's only going to be a problem two years from now or whatever? Yeah, OK. Because ‑‑ if you source out your IT and you don't have anybody you can trust any more in your own organisation to tell you what is going on, so two years is a set date. No, it's not by year 2000, which has been announced 2000 years in advance, it's something that had happen gradually but it will hurt. Then we start to take care of it when it starts to hurt. OK. And in that case you will get hurt. Frequently, you don't make much friends with these reasonings but every once in a while you are on to people who actually listen and understand and you get things done.
So, those are the things that you can run into, no easy answers, it's just things you want to be aware of. And if you are an ISP or anything similar, you might have to deal with people asking you, but what is this IPv6 thing about? You ‑‑ actually you might want to tell them this sort of stuff. Now, this ‑‑ some more stuff coming up which is more important to you than enterprise. There is the political layer sort of thing, including legislation, jurisdiction, how many IP ‑‑ how many bits from an IP address do I have to mask out to anonymise address, whole bunch of things. But that is not that exciting for ISPs. And then of course, there is layer 11 but so far the churches have decided right now it's probably not the very best time for another crusade. So, these are things you can see in an enterprise context when you try an IPv6 deployment.
And one more thing, which is not on the slides because I only use the slides for these funny pictures, there is a widely ‑‑ approach to IPv6, get the web servers v6 ready and mail servers, basically the perimeter of the company and keep doing IPv4 inside forever. Well, except where you use as SNA or IP X or you never got rid of it either. There are a couple of problems to that.
First off, if you overdo it, you will wind up with another IPX SNA or old‑fashioned network technology that is getting rather expensive these days, but there is number of problems you have to deal with if you want to do this. You have to get your layer 8 ready to deal with the situation, and that's not just we do a software up grade, whilst training, convincing people, dealing with all the stuff I have told you before. So we outsource this. Can do that. Find an outsourcer who can take care of these things, how do you know if they know what they are doing? And I have done a training on IPv6 with an outsourcer, basically, they got a one‑day crash course training what basically is IPv6, and next day they had to write a tender or offer or whatever for one of their major customers so they would do IPv6 for them. People had one day of basic overview sort of thing, was supposed to run IPv6.
These things happen. You still have to have IPv6 in your DNS, it can be a bit of a problem, in your network of course, and that can be scary to some people. You still still have to tell your developers, web designers, whatever, yes, it speeds things up, but don't put little IP addresses in your HTML text. It's ‑‑ this happens, I have actually seen a content management system which has advance acceleration feature replacing names in the HTML file with IP addresses. I don't know what madman came up with this, but they actually sold it as a feature.
What is even worse, and that is where things get ugly, you need to get the monitoring up and running, and that is frequently not trivial. You basically have to take your entire monitoring of your external services and duplicate it to monitor the IPv6 side as well. So, that can be quite a problem, of course you have to test things and see they actually work and they are not just expected to work but they actually do work, with a huge zoo of software you have can be quite a bit of a pain, and the biggest problem with this approach actually is if you do that, and something goes wrong, you get maximum publicity. So if do you this in a rush job, you are really, really asking for trouble.
OK. So much about that. In case you have any comments, questions, whatever, there is some contact data and in case some of you ever make it to Cape Town, be careful, there is some weird locals who have habit of hanging ‑‑ questions? Completely boring, doesn't have to do with ISPs, right? Yes I know. But your customers might be grateful if you have a bit of an idea if they might expect from IPv6 which is going to be different from your business.
JEN LINKOVA: Do you think it would help to have a kind of frequently answered questions list? Like why do I need this, why do I need to start now? What all this v6 traffic is about? With properly animated pictures or something like that with kittens in it?
BENEDIKT STOCKEBRAND: And mixers, blenders. Yes. The problem is those people read these things are the one don't have to convince because they are bright enough anyway. It's very difficult to reach the people who should realise but they are just so far away from technical questions that they can't be bothered. So, you reach the few people who realise that year 2000 is five years, we should take care of it now but you won't reach the ones who wait until October '99 to speak in year 2000 terms.
JEN LINKOVA: I will think more about the situation when there is someone in the company who cares at least a bit, otherwise they probably would not be even talking to you and that person they need to convince not so technical people who probably do not care yet about this. I am curious how much would they ‑‑ how much would it help to have a kind of explanation, so people don't need to come up with their own explanations every time.
BENEDIKT STOCKEBRAND: Yes. I mean, that is why I am telling these things, for what, 12 years now. All you can do is really spread the word, make sure people have the arguments they need to tell their bosses, but I don't expect the vast majority of people to realise that they should have taken care of this before ‑‑ most people will only realise when it ‑‑ the business, nothing we can do about it. We told them over and again so it's not our fault but of course, every bit helps. That's for sure. But, don't expect too much. This is mostly ‑‑ so I make living out of IPv6 trainings and consulting but it's not, you can't expect to convince everybody within the next 48 hours or so.
Ray: There is a list available for managements to see what they should do, why they should do it and yes, I have even made them read it. But, there is different kind of companies, there is companies that have shareholders and there is companies that don't have shareholders and some commercial companies are so commercial that all the money is needed to satisfy the shareholders.
BENEDIKT STOCKEBRAND: It's actually worse than that, it's true but worse than that. If you are at a Stock Exchange with a company, there is a huge, huge pressure to get the next quarter report right, no matter what is going to happen two years from now, it's the next. You want to place the company onus ‑‑ owners who only care about the next ten milliseconds before they sell the share again, and that's with things ‑‑ where things tend to get ugly. I had more than average average of my customers being in privately owned companies, where the company owners actually cared about their company. That's just relative ‑‑ not exactly IPv6‑specific but it's a huge problem there. OK. Nobody else? Jen, your turn again.
JEN LINKOVA: Thank you very much.
(Applause)
Now when we just have heard how difficult IPv6 for enterprises, we might hear something probably more positive for ISP. Sander.
SANDER STEFFANN: Hi. DHCP kit. So, this whole story started when I was doing a project at Solcon, Dutch ISP and deploying IPv6 for fibre to the home customer. And basically the set‑up was really simple: Every customer has one line, one fibre, we just want to give them a static/56 per connection provided with the DHCP prefix allocation and identified them by their remote ID which the fibres are provided by KPN and KPN relays them with the idea on it and we just want to see, OK, this customer we give it this prefix, done. Sounds really simple. So, then I started looking for DHCP server but something I ran into is all of the DHCP servers really take the D in DHCP very seriously, they are all dynamic and you create dynamic pools and if you want to do something static and don't want to do anything really dynamic you have to work around it, you have to define pools and then exclude stuff from pools again and assign static stuff just to do some static assignments. And yeah, that was a bit clumsy.
Also, the stability and the maturity wasn't always that good, we had some problems with the DHCP server killing itself when it didn't like the configuration but only when it didn't like the configuration the moment it actually was about to send a reply, it would load to configuration, it would be fine, requests came in, we told give the customer this prefix and it would say say oh, I don't like the prefix and exit and end of DHCP server. This week I found out who wrote that code, so all blame has been placed at the right places.
But ‑‑ I am just kidding. This was a problem, it's so simple, I just want ‑‑ I get a request and send a response. So, when you can't find existing solution you write your own, right? So made it available under OpenSource licence, GPL, presented here at the RIPE meeting and I hope this can be useful for other people as well. I already got a phone call yesterday from an organisation that was like, oh this is cool, can we use this to provision our apatchy Cloud stack platforms? So I am going to work on that in the coming weeks.
So, what were the goals that I used when writing this. I wanted something really flexible that I could customise and adapt to the requirements, easy to use and configure and to be honest in this case the performance wasn't a big issue, we didn't' have tens of thousands of users that would simultaneously start sending these peer requests so currently I can do like 100 transactions per second, so that is like the four full packets. So that was good enough for now.
So, what is the result:
The result is that DHCPv6 library for Python and it's a framework. The basic server handles initiation, open sockets, gets the packet and parses it and hands it off to a handler. So, it really is completely flexible. It just receives the packets, parses them and that's it. So, this way, you can actually build any response you want based on whatever you want. Of course, I already included code for many of the standard options, the serve ID, client ID, you know the requested options, all that stuff I already provided but you can just configure any way you want.
So, like I said, I have two different handlers; ‑‑ message handlers. So I have two different ones, standard handler which implement the basics of the RFC and another one that just receives the packet and dumps it on your standard output so you can see packets coming by for debugging and seeing what is coming in, it was just to play with. If you want something cheatly different it's just a matter of writing a small Python module.
So, what does that standard handler do? Gets the incoming message, wraps it in a bundle and that contains the incoming message, it's where we start building the response. It also does things like split up the relayed messages from the final message, so you can actually see the whole relay chain and parse it, and then it calls a bunch of option handlers which can fill that response. And once you are done, the response is sent to the client.
Now, option handlers are the bits where the work really happens. So, there is a server ID option handler that actually inserts the server ID in the response that checks if a message comes into the multicast address, if it's actually sent to this server or to a different and if it's sent to a different server it will just ignore the request. This is a client ID option but it's like really, kind of low level but with a nice wrapper around it.
There is an options that ‑‑ handles unanswered IA address options, so if after all the handlers have done their work and there is still a request for an address that hasn't been answered it will nicely inform the client that apparently no address is available. Small things like this, it's all really simple small components.
Well, of course, the most interesting ones are the ones where you handle the addresses and I have one simple implementation which hands out static address and prefix based on either the DU ID, the interface ID, the remote ID and can get that data from CSV file, DBM, it looks like this is the idea, give this prefix. So, the whole ISP configuration and provisioning is just a simple CSV file. Other stuff is easy to write as well, we are going to look at integrating with apatchy Cloud stack and can use that as a pack end.
And basically based on this, we are now running this on a live network at a pilot at sol con and currently we use the CSV file from the customer database and just send the process a signal so it reloads the CSV and we are done and that is it. Introduction we are going toit a bit differently, put it in a database because updating and the lookups are a bit faster than looking in a CSV file, but minor implementation difference. And this is working very well. So, with this, you can just have a really simple DHCP server, integrate with your management systems and deploy it.
So how does this look in real life? I will show you some of the configurations. Main configuration I chose a simple file, which interface to listen in, always going to be a relay in front of it, I don't have to listen to the multicast address, I will get it over Unicast. I can set the preference option and specify the name servers and NTP servers, put into the response and the interesting one is CSV based fixed assignment request coming from this prefix, assign addresses from that CSV file, done. And the real configuration is a little bit longer because there is some more options for SNTP, some domain names, search list, but this is ‑‑ this is the core of it.
And the CSV file looks like this, obviously I just changed the IP addresses a little bit but it just says remote idea this, we don't give an address because we run the link completely unnumbered and we just specify the /56 prefix we are giving each customer and this is our whole v6 deployment, just small 80 file, CSV file and we are done. So, what we are going to do now is we are now running a pilot with some customers and going to add more and we are going to update firmwares on modems to enable more devices to send DHCP requests, but yeah, everything is running really well, low load on the server so this is actually already good enough to run in production. But as I said, it's a framework so the intention is that people can take this and integrate it with whatever system they want.
So, then I started using it. And I noticed one thing that I didn't like. I was looking at the log files about what was happening and reading log files is really boring. So what I did is I wrote a new handler that in the first phase just captures the request, in the last phase captures the response, push that is to a process that stores it in a small database aye added a small interface I can see for each client which ‑‑ what is the last request they send and what is the last response we sent back. And if I want to debug significant just search on the customer I ID and can see what their DHCP client is doing. And this was written in one afternoon, a couple of hours. That is why I really like doing this because you can really do whatever you want with it.
So where I am going to take this, keep developing, adding more handlers and tools and I hope this is useful for others as well, for ISPs, for people who want to test CB Es (P) and to keep developing this I asked the S.I. D M to fund this and to support me financially in the further development. It would be nice if they agree, but even if I don't get the funding I will keep supporting this, it just will be a bit slower.
So what am I examining to do? ? (Going) well performance Python code, not up to speed, so there needs to be some improvements there, speed of the parsing and the generating of the packets a little bit, build in some multi processing because Python doesn't really do multi threading well so we can paralyse some stuff, it can always be improved, just as unitesting and test coverage I can currently at something like 47% of the code base is covered by the unitest but I want to get as close to 100 as I can. And of course, I will keep adding more options, more handlers, somebody asked if I could build a radius back end so the DHCP request comes in and the server does a radius request to get the data like Cisco routers can do so this is on the road map. And I am actually hoping that you will have some cool ideas as well so if you have something please speak up this is just ideas I came up with myself.
And finally, some links if you want to look at this. You can just do a ‑‑ install DPCP kit, this documentation and I am really curious what you think of this, do you think this is useful, what would you like to do with it, please give me some feedback.
AUDIENCE SPEAKER: From telecom. We will be interested in stuff like this, we actually are using the internal DHCP v4 and v6 server in our equipment, which is pretty much known to be not reliable for the long‑term. We know that at some point we will need a replacement and stuff like this looks like the good replacement. We are especially interested in the reference part how we do things right now, continue to to things like this as centre reference database which is reducing and get data from the radius and with DHCP, so yes, you have the interest in stuff like this. Thanks.
SANDER STEFFANN: Thank you.
AUDIENCE SPEAKER: I would like to ask you a question because on the very first slide you said all DHCP PD existing servers were not good enough for you or something like that. I would like to ask two things: What existing DHCP DB servers have you tested and have you ever tested for your proposals the KEA because I know that Tom who is the head of the ‑‑ is very keen on implementing anything in a KEA, this basically his child.
SANDER STEFFANN: Fair question. I looked at ISC and at Dibler and wide, at the time I didn't look at ‑‑ when I started this I didn't look at KEA. KEA would work in this setting, so I think I wrote this before I started testing with KEA but that would have worked because it also ‑‑ it's a lot easier for example, than the ISC server to make static assignments and work with them. The most important problem we ran into the DHCP servers keep ‑‑ what we wanted for our customers is they connect one CPE to the fibre, we provide CPE and if you want to give them/56 you swap CPEs and the server refuses to give them the prefix because it still has the lease for the old one and to work around issues like that, was also one of the problems we had. So which is why we just made it really simple in this case. But KEA would have definitely ‑‑ would definitely be useful in this scenario.
AUDIENCE SPEAKER: Just a short comment. I think that ISC, KEA will be a new ISC and Dibler which you also mentioned, it's some other project and I think it's maybe not abandoned but it's mostly not updated or updated in very, very ‑‑ at the lowest possible priority so I think the KEA is the only competition right now.
SANDER STEFFANN: No, it's ‑‑ totally valid, I fully agree with you. This DHCP server just has a kind of different focus, more customisable, people can write their own plug ins, definitely no intention of bashing any of the other ones.
AUDIENCE SPEAKER: I just asked for curiosity. Thank you. That
SANDER STEFFANN: That seems to be it, thank you.
JEN LINKOVA: Thank you.
(Applause)
So surprisingly, we are ahead of time. So now we go into start with lightning talks.
LUUK HENDRIKS: I'm PhD student at the University of Twente in the Netherlands, not in Holland. And we do a lot of security research in our group, network security that is. And we have people focusing on DDoS stuff, which we are already seeing on IPv6. We have people focusing on compromised detection in SSH and http, we will see those things in on IPv6 as well. But there is a whole other category of threads that I think we should worry about. Threats that are based on new things in v6 because I hope we all know v6 is more than just a bunch of big address space we have threats that are based on new specification of this version of IP. So, suddenly this applies, this is not my quote, this was used on I think it was the v6 hackers list a couple of weeks ago, I guess it's older than that use and it applies to well, every field, every search and other things. But it especially applies here because people use this as an excuse to not look at these threats for some reason. And we have seen this in v6 deployment itself. Why should I deploy v6? I don't need it, this is what companies say and if they don't have an ‑‑ monetary incentive to do it then there are very little reasons for most parties, right? And even worse is I have talked to some security companies and they use the same excuse: Yeah, we don't see a lot of threats on v6 so we are busy enough with the v4 stuff, maybe come back in a year or two or 20, yeah, we are not interested. So we are not focusing on this. And if you don't focus on it then you won't see anything because you will need to detect something in order to be aware that something is happening and then spend some resources on it and else you end up in this vicious circle and nothing will ever happen, things will happen but you won't see them. If you check out the literature, academic literature, a lot has been written on these threats in a theoretical way. If you try to find studies that do actual measurements on the amount and the types of threats that we see in a network, in the Internet, those are pretty much non‑existent. If you want to have a conference of ‑‑ comprehensive overview of most of the threats newly introduced in v6 so on the network level, check out this paper. It's really easy and it gives a huge list of all the nasty things that are possible in a v6 scenario, right. So what we do is we take a bunch of these Threats and we want to make them visible on the network and detect them. We do a lot with NetFlow, IP fiction, that is our weapon of choice. A lot of people are have this information already and do stuff with NetFlow, accounting or monitoring or whatever and if we want to come up with an academic solution and you want to apply it in the real world you should be aware of what people use in the real world. So NetFlow seems like a good, a deployable and also a scaleable approach, so what do we need?
First, what we don't want, we don't want to do so much intensive stuff on our flow exporters that they don't have time any more to do actual flow exporting, that is a requirement. We tonight them to abuse them to do our, you know, actual IDS work or whatever, save that stuff for your collector.
So, if you then look at specifications, and specifically the IP FIX elements standardised by IANA, well most of the stuff we need is actually there, so, for example, one of the threats in this paper that I mentioned is abusing the flow label, the IPv6 flow table for cover channel stuff, you have 20 bits in the flow label, you can put whatever you want in there and people won't notice it's there. So, cover channel. So you start reading the list, OK, ID 31, flow label, IPv6. I guess that is it. That is the piece of information that I will need to do my analysis and to possibly detect cover channels based on this flow label. OK, sounds good. Using the traffic class field also a possible cover channel thing, then there is a lot of new stuff around v6 like ICMP 6, can do nasty stuff with that, we want some information about ICMP types and codes, 6 in this case. Well, if you continue reading the list and yes, it's all there. It's a good first step in research, right. You are able to get the information that you need, well, that is very convenient.
So how many of these fields were actually exported by our dedicated flow probe? What do you think? Who thinks it's three out of three? Yeah. And exactly, it's zero. Or maybe 0.5, because the ICMP six information was exported but in the ICMP 4 field. That is why we have standards. Then, I contacted the vendor, and said well you guys do things with flows, right? Yeah, yeah, yeah. IPv6 flow label, care to explain why you are not exporting that field? It's literally, you know, your core business. Well, we never had such a request. So yeah, can I blame the guy? I don't know. It won't help me anyway. But luckily, for our hardware that we use we can write plug‑ins and we can realise exporting some stuff that is not exported by default. So, great.
IPv6 was basically made for this so you can define your own information elements which you can then export and use on your collector. I only have a couple of minutes so I don't really have time to explain IPFIX stuff here. I hope people are familiar with. IPFIX 101 ‑‑ 401 packets go in and aggregated flow record comes out saying there are so many packets, some many bytes from point A to B, so source and destination point.
Started ‑‑ came up with some things that are related to a different type of attack so not flow label cover channel but a huge new thing, a different thing in v6, right. For example, the minimum offset, we might have heard of this attack where you rewrite parts in fragmented traffic so you overlap in the first packet the information on destination port, for example. So you start sending a packet to port 80 and overlap rewrite it going to port 22. Nasty stuff, right? No way to detect this using the default flow data that you get.
So right now, we are generating and testing this with attacks we generate ourselves, but we want to deploy this at big networks that have been v6 enabled for years and years, so it's typically academic networks and ‑‑ and then in the long run, I want to say something about how are we doing in terms of v6 security? How safe are we? What nasty stuff is going on?
So, that is basically why I am here. Did you look at the quality of the stuff that comes out of your exporter, I am talking about v6 stuff. Is stuff actually exported and is it valid what comes out of there. Or is it bogus? Because I have seen happening that, too. Which elements would you like to see standardised, if any? And what other uses of information elements do you see? Because if you want this to be adopted by ‑‑ if you want this to be adopted by companies, by institutions, you will need to sell it any way, because security, they are not interested in that, we have seen that. I am here all week, please go into this discussion with me, otherwise find me on line. Let me know what you think. Do we have minutes left for questions? That was included in the ‑‑
JEN LINKOVA: We are ahead of time a bit, so we do have time for questions, please.
AUDIENCE SPEAKER: A question. Did you also have a look in the IPv6 headers that you can have the packets, certain switches were limited by ACLs to look at 3, 4, headers max in the ACLs, and are there actually in your research already the EC attacks, people using those with regular info and adding more headers after that to see the info that they want to get into?
LUUK HENDRIKS: We didn't look at this yet. This is certainly on our to do list. You can do so many nasty stuff with the headers, it's almost fun. And actually this recommendation stuff that we implemented, was a first step into the direction, right? Now it's quite simple so we only look at, you know, if the first header is fragmentation, so if it's protocol 44 we do our logic. So we still have to look into this but this is certainly on the agenda, yes.
AUDIENCE SPEAKER: Cool, it would be very interested in reading about that because I think it's coming up and if it's not now, it will be tomorrow.
LUUK HENDRIKS: Yes, definitely, thanks. Lip lip on this actually, comment, a ‑‑ playing devices around could not look into header after the first one so they could tell you there is IPv6, there is a next protocol, Layer 4 protocol or extension header and until recently they just were not able to tell you what is behind the first additional header so it's actually ‑‑ it's getting better and happy that couple of RIPEs ago about the measurements so for short headers you still have good chance to get your packets through if you are adding more than one header or making the single one quite long you probably have pretty good chances of your packets being dropped on the path. Any other questions?
AUDIENCE SPEAKER: So this is probably completely out of form so I will apologise to you and to both. I was going to make a follow‑up to Stefan's, I am from ISC so when KEA was mentioned I wanted to share that I am here and the KEA team will be very excited to follow up with Stefan and the gentleman who came and talked about KEA, if you find me and I can share information with you, contact and all that, I enjoyed your talk too, I had to put in this plug before this session ends. Thank you.
JEN LINKOVA: Thank you. OK. Now, we are actually have two lightning talks about best current operational practices so those who attended both on Monday could go back to reading your e‑mails. My pleasure. For next ten minutes. I will wake you up when finished. So, Sander, please.
SANDER STEFFANN: Me again. A short lightning talk about what we did in the BCOP task force on Monday. And I am actually presenting this here because we want involvement from this Working Group.
So, you have just seen Benedikt's presentation about IPv6 and enterprises and enterprises have many questions about IPv6 in all different forms and about a lot of different subjects. And it's difficult for them to find answers. Sometimes good documentation doesn't exist or if it exists, and you don't know what you are looking for it's often hard to find. So I think the thing enterprise need most is guidance. So, I was working with Jan Zorz on this and we started gathering documentation on IPv6 guide .net, a random domain name that I happened to have lying around and what we did is started writing it like a story, like a martive, so somebody can go this and start reading on high levels how you should work on IPv6, you should prepare, build a lab, do your implementation, think about your monitoring but on a very high level. And from there link to pages like, if you talk about building a lab, have a separate page that explains how you can build a lab and what you can do and/or if you mention somewhere you should make sure your equipment is compatible, how do you do that? We have the RIPE 554 document here. And that is really important. I want to make sure that there is an overview of what is available and fill in the missing bits. But definitely not rewrite everything or make one new text about everything. I think it's important that to provide a place where this is a narrative that somebody can read and then point them in the right direction and help them to start actually deploying v6.
Just some sheen shots of what it looks like currently. Very simple mark down based thingy. And I already started working a bit on this. I am already writing some of the overview, the glue text. I am going to write on announcing IPv6 space, there are some organisations, enterprises or governments that have huge blocks of v6 and are actually considering just deaggregating some stuff to a /48. If you have a huge block that might not be very appreciated by the community so I think we need to give enterprise some guidance on how to deal with this. Routing, deaggregation, as I said, these are the subjects that I am personally going to work on because I think they are important but there is much more to be done.
So, I think this can provide guidance to the BCOP task force and where we need D cop documentation, once we start writing this martive, we will spot holes in what is available and what is not available. So help is really appreciated in building this up and finding what needs to be ton. So I propose this as an action item to the BCOP task force and as it is related to v6, the deal we have between the BCOP task force and the other Working Groups is that we involve all Working Groups that touch the subject. So of course we wouldn't to this without coming here to the IPv6 Working Group and asking for your input and your guidance as well. So mostly technical part, check the advice that has been given, especially if we are going to write some new text so I would ‑‑ I would like to invite to you participate in this to make sure we have a nice overview and give enterprises some guidance that they need to understand v6 and understand what they need to look at and point them in the right direction to RFCs, RIPE documents, whatever they need to get the job done.
Any questions on this?
ROB EVANS: Just a quick observation like 532 consists which is the RIPE Routing Working Group on route application.
SANDER STEFFANN: Yes. We will definitely use that one.
AUDIENCE SPEAKER: Well, with mixed hats on my head, Jan Zorz here, so we are ‑‑ we started working on this in bit different manner that we have been working on other BCOP documents or RIPE documents when we have been sending the drafts to mailing lists and asking community for the feed pack. This one is in a forum of like a web page so how do we ask the community for the feedback? How do we work on this? Can we sort this out?
SANDER STEFFANN: Good question. I would suggest to start with, the BCOP task force mailing list to discuss the content. And I was already thinking about building something into the website where people can actually add notes to different parts of the text so we can keep, like a more inband mechanism for keeping track of what people think of it. I am still looking at that but these are the two things I currently have in my head.
JEN LINKOVA: I actually, because we have had some very positive experience on working on help desk document between both BCOP and v6 Working Group, probably it would make sense to discuss v6 related technical aspects in v6 Working Group and keeping BCOP mailing list, cc or include it so basically I would suggest not to exclude v6 Working Group from the technical part of the discussion.
SANDER STEFFANN: Definitely not, that is why I am here.
JEN LINKOVA: And second comment: Are you considering something like Git hub, any other way to have kind of versions and related to send a change which will be viewed by collaborators and approved and submitted?
SANDER STEFFANN: Not a bad idea, I will look at that.
JEN LINKOVA: Probably I might just announce or you can send a mail to the Working Group mailing list so we can continue with a volunteers on the ‑‑
BENEDIKT STOCKEBRAND: I have done the odd video for video block which is obviously a completely different kind of medium I am using there and it's not exactly nice thing to have another version and change things. Would that be any use for you?
SANDER STEFFANN: Yeah, sure. All content that is useful to enterprises, is very welcome.
BENEDIKT STOCKEBRAND: I have ideas to do some more stuff once I find a bit more time but I will keep you posted on those, then.
SANDER STEFFANN: Thank you very much.
JEN LINKOVA: OK. I will try to be quick, providing probably some people have seen it on BCOP meeting. So, we have got dual stacks networks for a while, we have some BCOP documents for this, probably it's time to think how can we deprecate IPv4 and start building v6 only networks, we have v6 only networks, other conferences around v6 only networks for a while, so probably does make sense to start documenting current operational practising and choosing the best to make sure we are doing it right from day one and we do not have to work with bad habits later. So, it's how it started here, here is some small group of people that sit together and wrote a kind of draft skeleton of document to document how you build what we call eyeball network, I mean not network infrastructure, not data centre, it's kind of question because we were thinking initially about conferences network but probably access ISP might be included or excluded from the scope, we don't know yet. And so far we have draft which covers different stuff like configuration, scalability, first hop, and again we are talking about two‑minute cases, building kind of uncontrolled hot spot conference network or enterprise network when network administrator might want more control over the devices and who is doing what. No issues, troubleshooting, actually v6 only network at RIPE I help a lot with finding out what is going to be broken on such network. So, actually I am here because I need help. I could not do it alone because otherwise it will be my operational practices, I am not sure it's going to be best operational practices. So please if you have any experience in building v6 only networks I would love to hear from you and you don't need to commit to writing anything, you can just share the practices and we can document them. And it's it for me. So, any comments, volunteers or anything to the mic phone? No, as I expected. OK. Don't be surprised, then. Anyway. So, now, I think we going to the most interesting part of the session. So, again, I hope at least some of you in v6 only NAT 64 in this conference. I do know you have been using it Colin provided me some statistics so I can tell between 10 and 15% of devices connected to this network so I think is great right. So people are trying it. And if you look at the history, so it's been happening for a while, until this meeting it was pure experimental network, right, it had dedicated special SSID and special password I know, best effort. No known support. Thanks to RIPE NCC this time we are slightly moving to the standard configuration, it has a standard SSID name and password and fliers saying no support on one side of the flier and supported best effort on the other side of the flier. And so about, what I say, 10, 15% of the users, we received 9 ‑‑ it was eight when I was making this slides ‑‑ 9 currently, 9 issues reported to the mailing list and 3 or 4 of them I fixed in the newest version of software being used or it might be fixed in the configuration with open VPN for example, you have to configure server to accept connections over v6 so about two thirds of the issues only could not be immediately fixed without Menno intervention. The question is, we have been into pilot stage for a while. What is our criteria for exiting the pilot? What would community like to do with it? What RIPE NCC would like to do with it. So now, Marco.
MARCO HOGEWONING: I didn't bring any slides. I did send a apologies for the lengthy e‑mail to the mailing list that outlined the position we have in RIPE NCC. This, for us, is basically the decision that happens in any business enterprise or ISP that has to deploy IPv6 and that is one where we have to do our risk assessment and find budget. I would like to take that point and split it up because one of the things is that the set‑up we currently run, we are pretty confident is not suitable to support the thousand or so devices we need at RIPE meeting. In order to to that we need to do more research, we need to invest in probably ‑‑ probably invest in different hardware or more hardware, that will take staff time, that is the part of the discussion and decision we have to make and hopefully our membership will support us in making those decisions and allocating budget for that.
And in that sense it's just a basic business decision, how far staff time do we need and we haven't really figured out what our estimate is, we will work on that and bring it forward to the community.
So, the other part of that, like I said, is harder, that basically to use a stupid marketing term comes down to goodwill. A lot of people come to this meeting in the expectation that they can do their work, a lot of people can only come to this meeting if they still can do some work remotely, and they can keep check with their offices. Over the years we have invested heavily in keeping the network to the quality that it currently has and do I a lot of conferences and not to brag but this is really one of the best conference networks one can get. We have invested a lot of time and money into that, our tech team are doing a really good job with that. At the same time, it's still a really fragile set‑up, we have to build this up in pretty much 24 hours, role a ‑‑ build a network that supports 1,000 clients. We have to be careful in making changes we don't lose what we got, and that is for us, the key criteria in rolling further. After we have built it up and scaled it up to the point where we are confident in saying we can support thousand users and we have the resilience we want into the network, the end decision for us is will this keep our customers and you are in that sense our customers, happy. And for that, we really have to be sure that nobody bumps into problems and as you have seen on the mailing list there is still small nits, open VPN doesn't work, F 5, couple of RFC clients that still struggle with this. For us, that is really important that those will be resolved and we are ‑‑ before we are confident to take that step, so that's our current position is A, we need to work on the budget and allocate the time, that will take time, probably throughout next year, we hope to get something going. Investments, unfortunately the budget has already closed so it might be hard to do this under the 2016 budget, even. Once it's there, we hope at that at that point the community and everybody who is listening in and doing their job will make sure that the hardware and software that is then current is sufficiently supporting IPv6‑only and that we are confident to make that swap and we don't end up with any of you or any of your participants, fellow participants that are not in the room, making happy and I think that is very important because you are the ‑‑ the you are not all of the RIPE meeting. There are many more people out there doing their work and we are also making the decision for them and it's really important to make sure that they also get catered for. So, that is basically our position, I ‑‑ add it in the mailing list and I am happy to take any questions or comments from you.
JEN LINKOVA: Let me thank you guys for doing this because it's great, it proves that you can enable v6 only income and the sky doesn't fall. So thank you very much for this, yeah, I hope we can continue at getting better and better. Thank you, RIPE NCC.
(Applause)
JAN ZORZ: Thank you. I would really like to see if it's possible these eight issues reported, like a detail, what was actually the issue, because if that eight issues includes like exploding computers then we don't know what is next, right? We need to solve these things and I would really like to see these issues documented and how were they solved if they were solved because this is basically influencing on the decision on what is next.
JEN LINKOVA: Let me answer this: So, currently I am collecting the information, right, and I was pointed out to this nice Wiki page which lists all unknown IPv6 only issues and I hope we can just update this, because as I mentioned some of those is just fixing recent software issues, need instigation and some are known software bugs and developers just haven't fixed that yet, it's been opening for a long time and apparently they do not pay attention.
JAN ZORZ: This would be also very useful also for outside this community for vendors to fix it in terms ‑‑
JEN LINKOVA: I think updating is a reasonable approach and then just let people point into the Wiki page.
MARCO HOGEWONING: For us an issue is an issue no matter whether your computer explodes or a kitten dies but in particular the two that really raised the red flag for me were VPN issues because security people are notoriously to be conservative so if any issues with VPNs occur that is the hardest to fix especially if that would involve rolling out new software versions because convincing to role software version that is a different take.
JEN LINKOVA: Because it was my issue, like two RIPEs ago and I just asked my VPN people so probably as we have just seen it's it's a nice excuse, nobody asked us to do so some of the v6 issue could be resolved if you asked kindly.
SANDER STEFFANN: I just wanted to say thank you for the work you put into this, I know I have been a little blunt on the mailing list, more than I intended, but I really like ‑‑ I mean, I understand we can't just change this in one go but I really appreciate we are taking the steps to get there, so thank you very much.
BENEDIKT STOCKEBRAND: I have seen the discussion in the mailing list aye understand your concerns, Marco, but there are two things: First off, if this IPv6 only network had been established like 15 years ago, just to really push it to the extreme, degree in theoretical computer science, that is the way to do things there ‑‑ we would have avoided a number of problems because we would have known by now much more about the possible problems we have so just trying to delay things doesn't make them any better; it just gives you a chance to leave that job before they blow up in your face. That said, I would be reasonably content with that every single RIPE meeting we take some kind of step forward, something that you are comfortable with because after all it's your responsibility, what I don't want to see that we at some point kind of stall this approach because that will make it only worse much afterwards. I think we will have these discussions going on for a couple more years, and you will be nudged towards getting this done over and over again, just one thing: Don't really ‑‑ really, don't take it personal, that is just the way this sort of stuff is going to happen. OK.
MARCO HOGEWONING: We will see what goes on and I am happy to work with the to see what can be done as the next step, but as I said, we are also working under some constraints regarding staff time that we have to take into account before we can move anywhere further.
JEN LINKOVA: Yes, so my question is basically, could community do anything right thousand to help you or we just take it ‑‑ you guys need time, we keep it as it is for a while and we can await you to come back at some point and tell us what the next step?
MARCO HOGEWONING: I think especially in keeping report the problems, report when they are resolved. I know we have seen issues in the past with Android that now seem to be resolved and there are some other things that gradually move forward and not only provide us with a list of problems but also preferably with a list of solutions and stuff that is fixed bus that would really help in assess where we are in terms of readiness to deploy.
JEN LINKOVA: I actually would like to ask people what to they think might help with it because from my perspective, I say put this on the slide, the only thing missing is basically this, lack of support which might actually scare people when they read it on the flier that no support provided, they might just not use it. I don't know if anything else is missing, how can we make people using it and reporting it or we think we are good and as long as this network is advertised object on the slides and the flier is there and we are good and we leave it as it is? Any opinions on this?
AUDIENCE SPEAKER: ‑‑ even Android says no IP address configured when connected but how people would know it exists, I guess you can have that in bold on the badge, you know, they can give you a little maybe ticket like for T‑shirt that this network exists, physical things work best, that is my opinion and maybe that card would be turned into feedback form, physical items, not on line items.
MARCO HOGEWONING: I don't know where you are going. We have provided fliers at this meeting, we have the physical ‑‑
AUDIENCE SPEAKER: Just more awareness of that service if you think somebody else don't know it exists, what else do you mean, what other avenues?
JEN LINKOVA: So we have those fliers so I am just curious if people ‑‑ I probably have one suggestion on the slides when it was linked to technical documentation, it was kind of long, there is no way can basically remember it and type it so ‑‑
MARCO HOGEWONING: We will take it into account for the next meeting and provide a short URL.
SHANE KERR: So thank you for providing the name change. I think that for the next meeting, having it be an officially supported network doesn't seem like a stretch, we could bump that 10% to 20% in our typical IPv6 way of doubling every six months. Yeah, I mean, to be honest, I found the fliers a bit snarky and
SHANE KERR: We don't use, this use it as your own responsibility. This is not somebody is bring ‑‑ to bring before international tribunal for if that doesn't work. This was yesterday's presentation. I would like to propose that would be the next, officially support it, no new hardware, exact same exit strategy as we do today of falling back to the non‑NAT 64 and that seems quite possible. I am making a speech, I apologise. We will certainly take your comments back and consider those. Two‑year point is 20%, I don't know where the current limits are on the current system but just ‑‑ that is risk that this becomes too popular in the current state.
JEN LINKOVA: I would love to hear about v6 becoming too popular.
AUDIENCE SPEAKER: Martin speaking from my own experience, I won't blame anybody who says it's not supported so thanks for, again and again, proposing this, but I believe that maybe many of us don't have time to troubleshoot their own laptop and they don't want to waste time and they would like to catch up with the real world which is what is happening here and B, also, on line with their VPN mail blah‑blah‑blah. Maybe the idea for what is next, next time if it is to be reconducted, maybe the easiest way would be to propose survey with checking box so is people who have all green, great, congratulations. People who can't for whatever reason, they might maybe check reasonable number of boxes and maybe else others ‑‑ I think that recurring patterns would help people who deploy to ‑‑ with MAC OS, that version, that VPN version blah‑blah‑blah there is problem, it's recurring. Maybe there is something to fix. Don't rely on experts, I mean, there are many experts but maybe there are people who would like to taste, test but not to spend too much time but again congratulations for bringing that because we would like it to be production‑supported but that's it. I won't blame anybody again.
MARCO HOGEWONING: Thank you, and to that comment and other comments about like RIPE NCC should consider supporting that we really need the community to document this, document the issues, document the work around, please feed that into the Working Group discussion. And feeding that into Jen's BCOP document because at some point we have to have that list, we need to have a frequently asked questions list on this before we can support this and I would very much ask the community to prepare that for us.
JEN LINKOVA: Perfect timing. 3:30. So thanks again, I think we have made a great step forward.
MARCO HOGEWONING: Thank you.
(Applause)
JEN LINKOVA: Please keep using it. And two last announcements. Please rate the talks, and please give us feedback in any forum because I am going to ‑‑ I am going to work with my co‑chairs and start building agenda for the next meeting and I would like to know what shall we discuss on the next meeting.
And the really last announcement, if you are registered for general meeting, please go to desk now to pick up the stickers you need to enter the meeting. And this concludes the first session of IPv6 Working Group. Thank you for being here, thank you for participating. Please come back tomorrow.
