JEN LINKOVA: Okay, so we start. Good afternoon. Welcome to IPv6 Working Group. The DNS Working Group is in another room but I'm glad to see people interested in IPv6 too.
Agenda for today's session on the side. Anyone has any comments, indicate.
After we finish this session, please rate the talk. Because we really would know what you like and what you don't like. And I think we should ‑‑ I just remind you once again, we have this code of conduct, so please be kind to each other. Really, it really helps, actually, to have discussions. And again I said it yesterday, I'm going to say it again today, if you keep reading your e‑mails please use IPv6 only network, it's IPv6 Working Group after all. Great.
The first presenter of today is Geoff on IP performance measurements.
GEOFF HUSTON: I am Geoff, I am with APNIC, I'd like to talk this afternoon about IPv6 and performance. Now, what I really want to talk about is not IPv6 performance. What I want to talk about is the relatively between v4 and IPv6. So, I'm kind of going, I don't really care about TCP. I don't really care about jitter, I don't really care about a lot of things that conventionally performance is on about. What I want to look at is the hypothetical case that if you're trying to do something to a server, and you have got v4 and IPv6 at the same time as almost alternatives, can we say anything about the difference between those two protocols if we try to use that many in that way?
So there are actually two questions that I can figure out that are relevant at the IP level so I'm just talking about the IP level, what's going on where a change of protocol at the IP level might change your experience.
The first thing is actually reliability. Do connections actually work? Because, we certainly heard that there is a whole bunch of middle aware, there is a whole bunch of stuff out there that plays havoc and if it's hold stuff and it's IPv6 unaware it might break it. Who knows? If there is a whole bunch of stuff out there inspecting packets because it's good for you to have to have your packets inspected by the state, but because it's Government work they don't build it for capacity, just against a budget. So everything goes and nothing works because it wasn't actually built to handle gigabits. So v4 might be busted. So first issue is: Do connections work and is one protocol worse than the other?
And secondly, it's not how fast connections are, it's relativity. It's is 6 faster than 4 or not? Same end points. Same end points. Is the choice of protocol going to make any difference? So will TCP connection attempts actually work? And to be perfectly blunt, the question you all want to know is is v6 slower than v4? Fast indeed. You hope...
So, the way I did this measurement is to look at the world like you from the edge looking inward. So I'm not trying to mention the infrastructure. I'm not trying to look at routers, I'm not the ISP looking out to the customer. This is you, the user looking back in towards the infrastructure.
So, we embed a script in an online ad and the ad just generates URLs to fetch. It's like you doing it yourself, we do around 10 million a day these days, and what we do is rather than just pinging the world in general, all of these folk come and actually go to a small number of servers that we are instrumented like crazy. So, we look at the results at the server level to determine what's going on.
So, this is the measurement count since 2011, since we have been doing this for an awfully long time, thank you Google and they have been very good in supporting us. And for a while they were doing a few money thousands measurements a day all over the globe. This year we decided to get serious and Google has decide to get serious and over the last few days we actually had 10 million measurements per day from all over the planet, which is stressing out a whole bunch of equipment. That's what it looks like on a linear scale. This is on a log scale. You see the amount of IPv6 that's actually been generated through there. The log scale it's things a bit trickier, but the amount of v6 is increasing as compared to the total amount of work there. There is a certain amount of v6 capable machinery.
So, the first thing to look at, reliability. If you make a connection attempt, does it work? Now, looking for something that isn't there is really bloody difficult. Right. Close to impossible. However, TCP can give you a kind of half indication of failure. The three‑way handshake is brilliant. Because, you actually need a packet in, a packet out. And a confirming packet back in again. And what we're looking for is that asymmetric case where I can see you, cannot see me so you can SYN my server into the SYN, I'll try and send you a synac, it's not going to get there maybe, I never see the closing ACK. So I'm looking for the amount of connection that is break in this way. Did this four years ago, 2011, and at the time ‑‑ now you guys have a short attention span, so this is difficult. But you know, you might remember a thing called Teredo, really popular then, you might also remember 6to4, everyone was doing it. And the second data set I want to look at it the dataset that goes up to a couple of days ago, in fact last night. Really late. But I have some missing v4 data. I want to compare these two datasets. This is the first one in 2011 and to say that v6 was shit house was quite complimentary. That green line shows that connection failure of 40%. 4 out of ten. If you ran up a v6 service all you're doing is despising your customers because that's just crap.
It's really bad number. And the real question is why was that so high? V4 is sort as you'd expect, apart from a couple of spikes. Sort of what's going on? Area there spikes? If you blow up the v4 curve what you see is it went down over time. So, the absolute failure rate was pretty constant. Now, I heard someone, who every, what, five minutes wants to scan the entire v4 space and ping it, and if he is doing it over say let's port 80 using SYNs for example, I get to see him again and again and again and again and again. And it's not you personally, it's your mates. And that's what I think I'm seeing in this kind of failure rate, that it's not really bots, it's not really anything bad. It's just a bunch of demented researchers doing all the same thing. So, I suspect that what I'm seeing in v4 is just a bunch of under grads and post grads and everything else and not much else. The spikes, that really was a DOS attack. Nothing much.
So, what about 6? Here is a clue. The spikes were when my local Teredo relay failed because I was running Maredo. I was looking at this thinking if I break this up, all 2001 addresses are Teredo. All 2002 addresses are 6to4. All the rest are Unicast. All the red down the bottom is when you don't tunnel. Everything else is crap. Tunnels are evil. Say that again with the word lisp and nothing changes. Tunnels are evil. Couldn't resist. That's how bad it got at the time. That was 2011.
So, 6to4 is indeed an invention of the devil. Most of the time it's your own fault, you block it and nothing works, this is terrible. 6to4 relay servers breakdown. 6to4 is shit.
Unicast was a little better. When I say 5.3% failure rate. 5.3% is crap. If you built the network that dropped 5.3% of all connections would you still have a job tomorrow? You wouldn't. What was going on? Why all the failures at the time? I found 7 folk using FE 80 addresses, I'm like hell, local, global, who cares? I found someone using ULA source addresses. Again, what the hell. Depricated site locals, well deprecation is merely advice it's not an instruction. 16 folk used something that I just just do not understand. The good news is that 3 FFE had finally died. That's about the best thing one could say about the profile at the time. We fast forward to now, this is across January to November of this year and I found 24 million unique v6 end points. Success, the failure rate, well it's still pretty shit. 4.1%. Sad face.
This is the daily failures, there is one anominally in the daily failure rate which is over there on September 1. Google announced on September 1 that Flash was to sell abysmally awful that you could no longer do ads in Flash, which meant me. Oops. So we had to turn that off on that day, and we were doing then exclusively HTML 5, that gets mobiles and other devices like Apple that despise Flash, and interestingly the new device set works best than the old device collection. And we're getting if you will, a better class of failure, i.e. lower failure rates as a result. That's kind of good news.
6to4, God... tunnels are shit, right. They are shit. So one quarter of you just haven't got this yet. Tunnels are crap. Stop using them. 27% of all the v6 we found came in over 6to4. You're nuts. And so the 6to4 folk, they are still getting a 9% failure rate. That's the solid blue line. So you know Teredo ‑‑ 6to4 is crap, stop using it. The rest of are you actually not bad. Instead of sort of 4 or 5% that red line at the bottom is hovering below 2% these days. That's better. Not brilliant. It won't give you full marks but you know 4 out of 10 better. That's pretty good. So that's the kind of slide that summarises it all. Unicast failure rate 2%. Of course not every failures occurs all over the world. This is the worst people in the world list.
Sent Ross, somewhere in Mexico, you are hosed. 99.6% of all connection attempts didn't work.
.down here somewhere is Teradon running Sky Mesh, George Michaelson is in the other room ‑‑ no, he is down there ‑‑ he is the customer, his failure rate is still astonishing 30%. Call them up mate and fix it. If you are on this list you should be doing something about it because this is unseparably bad. So, badness occurs in a lot of places.
I thought if I was going to show the bad people I should actually show the good people. Now these are the folk that not only never failed, but I got the highest number of samples. So, I got the sort of huge sample count. And what amazes me, and it is a cone dense and I did not manufacture this, I promise, is that number 1 is a company call Voxility right here, who, you know out of 3100 connection attempts, they all worked. Wherever you may be.
And down the bottom, 608 also from Romania, Roid.net which is I think their academic network, again a perfect score and the folk in between, it's basically ranked by the number of samples they all did well. So it is possible to do it right. It is possible. It's just these folk haven't figured it out yet.
So, you know...
279,000 addresses, 143,000 with 6to4. 118 addresses were still Teredo. Lunacy never dies. There is a dim dark corner over there saying I can run Teredo. They are all Hurricane Electric, someone still hasn't figured out between local and global scope, 92 of you. 709 just can't type. You are using someone else's address. It wasn't allocated. Another 1358 are using an address that isn't in the routing system. Get a clue. And of the rest 133,000 should have worked and there are actually 102,000 unique /64s. So there might be a bit of privacy addresses. But basically that's why the error is there. It's a certain amount of typo work, a certain amount of I have no clue what's going on and a certain amount of deliberate ignorance about local firewalls.
What about v4? The v4 the failure rate was 0.2%. Twin, 300 million v4 end points. The Google is just amazing. The failure rate is actually a little bit higher at 0.3%. 1.1 million failures.
I lost some data through September to October, which is why I was working on these slides right up until last night. But, you know, what you actually saw was that the failure rate drops over the year. So, I don't know why it sort of rose from 2011 to 2015. But you know, continuous improvement objectives were met through the year, we actually managed to half the failure rate across the year for v4. That's kind of cool. That's the comparison, v4 is still what you're aiming at. V6 the red line is still where you are at. Nowhere near good enough. That's the real comparison you should worry B you're nine times worse than v4, make it better to work children, to work.
So, yeah, right, underline that. You know it.
So the next thing, speed: Because it's all about speed, isn't it. How fast. And the real question: Are you slower or faster than v4? Now the thing about this is, I'm going to dive into SYNs right now. Now, what we actually have here are looking at the profile of v4 and v6, so every session starts with this SYN, right, and pretty typically your application is not driving this. You just simply go operating system open, do something, and the kernel does all the work for you. So there's very little in the way of operating system to make life slower. So, what I'm doing is deliberately just measuring the time taken for the SYN to complete. So these aren't the broken be ones, these are the good ones. Now, this is really good. Because when the SYN completes, guess what it does? It asks me for something. And as soon as it asks me for something, I know who you are, I know what you're asking for, I know your operating system, I know a lot about you but I also can figure out the pairing between your v4 and v6 address. So, now I have got sort of both sides of the dual stack persona. Now, I have only got a single RTT, but with LAC luck I can get the RTT for your v6 dual stack persona and the RTT for your v4 dual stack persona because the test that Google ad delivers to you is a v4 URL, a v6 only URL, and a dual stack URL, so if you complete all those tasks, I have got you, I know what you are in both protocols. So now I can actually do the coupling.
I know both sides. I can compare your RTTs. Because now, same end points, different protocol. So, let's put it into buckets and plot it. So this is ‑‑ astonishing large ‑‑ large numbers of billions of sample points. Now, this is 2012. The blue was Unicast, and the buckets were in hundreds of a second. This was one day, most of the blue was in the middle. 6to4 is green. A lot of that is over on the v6 is slower side because tunnelling is evil and it's slow and 6to4 is crap. And Teredo is right out there as being as slow as hell because Teredo is really, really, really crap. So this is not surprising. If it was anything else we'd all be wondering what reality we're living in.
Fast forward to now. All of November. V6 is faster, v6 is slower. That's surprisingly symmetric, now it is a log scale, so be aware down the bottom it's kind of noise levels but predominantly up in the 1 million point it's almost the same number. Almost the same number. And just to prove that tunnels are really shit. There is the 6to4 distribution. V6 is slower because 6to4 is just stupid. What can I say? Don't run it.
I can do a little bit of fancy graph work to overlay the two. But I think this one is actually the killer. This is the cumulative distribution. And what you see with the green line is that it hits the zero point at around 18 to 20%. So around a fifth of the time 6to4 is not too bad compared to the equivalent 4. For 4/5 of the time v6 is slower in 6to4. But v6 is surprisingly good. It's kind of down the middle line. It's not faster. It's not slower. So, that's using a 10 millisecond resolution on the bucket. These are just computers, you can do what you want so how about if I use a tenth of a millisecond and I actually up the magnification, so I'm looking right at that middle point. 48% of the time v6 is faster, 52% of the time v6 is slower. It's really marginal, it's really marginal. And quite frankly, between the point 2 marker and the 0.8 marker you are within a 100th of the a second in RTT. For 80% of the time what I'm seeing is the two protocols are zero point the same. They are equally fast. So that's the kind of good news. Is it as fast as v4? The answer is yes. 70% of those cases you are within 10 milliseconds, absolutely. So their performance in RTT terms just fine, as long as you're not tunnelling. If you are tunnelling, God help you.
Is it as robust? No. V4 sets a benchmark, you really should be able to get perfect connection attempts. Even 0.2%. I am blaming all the researchers in the world. I think the protocol performance is better. V6 should be as good as that and quite frankly that's what we need. We need that level of performance because the product managers go oh yes it's not as good is it? They are right it's not as good, it needs to be better. So 1.8% where it's currently at. Okay. It could be better. It could be a whole lot better. And that's your job.
So, yeah, if you can get the connection, that's fine. But the odds of getting that connection are basically nine times higher or at least the odds of losing it are nine times higher in 6 than in 4. That's the problem. So that's it. Any questions? Thank you.
(Applause)
AUDIENCE SPEAKER: Robert Kisteleki speaking as an individual. Not Jabber question. Two things. One is maybe it's also just in the noise, but do you have an idea if overlaying that works such as Tor have any impact on this? First of all, I don't know even know if they do v6 or not and what you would measure if they did. Or if people are using it. So I can only imagine that would confuse your dataset a bit.
GEOFF HUSTON: The way do I this work is that I get this v6 address and this v4 address and I use a tool pretty much like a RIPE Stat tool that says look up the BGP on the day, consult with origin AS announced this and then do some assumptions about countries etc. As a result. I have no idea if A) you get ads in Tor, I have no idea, and B) if an ad ran what it would look like. So, it may be there, it may not. I have no idea what the signal would look like.
AUDIENCE SPEAKER: The general recommendation is that if you are using Tor then you don't do any ad blocking or stuff because you could be identified based on the number of plug‑ins or stuff like that. I think you do see ads. I have no idea what the impact is, that's why I'm asking. It might be interesting to look at how many exit nodes you see.
The other question I had was about the amazingly bad Mexican provider on v6. So, they have like 0.31% success rate. I wonder how that relates to their success rate in v4 space because it might turn out that they're a virtual provider, you think they provide something but basically they don't.
GEOFF HUSTON: That's a really good question. I was just after the laughs on that. I just wanted the top worst v6. What the hell...
AUDIENCE SPEAKER: I wonder if they are on in v4 space.
GEOFF HUSTON: Good question.
AUDIENCE SPEAKER: Benedikt Stockebrand, speaking for myself. Do you have any why v6 is still worse than v4 when it comes to reliability? Any indications on what the causes could be?
GEOFF HUSTON: My suspicion is still that there are firewalls and filters out there, particularly in the corporate world, that block incoming but let outgoing go out. And so what we're actually seeing is, you kind of ‑‑ because the entire ad is delivered in 4, and almost everything works, except that nasty v6 only object you have to fetch. So, you kind of do the DNS dance, you get back a AAAA record and you go, okay, let's try and get, it you send me a connection attachment, that's your first v6 packet that leaves your network, I try and respond, your so helpful firewall says that's just so bad and evil I will drop it on the floor, and because you're running dual stack, and because I'm one of the few loonies who offers v6 only, I am the only error you had all day. Everything else just works. So, it's not an obvious error that folk actually trip over.
AUDIENCE SPEAKER: That should be a nice thing to figure out if we could test for that somehow, because it sounds very, very plausible from what I have seen elsewhere.
GEOFF HUSTON: That's right. Never underestimate just sheer blind negligence rather than malice. Folks just drop packets on the floor because it's easier.
AUDIENCE SPEAKER: Chris Baker. Did you ever think that that the distribution rate possibly heavily coming from Velixity could be a signal from an outflow box, so that would be more apt and better designed to true the end protocols? So the thing actually giving it is designed to get the resources on both because to profit it needs to? And is it possible to look at that?
GEOFF HUSTON: So, all the ad network does is deliver the script to your browser and stops, and I'm not measuring that. The script, on impression, fires up and now it's like any other script running on your browser, any other script. So it sort of doesn't matter how it got there at that point so now you are fetching URLs, so the ad network is now receded into the background and you and the ad servers are having a final conversation, that's what we're looking at. So I'm not looking at ad blocking or anything else. I'm not looking at the performance of ads. It's wonderful the ads networks are there and I love them dearly but that's not what I'm measuring, I am just measuring the script in your browser.
AUDIENCE SPEAKER: I am wondering is the data available about that ‑‑ the dataset that's showing the distribution of who is able to render the ads properly.
GEOFF HUSTON: Who's able to render ads properly is none of my business. That's Google's business, Mr. Google is the person you should be asking, not me. Thank you.
JEN LINKOVA: I'm surprised people start reading the documentation, you do not see the documentation prefix as your source in failure rate?
GEOFF HUSTON: The documentation prefix?
JEN LINKOVA: Yes, I have seen it a lot when I look at what I'm filtering, because people do read documentation and will see even ‑‑
GEOFF HUSTON: You must read the documentation. Route them globally but don't use the documentation prefix. I didn't see them. I just found those. We're getting better slowly.
JEN LINKOVA: And second question I have, I probably ‑‑ I'm not sure if you can show the slide with the highest failure rate for autonomous systems. I am curious what kind of networks, are they, are they ISPs, which might mean CPs or it's ‑‑ because on the best I see like non ISP networks, right?
GEOFF HUSTON: There is two kinds of issues going on here and I think they are confounding part of the problem. The first is, there are, if you will, corporate systems and networks that block incoming. So what we do notice is the failure rate is higher in weekdays than on weekends. Point one.
Point two, there's an awful lot of folk who are derolling their own and it sort of sets something up and then taken away bits of it. They replace the CPE, done something to their environment but some of the machines still have a bit of legacy flying around. So I suspect that our little folk, Teradon out there, with the 30% failure rate, is actually, I'm not sure if Teradon does v6 native ‑‑
AUDIENCE SPEAKER: Yes.
GEOFF HUSTON: Well, that theory got it shot out of the water, they are doing it bloody badly, that's all I can say.
Thank you.
(Applause)
JEN LINKOVA: Now we have two success stories. After we have seen how v6 can fail. I hope we can hear about how it works perfectly well.
TIMO HILBRINK: Well, that's kind of the story, it's not a very exciting talk, bus it is something we did a long time ago and we were just keeping it running, but in the last RIPE meeting in Amsterdam, during the v6 Working Group, some people stood up and said, why don't providers that already have deployed IPv6, why don't they explain what they have done, what they came across, what was so difficult and how they fixed things.
So, I thought okay, well we'll just dust off this presentation and give you a little update where we are and show you what happened in the meantime. Some of you might have seen a lot of this content already from me or Marco in the previous years but of course there is a lot of new people in the community so I'll just go through it.
For us, it is business as usual but I came across these snippets on these news websites, this is from May this year that kind of surprised me, but I think some of them were actually, I think the Microsoft one was about private IP space or something in their data centre or something like that, but for a lot of us, it is all we know this, but for other people it's a little surprise they are running out of IP space.
I work, obviously, I work for XS small, I'm a network engineer. We are a Dutch ISP, one of the first Dutch ISPs actually. Got bought by the incoming telco and we're doing a bunch of services typical for an ISP.
We started with IPv6 back in 2001. We started with 6 bone space and we set up stuff on our backbone, configured some routers to do peering, and just there was actually nothing, not much use, we had one or two machines in the office that could use it, but that was about it. We set up a tunnel server an a bit later, and a free read only IPv6 use net server that is still actually running. Just to generate some traffic, get people, get packets, get some traffic and see what happens.
But we, as we are an ISP we wanted to provide IPv6 dual stack to our customers, to home users. And we started working on configuring our BRAS and talking to our CPE vendor to get this off the ground. Eventually we got, from a CPE vendor they were quite interested and we started setting up this idea how to do this. We do PPP over the link to the end user. And over that we do a single session dual stack, so that is IP CP and IPv6 CP, over that we do DHCP prefix delegation and we also provide a DNS resolvers over DHCP 6 and then we assign a /48 per subscriber.
The CPE that we use is from AVM, and they have worked really well with us since about 2008, 2009, to develop the IPv6 tech in that including all the stuff you need to configure like firewall and all the other things that you need in your home network. And by now, like, by now a lot of CPEs are already v6 capable standard but back then they were one of the first ones.
So then you need to get people to use it and how do you try it out because a lot of it was untested technology at the time. At least for us. So, in 2010, we started up and we approached some of our customers, to see if they wanted to try this with us, to test it. And we got about 300 people that were interested, and it was all very simple, so we had a provisioning tool with v‑I, so it was just a tech file with some prefixs in it and customer circuit IDs, because our internal provisioning system was not ready yet by far. But, at least we could get going this way and we had some CPEs that were front working and the people that were in that pilot group they were actually quite positive and they were quite happy with how it worked and they were almost surprised that there was very little issues.
So, it was very positive trial. And then you have to move on. So we finally got a lot of stuff in our network in the provisioning network ready, and we got an option in the customer portal so that people could switch it on for their connection, so they didn't actually have to approach us individually, but they could just switch it on for the connection. It was an option.
And then it was actually the IPv6 support was all finished and the production was ready model and it came out of the factory with v6 enabled.
In 2012, we decided okay, we'll just enable it standard for all new customers, so every new customer would get a v6 prefix. And then in June last year, we decided well now we'll just switch it on for everyone, because it's working and it's fine. We're about 175,000 active v6 connections end users in our network at the moment. And the rest of the people, the rest of the customers, they either have a CPE that is not v6 enabled or they have deliberately switched it off for whatever reason, I don't know.
During that time, we also slowly set up a v6 on our other service so DNS, colocation, web hosting, and we decided that company wide, all new products, all new services that we have, have to be v4 and v6, otherwise they won't launch.
Within the Netherlands, we were one of the first ones. By now a lot of networks have started appearing in this list. And this is the list generated by the folks from APNIC. And you can see that from the ISPs with the large sample count, that XS4ALL is one of the few in that list. I must say that ‑‑ I will mention that in a minute ‑‑ this is over the years how you can see how we deployed, and you see some big bumps there, that was when we switched it on for new customers. That was in 2012, you see down the bottom, the line starts going up. And you see very clearly the big bump when we switched it on. We switched it on for everybody in June last year. If you compare this to how it looks in the Netherlands you can kind of see that from the early part in the left side of the graph, you see the gradual growth that we made and the little bump you can see back there as well. And then there's a big bump here at the end and this is actually finally KPN the Dutch incumbent telco has tart started rolling out v6 and doing a really large numbers because they are a really big telco and they have a lot of customers. I think they are about 7 or 8% of the network's v6 now of the customers, so finally they are moving. We did help them a little bit. Not so much technically, but more organisational, like how do you train your support staff and how do you deal with complaints and what do you come across, so, we ‑‑ with some cooperation we got them finally on board as well.
A lot of people asked us over time, you have so many v6 customers, but you know, is anyone actually using v6? Is the services, is there traffic? Yes, there is, this is a typical graph for us, so we do about 40 gigs in the peak of v6 traffic and that is quite substantial. I'm sure that the bigger networks in America see these figures in multiples, but for, I think for a European network this is quite a large amount of v6 that we're seeing.
So, yes, there is definitely traffic. Speaking about traffic, when you think about deploying v6 and you haven't done it yet, you need to think about how your traffic flows are, I think this was mentioned in the connect Working Group as well earlier. But v6 routes can be very different from v4 routes. So, if you start deploying v6 and you get actual gigabits of traffic, you really want to know that this is coming from the same place as where your v4 is coming from or at least an equal cost entrants in your network. So, you need to look at your peers, at your transit, to do they have connectivity. If you have an CDN in your network that is only doing v4 and the v6 is suddenly coming from outside the CDN, that's scenarios you have to think about, as you saw, we deployed quite gradually so we kind of saw this slowly. But if you do a big deployment and you wrap‑up very quickly, this is stuff you really need to take into account.
And then what do people complain about? What doesn't work? What our customers complain about is this sort of stuff. This is v6 hosting company that has switched on v6 but the customers have not changed their web server configuration and you get this sort of Apache standard screens. We see this quite a lot and as people always say, it works for everybody except for XS customers that's because we are one of the few ISPs in the Netherlands that do v6. I hope when KPN are a bit more advance with their roll out that people will say it's broken for them, so then we can say it's not our problem so all these hosting companies start pointing at us saying but it's only broken for XS for all. It's probably something in your network.
Other varieties of it is this sort of stuff, where people have actually a gateway in front of their web server that does v6 and then translates it to v4 towards the web server and then they move the web server to another IP address or another data centre and the gateway is lost and you get these sorts of screens, we see this regularly, about a couple of times a month I see these sort of complaints going in by either to our support on social media and stuff.
So, that is the third party websites not configured for v6. We also saw some problems when we switched it on for everybody, we saw that Microsoft outlook 2003, really people still use it, can't handle a dual stack server address. So it won't connect to Imap server and when that Imap server is dual stack, it will crash or break, it doesn't work, but the obvious solution is to tell these people to upgrade to something more recent. We did come across this. Quite a lot of people complained about this. In the beginning we had some problems with Outlook 365 the Microsoft product. I don't actually know what went wrong there but that seems to be fixed now, because we don't get any more complaints about this.
So, the lessons learned, or the advice I could, from an ISP perspective, I could give to people who are thinking about it or are, do need to deploy v6 because they are running out of v4 space for example. You need to start with an address plan. You need to have an address plan. Involve your vendors, and create a friendly user group. This is something really helpful, was really helpful to us because the people, you can get good feedback from them, they usually a bit more technically advanced customers, so gives you some good feedback and help you to debug problems. And take small steps at the deployment. Don't do it in a big bang, then it is quite uncontrollable. And just take your time if you have it. Of course you need to educate your colleagues, your help desk, your management, explain to them what v6 is why it is important. And review your peering and transit agreements so that your traffic is going the right way.
That concludes my presentation and we're now just thinking about changing the company logo to something more suitable.
(Applause)
AUDIENCE SPEAKER: Robert Kisteleki. Individual again. I would love to go with you guys. I live in Amsterdam. Unfortunately my bandwidth would drop to one‑quarter of what I have now. I have to make this judgement of which one I want. My current provider says, yes, we do v6, and I just don't see those bits. But in any case, excellent talk, thank you very much for sharing the details.
The actual question: When you have such a report from a client that says look, I cannot get to that website and you figure out that that's because they didn't configure v6 properly, what are you doing about it?
TIMO HILBRINK: When we get these complaints we actually chase the hosting companies.
AUDIENCE SPEAKER: Does it work?
TIMO HILBRINK: Yes.
AUDIENCE SPEAKER: Does it scale?
TIMO HILBRINK: No. It works, it works, but ‑‑ we do, we call them up or we mail them and we have a lot of contacts in the hosting or in the Internet community, in the Netherlands especially, so we're always able to find engineers that can take it up and, they will fix it and they get to talk to their clients and to their network engineers and they'll fix it. It works.
AUDIENCE SPEAKER: Hello. What is the percentage of your IPv6 traffic out of your total traffic? You gave us the total number, but what is the percentage for information on a fresh new deployment which is only a few weeks old? We are at 32.
TIMO HILBRINK: That's in the right direction, yeah.
AUDIENCE SPEAKER: I was just asking the v4 traffic total volume, so it's the same question essentially.
TIMO HILBRINK: Do the maths.
(Applause)
JEN LINKOVA: I would say it was a success story, I don't know why you were not sure, it's a success story. I think it's quite positive. It was very positive from my perspective. It works. Another deployment experience. Success story.
OSKARI RASI: Hello. I am here to tell you about DNA's IPv6 deployment experiences.
What is DNA? This is Finnish operator. DNA providers connections to private consumers, business customers and operators. DNA has in total more than 3.5 million mobile and fixed customers subscriptions. DNA is the largest cable operator in Finland.
Here you can see what kind of market shares DNA has in different areas compared to other players in finish telecom market. DNA has nationwide backbone network. Here is a simplified view of the backbone topology. We have more than 22,000 km of fibre and 1,500 POPs in Finland. DNA is providing Internet access with fixed and mobile technologies. In fixed side, we are providing ethernet and cable and DSL access.
On the mobile side we have 4G network that is expanding rapidly. Now we reach about 90% of population with LTE and by the end of year 2016, DNA will reach 99% of finish population, and of course there is still 3G and 2 G network coverage also.
So, there is actual topic of IPv6 launch in Finland by DNA. We have had IPv6 capable backbone network for ages now, and of course it's time to bring it to the customers.
Finnish Communications Regulations, why the Finnish broadband content and web server providers participate in national IPv6 launch. IPv6 launch date was held on 9th June this year. And DNA decided to participate and enable IPv6 to all consumer products that could support IPv6. And the date of the national IPv6 launch was selected as the target date.
At first, we enabled IPv6 for DNS own staff so we could find out the possible limitations. When enough Internet testing had been done, we started to enable IPv6 to different products. IPv6 first IPv6 came to one mobile subscriber and then after a short time, to another mobile consumer products. IPv6 to cable was also enabled in two stages. First a smaller group of users than to all the rest.
For DSL and ethernet customers, we have three different IP termination sites, and for them IPv6 was enabled in three stages, one BNG at the time.
Mobile devices with IPv6 dual stack. IPv6 is only enabled for the latest mobile devices by default. For the other devices, you have to enable it by hand, and this, of course, causes low adaptation because normal users usually won't do it by themselves.
Address distribution for users is done so that mobile users get /64 prefix, cable and DSL users get /56 prefix with prefix delegation, and /64 prefix if bridged connection is used. We also prepared our customer service for an NIX launch. We made IPv6 trouble shooting documentation based on RIPE 631, trouble shooting for residential ISP help desks. I would like to thank all the authors of that document for creating such a great document. We also made Finnish translations of the test IPv6 come or at least most of the site.
Challenges. First of all, we had to make IPv6 enable to three different IP mobile technologies. The challengeses were mostly not related to IP terminating system but rather the related systems. There was quite a lot to work to be done to get the account working in mobile broadband and the system for mapping IPv6 addresses to customers needed to be available so we could answer the queries made by the legal authorities.
IPv6 works even to IPv4 ‑‑ IPv4 works even though IPv6 would be broken, and in most cases, customers don't even know this that the IPv6 is broken.
Google blacklisted one of our AAAA resolvers twice, and this resulted in quite a big drop in our IPv6 traffic.
I guess I have wrong slides on...
So, here you can see the effects of Google's AAAA resolver blacklisting had on one of our mobile Internet APN's IPv6 traffic, but this had no effect on customers' experience because IPv4 resolution was still working and customers could read Google services with IPv4. Unfortunately, we haven't got any explanation from Google why this happened.
So, how it flows. Here you can see summary graphs of our current traffic on the top, the traffic graphs and below are total traffic graphs and below are the IPv6 traffic graphs for input and output traffic. On the cable, about 18% of the total traffic is now IPv6 traffic. And this is daily traffic.
And here you can see the growth of IPv6 DNS queries. This is a yearly graphic.
And for the last slide, I would like to show DNS IPv6 traffic and Finland's IPv6 traffic, how Finland's IPv6 traffic is seen by Akamai. And as you can see there is still much room for improvement even in DNS traffic volumes, but that's more and more user devices, routers, mode items and such support IPv6, there will be growth. It's great to be in the game now and not panicking later.
DNA is happy to give IPv6 to our customers.
Thank you.
(Applause)
JEN LINKOVA: Thank you very much. So any questions? Natalie?
AUDIENCE SPEAKER: Yes. I have a question for you. Natalie, RIPE NCC, what was the most expensive part of your IPv6 deployment? So in the whole IPv6 Doe employment, where did you have to spend money? Do you know?
OSKARI RASI: I don't know actually where the money goes. I am just a technical guy.
AUDIENCE SPEAKER: Did you have to buy many new equipment, for example?
OSKARI RASI: No, we didn't. And actually, I think I skipped one slide, because of the problems maybe, I could go back. We had to make IPv6 enabled in three different IP core termination sites and those ‑‑ we had the most problem with enabling IPv6 to DS lamb, because the DS lamb equipment, when one of them doesn't support the IPv6, so, about 70% of our DS LAMs can't forward IPv6 traffic. So that's, I guess, the main problem for us, but I don't know about the money part of things. I guess no big investment.
AUDIENCE SPEAKER: At that SHA, consultant for German Government the just going curious. What did you do about your IPv4? Do the customers still have an official dynamic address on the leased line or do they get Carrier‑Grade NAT from you?
OSKARI RASI: On the mobile side we are Carrier‑Grade NAT on some of other connections, mainly the phones, use Carrier‑Grade NAT for IPv4, but there is also native IPv4, probably IPv4 going straight to the customers. If they are using mobile home routers, and IPv4 dual stack is everywhere.
AUDIENCE SPEAKER: Do you have a plan how long will you keep it this way? Do you have enough addresses?
OSKARI RASI: For now, we have enough, so it will be ‑‑ we are thinking about going IPv6 only for the mobile side as well.
AUDIENCE SPEAKER: And from a German perspective, there is a little problem with us, do you force users to use your router or do they ‑‑ are they able to use their own equipment as a CPE?
OSKARI RASI: Users can use their own equipment but we also sell equipment that supports IPv6 and if they select something else, then there might be some problems with IPv6.
AUDIENCE SPEAKER: Hello. The question is only on mobile subscribers, is it fixed automatically assigned IPv6 addresses for that kind of customers?
OSKARI RASI: All is dynamic.
AUDIENCE SPEAKER: But why?
OSKARI RASI: But why? Because that's much easier to do. You don't have to do anything by hand. Or what do you mean?
AUDIENCE SPEAKER: I mean, is there a mobile customer will get all the time IPv6 address or not?
OSKARI RASI: Oh, you meant that. There will be different IPv6 addresses, at least on the mobile side, and it could change in the ethernet and DSL cables also, so it's not fixed. But I know some of our customers would like to have fixed addresses, and we are looking into that.
AUDIENCE SPEAKER: Is it fixed IPv6 address and fixed IPv6 sub‑net available for money or just not available?
OSKARI RASI: At this time, it's not available to private customers, but business customers can certainly have IPv6 static addresses and we are giving /48s for those.
JEN LINKOVA: Any other questions? I have a comment and a few questions. Speaking about the graph you showed us, I don't know when you say you you haven't got any explanations, have you dropped us any mail or it might be because in general we didn't have some brokenness, it moans that we detected something was wrong and I am happy to talk to you after this session to find out what's going on. So probably it went away.
OSKARI RASI: We contacted the staff and they removed the blocking, but we didn't get the answer why this.
JEN LINKOVA: To be honest, it's not always ‑‑ what we see ‑‑ we can see brokenness right. And cannot always tell what was that. We just see that some percentage of your users have problems with v6, and it's that's all we see. My question so you related to this because you mentioned a couple of times that users do not notice anything because v4 works. So how do you monitor? How do you know that it actually works?
OSKARI RASI: This is only IPv6 graph, so, the total traffic, there was no drop in that.
JEN LINKOVA: So basically you monitor by watching the amount of ‑‑ by monitoring the amount of v6 traffic you have.
OSKARI RASI: Yeah.
JEN LINKOVA: And interesting, because the previous session, DNS session, we had some discussion about DNS traffic over v6, what percentage of DNS traffic you see over v6 comparing to v4?
OSKARI RASI: That's a hard one. I can't remember.
JEN LINKOVA: I'm just curious, because there are different opinions on actually if there are any DNS traffic over v6 or not and I am glad to see there isn't in your case. Thank you. Any other questions. And now Natalie please.
NATHALIE KUNNEKE‑TRENAMAN: Good afternoon, I work for the RIPE NCC and I'm the IPv6 programme manager. I'm here again with an update of a presentation I did last time in Amsterdam. And I give you a bit of an update since then. But for newcomers I will get a bit of context, so don't worry.
So, lost stars is the name of this presentation and you already need a little bit of context to understand why I called it lost stars, because we have, as RIPE NCC we have members, and we measure how our members are doing with IPv6. So, we have this programme called Ripeness, where you can get stars. If you succeed with some steps in your IPv6 deployment. So the first step you get when you have an IPv6 allocation. And then you will get additional stars for announcing IPv6, having an IPv6 route object and the reverse DNS.
Now, in February or March, our research and development department, they did some research on Ripeness, the figures, and they found out that in the last few years, year‑and‑a‑half, around 460 of our members, and we have 12,000 in total, stopped announcing their IPv6 prefix from the global routing table and that number is quite significant, so it made me curious why that was. So, at the last RIPE meeting, I asked you guys, what you thought of the idea if I would contact those members, and ask them why they stopped announcing IPv6. And I heard approval in that, so I went further.
So, in September I asked those LIRs, I sent them an e‑mail asking why did you switch off IPv6 and I decided to do that in a form of a survey with just 7 simple questions, multiple choice, some were open, to get as much feedback as I possibly could. But before that I looked up where those LIRs were in our region. So that's this slide.
So, I was trying to see if I could find any trends in countries where people would switch off IPv6 than in other countries. And looking at the relationship between the amount of LIRs in a country and the amount of LIRs switching off IPv6, I didn't see many very big spikes. So that was already interesting to know that it was not ‑‑ there were two cases where it was a bit harder to define but in the end it wasn't really that big.
So, then I went reaching out. On the 24th September, some of you even may have received an e‑mail from me, with a little survey, asking why you have switched off IPv6, and/or if you didn't want to fill in the survey, it was anonymous by the way ‑‑ if you wanted me to give you a phone call for any reason, I was ‑‑ so that was also an option. And I was surprised to see that in two weeks, we got around 15% response, and for a simple online thing, that's quite high. So I was very pleased with that, also I decided not to stalk them. So not keep pushing if they didn't reply you know, it was anonymous and I didn't want to bug them too much because I know they were probably busy fixing IPv6.
So, apart from the survey responses, I also got quite a lot of e‑mails personally directed to me saying we have got multiple LIRs which prefix reactually talking about so I can look into this. Or I will fill in the survey, but I have some other questions, can you give me some pointers for an addressing plan or other stuff. So I got around 50 e‑mails, apart from the survey responses, with all kinds of questions, asking for training etc., etc. So it was a good side catch really.
The first question in the survey was were you aware that you were announcing IPv6? That sounds silly, but there were actually people who did not even know that they were announcing their IPv6. So, maybe they were not in charge. It could be, I don't know. But, okay, so luckily the majority was aware, and then the next question was then: What was the purpose of the announcement. So was it a test? Was it production? And again you have some people that don't know.
That's not funny, no. Then, as you can see, I was quite surprised that the majority, this, by the way, is not percentages, the number you see at the bottom, those are actually the numbers of the respondents, so 59 people responded, it was only a test. And then the rest said production or I don't know.
So, then I was curious, what were their experiences during the announcement? And as you can see, good news, the vast majority did not have any issues. That was good. There were some software issues though. And hardware issues, and there were even some customer complaints. Now, a lot of them also in common said it was a test, we ran into some issues, that's why we switched it off.
So, and the question why they disabled the IPv6 announcement, the majority said it was a test. A few of them said it was bankruptcy, and I'm not really feeling well ‑‑ sorry ‑‑ so there was some bankruptcy cases and with the bankruptcy cases, because Ilage, one of the community members in the last RIPE meeting, asked me, can you also figure out if they stopped announcing IPv4, so I did. I asked did you also stop announcing IPv4, and in some cases, yes, and that was because there were bankruptcy issues, etc., etc. I have to go back to the previous slides, because you see also another interesting one. Political issues. Those were from countries where, for example, there is only one upstream provider that doesn't provide IPv6, or stop providing IPv6, or that was a case where the Government, for example, does not deal with IPv6 and data retention, for example. These kinds of cases. So that was the political issues.
The majority that said that it was a test and stopped it, they also said because they could fill in multiple answers here, it was a test, and they were also restructuring their network. So, for example, they bought new equipment, they are not really confident with IPv6 yet and they are still practicing with that. So, restructuring network, tests, those were the most obvious cases.
So, then, of course, you want to know when will you get back online with IPv6. And there there was an interesting one, the majority said we don't know. And basically in the answer in the comments sections I got the answer "we don't know" because it relies on external factors, for example, hardware vendors, software vendors, etc. So, that is the majority ‑ I don't know, because it is outside their scope or because management deprioritised IPv6. So it was out of their hands.
I was surprised to see that nobody answered in two years or never. So that was good. So but in total the majority will say within a year, so that's at least promising.
The last question of the survey was, of course, can RIPE NCC help you with anything? Is there anything the RIPE NCC can help you to get back online again? And then you get answers, a lot of answers asking for the training courses, so that was good, since I am also a trainer. Tips for addressing plans, so we point them to documentation, the SURFnet document, best practices, RIPE 554, RIPE 631, help desk documents. So it was a good opportunity for us to ‑‑ if they provided an e‑mail address, if they decided not to be anonymous and if they had questions to provide them with that information anyway.
They also asked us to talk more to governments and reach out to them for the reasons like I said, with having one upstream provider or data retention issues.
A lot of guys that said currently we have everything under control, but if we run into issues, we know where to find you. So that's the last bullet point. We'll be getting back to you with that.
So overall, it was interesting to see that a lot of them were running a test. They came into some issues in some cases and they are working on it. There was one case that was interesting that we fixed with RIPE Stat because the provider sent me an e‑mail back saying, but I contacted my upstream who are announcing IPv6 for us, but they say it's working. So why are you saying it's not working? And I looked at RIPE Stat and I could see it was the scope of the ‑‑ was geographically very, very small. So it was not very widely visible. And then I looked more into the data and guess what? It was only on for an hour and a half. So that meant that the provider actually called their upstream provider and say you hey, I thought we had agreed that you would be announcing our prefix and they said yes, we do, and sent them a snapshot, but it appeared they only did that when they called.
So that was not too pretty.
Other ‑‑ another case was where the manager said I told my team to do that. Let me get back to you. And then an hour later came back and said done. So, that helps also sometimes. So, that those were interesting numbers to see. I keep following up, so I will do a check in a few months to see if what they said here in a few weeks N a year, in three months, to see if that actually will be lived up to.
So I thought yeah, after last year when I asked you if I should contact them, and you said yes, basically, to share with you the results.
There is one part of our presentation and then I have got two minutes for another part.
Talking about this Ripeness, this four star mechanism that we have. We have been giving out T‑shirts for the last four years for LIRs who have this four star Ripeness, right. So if you got an allocation, you were announcing it, route objects, reverse DNS, you could get a T‑shirt from the RIPE NCC. I saw some of you wearing that T‑shirt, the black IPv6 now T‑shirt. Then I thought we're in 2016 almost. Should we still reward that? Shouldn't we reward the next step, actually pushing the packets. So we decided that as of the 1st January, to only give out T‑shirts to LIRs who have the fifth star, which basically means that their website is IPv6 enabled or they have, and/or they have access customers over IPv6. So that's what we're going to do. But there is a little catch here. Because we have a threshold, and at the moment, the threshold is 8%. So that means basically 8% of the web sites that you are running have to be IPv6 enabled or 8% of the access customers, the idea was to double this threshold every year, yeah, I know recollect that's a bit steep, so that would be 16% for 2016, and 32% for 2017, etc., etc. I'm still dubious about if we should double the numbers every year or if we should have ‑‑ because we like to keep people aboard with the five star stuff, we should do it a bit more gradual to, for example, add up 5% every year instead of doubling it. So that's where I would like to hear a bit of your feedback as well, what you think would be the standard for the five star Ripeness and rewarding that with a T‑shirt.
And that concludes my presentation. Thank you.
JEN LINKOVA: Thank you Natalie. Questions?
AUDIENCE SPEAKER: Actually just one question. It sounds like it's really nice what you did of course, but it sounds like to me it's a little bit like, you know, the parents and children thing, you should do this kind of thing, and so, the response that you get is, how much of it is out of like reality operational need that they would do it or how much of it is somebody just being polite saying you know, I will do it? I mean without you going there, they don't even notice they don't do it. Was it turned off like after a couple of days? I mean would that happen you know, so they could do just out of politeness or really operational needs? So, I'd be interested about that. Thank you. Like do people just because they are polite or you know they really have operational need to do the IPv6?
NATHALIE KUNNEKE‑TRENAMAN: I don't know.
AUDIENCE SPEAKER: Because when you are calling them and sending an e‑mail you are from RIPE NCC and a lot of people also in this room, a lot of people think RIPE NCC is some sort of governing body, so they basically feel like parents are coming, you know, to chase after to do certain things ‑‑
NATHALIE KUNNEKE‑TRENAMAN: The reason why I just sent that e‑mail once is because I don't want to harass people. It was just a friendly e‑mail, anonymous, etc., asking, you know, I'm just curious why you have switched off, and if RIPE NCC can do anything to help you. By the way, as a survey I would appreciate if you fill it in. It was not mandatory whatsoever and I'm not intending to chase people.
AUDIENCE SPEAKER: A very quick line on that. It's just the T‑shirt you get, nothing else.
Gert Döring: I think this is a great thing, and some of the feedback actually means there has to be some sort of follow‑up work, especially the talk to governments. I have the nagging suspicion where this is coming from, countries where regulation requires that you use the state telco and the state telco has no v6, so that are they supposed to do.
On the related thing, I have an idea for extra RIPE stars. And I think if you have your IPv6 allocation for more than 15 years and have been announcing it all the time, every single day, you should get a couple of extra stars. Extra T‑shirts. Of course, I will get in trouble with my wife if I come back with ten extra T‑shirts.
AUDIENCE SPEAKER: Hello. Is the measurement done reliable enough so that having the something could be included in Address Policy at some point. Or could we do it, could we make it reliable and fair enough for this? I didn't mean ‑‑ I meant the fifth star.
NATHALIE KUNNEKE‑TRENAMAN: The fifth star is just an incentive, you know, it's just a measurement ‑‑
AUDIENCE SPEAKER: Can we bring it more than just incentive for a T‑shirt?
JEN LINKOVA: If you are trying to get an additional space ‑‑
AUDIENCE SPEAKER: Pretty much.
AUDIENCE SPEAKER: Lessee, CDC and DK knock. I noticed the Ripeness stars are available online. Is there a history to that so we can also see who is using the stars?
NATHALIE KUNNEKE‑TRENAMAN: You mean a list of name and shame or something?
AUDIENCE SPEAKER: Local reach out.
NATHALIE KUNNEKE‑TRENAMAN: What we do is we list the LIRs who have five stars, right. And that already gives you, and the percentages actually, so that gives you already, and if you know the threshold you know also how well they're doing in that respect because they are still on the list, so every year we raise the threshold some may drop list because they didn't make the threshold. In that respect it's more promoting them and showing people we are on this website so we are over IPv6, and... yeah... so, that basically means who is not on the list.
AUDIENCE SPEAKER: But who would previously there who are not there now Who were previously on that list like four stars going to three stars like...
NATHALIE KUNNEKE‑TRENAMAN: No, that we don't publish. No.
AUDIENCE SPEAKER: Sacha speaking as a regular T‑shirt user. So, I got it right, you were thinking about raising the percentage the threshold by 5% per year or doubling it. I'd like to be brave and say let's double them because going by 5% when you are now at 8 when I did my simple maths right would take me about 18 years to reach 100% IPv6 deployment, and I hope it will be faster.
NATHALIE KUNNEKE‑TRENAMAN: Can I do a quick show of hands on both, is that okay?
JEN LINKOVA: Yeah, yeah.
NATHALIE KUNNEKE‑TRENAMAN: Who says doubling the numbers every year? Okay. Only the hard core ones. Who says gradually, 5% per year? I see some favour in doubling, okay. Interesting.
JEN LINKOVA: I am afraid in this case by 2009, we are going to have 128% of IPv6 ‑‑ that's awesome.
AUDIENCE SPEAKER:. I have plenty of ideas for you. Why don't you think of multiple flavours of T‑shirts corresponding to the number of stars you hand out. So, and also the long time with stability, so for example, my five year, five star is like gold anniversary for marriage or silver, and that's not all. Those who lose their stars, you have either the possibility that you hand out T‑shirts I have lost my stars or the right to get back the stability T‑shirt and and with that you have stability and faithfulness and good behaving LIRs.
NATHALIE KUNNEKE‑TRENAMAN: I am way ahead of you. Because, every year, every year we are going to make a new nice designed five‑star Ripeness T‑shirt. So you only get that new T‑shirt out of the series if you get over the threshold again. So if you see a guy just wearing the 2016 but not wearing the 2017 shirt, you know he dropped off.
AUDIENCE SPEAKER: But do punish those who lose their stars with ‑‑ withdraw the T‑shirts.
NATHALIE KUNNEKE‑TRENAMAN: I end up with this huge pile of dirty T‑shirts.
JEN LINKOVA: Any more questions because I have a few comments?
So you ask people as a first questions, are you aware that you are announcing IPv6. Did you ask them if they were aware of announcing v4? Probably the answer is no, because they don't even know what IP address is, right.
And those guys who deployed IPv6 in one hour, could they come and give us a talk about the fastest IPv6 deployment.
NATHALIE KUNNEKE‑TRENAMAN: That was just announcing it in the global routing table, nothing else.
JEN LINKOVA: So I couldn't expect even a lightning talk next time. Thank you, it was great.
(Applause)
And I think that four minutes ahead of time, thank you very much. Please keep deploying IPv6, come back and comment again and don't forgot to, rate the talks. Thank you.
(Applause)
