Archives

Connect Working Group

18 November, 2015, at 11 a.m.: Connect Working Group



REMCO VAN MOOK: Good morning. Welcome to your second most favourite Working Group of the RIPE meeting because the most favourite one is at 4 p.m., Kurtis, right? Which is number cc services. This is connect Working Group, have a look at this picture. If you look at this pick fewer for more than 20 seconds and you don't have to run to the bathroom, you didn't drink too much last night. No people running off yet. It's only Wednesday. I am here today, without my lovely co‑chair Florence because there is two of her right now, actually, so she wasn't able to fly here, actually not allowed to fly here. So, but she is following us on webcast, so everybody say good morning Florence. Morning, Florence. The agenda for today, we have got a couple of lightning talks, one of them I will be pretending to be Florence, not nearly as good‑looking and then Thomas presenting, then Emile of RIPE NCC is going to be talking. Nick has a bad case of Wednesday morning so he won't be presenting. And finally we have got Martin Levy presenting Tryst which is pretty cool stuff that he worked on at this weekend. Scribes, Remco from RIPE NCC will be doing the scribing and Vesna will be doing the chat monitoring.

Any comments or suggestions? I figured as much. So, Matthew, if you are ready. Has anyone seen Matthew this morning? OK. Right. So we will get to that later, if he shows up. So nick, you don't have the worst case of Wednesday morning; somebody else has. Then we move forward to the second lightning talk, which is by somebody who is not here, so that is also lovely. Uta Meier‑Hahn who has presented at this Working Group before and announced study on Internet interconnection, wanted to present her results. She is in Brazil, right now, in some place with dodgy connectivity so she decided to record a video of her presentation and that is where we are going to be looking at right now. And there is a couple of charts that she talks about, they are in the presentation that you can download so you can have a look at what she is talking about. Right.

Uta, can you hear me?

UTA MEIER‑HAHN: Hello, Bucharest, this Uta Meier‑Hahn calling. I would like to share with you some results from a small on line survey I conducted about the regulatory conditions of Internet interconnection. And I am reporting from Phase One of the survey. The purpose of the survey is exploratory in three ways and that is, first, to examine to what degree different kinds of public regulation influence Internet interconnection today.

Second, to develop an understanding about what this influence looks like, in practice. And third, to get leads for further investigation as to where Internet interconnection regulation is in place today.

In this short input, I will limit myself to aspect one and two.

This report is grounded in a population of 132 participants who have answered the quantitative part of the survey. 74 participants also answered qualitative follow‑up questions. The participants represented a good mix of stakeholders, that is axis and transit networks, content related actors who operate their own networks, content producers and delivery networks, Internet Exchange points and a handful of others, such as academic and research networks but also software or infrastructure as a service networks and hardware vendors.

Initially, the participants were presented ten kinds of regulation. They were asked, first, whether they had encountered this kind of regulation, and if so, how influx they perceived it to be with regard to the interconnection practices. You can look up the detailed charts in the two slides attached in the appendix. What struck me most was that eight out of ten kinds of regulation had been encountered by more than 50% of the survey participants. So number one, Internet interconnection is all but an unregulated space. Number two, regulation is not binary; it takes many shapes and does not need to be labelled Internet interconnection to influence connectivity.

The second chart shows how influential the affected participants find the regulations with regard to the interconnection practices, and please note influence here is neither good nor bad. Every one of the listed regulations is perceived by one‑third or more of the affected participants as somewhat or even to a great extent influential. So learning number three, the temperature in the room is warm.

Now, what do the regulations and their perceived effects look like. Due to time constraints I can go into detail for the item rated first, when a regulatory authority imposes its own technical or operational standards. 44% of the affected networkers perceived it as somewhat influential or even influential to operating stand when this happens. Of what kind is this influence? Participants have encountered this kind of regulation in the following areas: In the realm of governmental surveins where regulatory authorities demand direct access to routing equipment, often at the cost of operators, by the way, and the realm of hardware, where network operators are only allowed to use licensed equipment, this is, for instance, the case in Russia. Also, at the level of business relations, where regulatory authorities intervene both in the content and form of interconnection agreements, for instance by posing MP LA periods at route servers or demanding the use of specific service level grams and also at the level of network architecture where regulators join the conversation about the number of links and the maximum utilisation that a network can or should have.

What themes for further discussion emerged from the participants' answers: For example, these, I am still with this first item of regulation, when a regulatory authority imposes its own technical or operational standards. And that is, first, networkers do not ‑‑ standards set by public regulators but they have doubts about the regulators' know how and insights into best practices. They feel regulators do not consult with them enough to produce logical and thought‑through standards. A second theme that emerged is of individual strategies, whilst some networkers try to follow and comply, other operators definitely manage around public standards wherein possible. Overall, there seems to belittle coordination among the operators. A third number of the theme is that operational standards setting sometimes travels on other policy tickets and I'd like to exemplify this by mentioning three cases and please note that I was not able to check them yet; these are answers of the participants of the survey. One stated that Australian ISPs and CSPs may soon have to provide network and security designs to the regulator in the course of implementing mandatory data retention.

A second case, in New Zealand, apparently, traffic was mandated in the course of the fibre roll‑out. A third case, in Russia, operators allegedly need to document and get permission for every network node and they can only use licensed equipment.

So we have data retention, we have fibre roll‑out and licensing, all with the effects on Internet interconnection.

To wrap this up, the survey explored networkers encounters with regulation, and substantiated that the plethora of rules are already in place that influence operations, also underline how much the local context matters. So I believe that we are in need of a very differentiated discussion about this topic, as I said I was only able to scratch the surface here, to learn about the nine other kinds of regulation and how your fellow interconnection professionals assess them, please look out, I will try to publish it before Christmas and I will make sure to distribute it to the RIPE community as well.

Thanks and have a good time at RIPE 71. Bye‑bye.

REMCO VAN MOOK: All right. I don't think she can hear us right now because she is in the sticks somewhere. But thank you, Uta. You can read back that I said thank you. Does anyone have any questions for Uta, not that she is going to answer them right now but it would be good for her to know she is heading in the right direction with her research and more specifically I would like to know from this group as well if you think this is interesting for this forum or whether it should be elsewhere?

AUDIENCE SPEAKER: Sal em. Lebanon. I am very interested in what she is to go, I am going to read the report. The question for her is, how about IXPs because hearing rumours that also IXPs now need to be regulated by some regulators so I wanted to know whether this came in her surveyor.

REMCO VAN MOOK: OK. That is a good question. I think she is going to be writing about that in her report. Anyone else? Comments? No. In that case, we are going to move on to Thomas King from DE‑CIX with his lightning talk about BGP communities for blackholing traffic.

THOMAS KING: Thanks, Remco. So, how much time do I have for the presentation?

REMCO VAN MOOK: Seven minutes.

THOMAS KING: That's good. I will do it in five. So today I would like to talk about black hole BGP community for blackholing. This is together with the other people named on the, here on the list. The idea for starting this work were actually coming from the IXP world but we extended it to be more generally applicable so I presented the motivation here from, to networks, network NP that are connected and do peering with each other, if there is a massive attack between ‑‑ it might be the case that the link between network A and B gets overloaded, for instance if network A is a transit provider or some internal resources within the network infrastructure of network B might get overloaded. We know that there are a lot of solutions out there how you can handle massive DDoS attacks, one is and that is not new, we take advantage of the existing technology but I want to describe this technology a bit so that you understand why we champion that. Someone called blackholing which means that network B, if network A provides such a feature like remote blackholing, it sends out more specific blackhole announcement and which creates a blackhole in network C and then all the pad traffic is dropped there, all the DDoS traffic is dropped there and the link and the infrastructure that was congested or overloaded might get relieved. That is a basic idea of blackholing, you probably already know that, it's nothing new. The thing is, we looked at how this black holes can be triggered and we figured out that there are different ways of triggering a blackhole and more or less depends on the network or on the ISP you are connected with, how they implemented the different triggers, and yeah, that was more or less like a mess the whole ‑‑ stuff, so I have I have ISPs that provide blackhole features and sometimes blackhole community, sometimes you have to rewrite next‑hop address, it's weird, especially in the ISP case we have ISPs that takes blackhole communities out of policy control, blackhole ‑‑ sorry, BGP communities base, which might ‑‑ makes it very special to handle because it's an exception in this policy control space.

If you look at ISPs it's a bit better, usually they use BGP communities for that but most of the ISPs use different communities for that.

So we are thinking about that we should come up with one commonly agreed BGP blackhole community which triggers a blackhole everywhere independently of how it's actually implemented and and without knowing all the details. And actually, we had quite a discussion on this topic in 2014 on the RIX mailing list, which is European association of sips and we agreed there that we should have that kind of blackhole community, we had a panel and presentation about that, we drafted an Internet draft for that, we submitted it to the Co‑Working Group, and we had a discussion with IETF 93. We got some feedback we should extend it from ISPs to ISPs as well which we actually did and we were asked to add more recommendations for operational questions and we also did that.

Though, the current state of the draft is that it's available in version 1 or 01. As far as I am aware of, all the comments and requests are resolved. And we are already got a Working Group adoption three weeks ago during the last IETF meeting. So, the next steps are we need more feedback, this is why I am presenting this to you, I would like to get feedback from you on what we are doing here, is it helpful, how could it be improved, and we plan for last call before the next IETF meeting, so, hopefully we will see this interdraft become an RFC within the next year. So that is pretty much it. Here is a link to the current version of the draft. Please have a look, and send me feedback, my e‑mail address is in there, you will find me around here during the RIPE meeting. Please come and see me. Thank you very much. Questions?

REMCO VAN MOOK: Any questions for Thomas?

THOMAS KING: Good. Thanks.

REMCO VAN MOOK: Thank you.
(Applause). Up next is me, pretending to be Florence, forgive me. And since it's her presentation, I am going to need my sheet of stuff I need to say.

So, at the first session of the connect Working Group in RIPE 69 London, Reese, Dean and Florence attempted to start drafting a best practice document for ISPs and how to work with CDNs and the objective for that was to propose recommendations for ISPs to better interact with CDNs and optimise their content delivery. So, it was about optimising load balancing, how you do caches and peering transit, cache loading and so on. And from there, we have the help of quite a few people from the CDN community for pieces of areas of attention were identified, first of all BGP prefix announcements, traffic engineering, second was capacity diversity planning, IPv6 and four was documentation, most notably peeringDB. This is a summary. If you download the slide, there is a lot more detail to be found.

But the problem was that this hasn't really moved forward and I guess everybody has been busy. And it's not going anywhere. So I am proposing to leave this as it is right now, if somebody is interested in picking it up again, all the material is in here, it would be good to at least hear about it at the connect Working Group, if somebody wants to take this to the BCOP task force, they are more than welcome to do that as well. And so, we are closing this chapter for now. Any comments, suggestions, questions? Anyone willing to pick up from here? It's awfully quiet. So, this is the weird thing. This room has usually has ‑‑ everybody in this room is usually, some of the loudest people at meetings, except for one you are sitting in here, and I am wondering why that is. Are you scared of me? I hope not. You should be, but that is a different story.

So, what is going on? Randy, thank you.

RANDY BUSH: IIJ. We are not really coming to the connect meeting; we are escaping from Address Policy.
(Applause)

REMCO VAN MOOK: Randy, you gave the game away, why do you think I never opposed to being programme against Address Policy, that gives me the perfect excuse not to show up. So that is it for the lightning talks.

Anyone seen or heard of Matthew yet? If he shows up before the end of the session, I will let him have a go.

So we are going to move on to the next presentation, which is, it's Andrei Robachevsky.

ANDREI ROBACHEVSKY: Thank you, Remco. Hello, everyone. I am Andrei Robachevsky, I work for the Internet society, and I am helping to promote this effort that was started one year ago, slightly more one year and few days, which is called MANRS and that is what this presentation is about. I am doing series of similar presentations so, for those who have already seen some of those slides, I apologise for this repetition, but what is ‑‑ I'm more interested in your thinking and your feedback and of course your support. So that is the initiative that is ‑‑ has two names, actually: One is the routing resilience manifesto but for some it sounds too revolutionary, and there is another one which is MANRS ways, nice acronym standing for Mutual Agreed Nums for Routing Security.

So what this initiative is about, I think it tries to address a very difficult problem of routing security that we are facing and the problem is that you alone cannot protect your own network. Protection of your network is in the hands of other participants in the routing system, so how we solve that problem. Of course, there are technology and tools and and extensions and protocols, that is all nice, but I think how we reach this disconnect between others protecting your network and you protecting their networks. So it has this social component that we are trying to address with building a visible community of supporters and of course it's important to very clearly articulate how well ‑‑ I won't say even good looks like but how the minimum baseline, how does that look like. And this document called MANRS, it also defines a very small package, it's certainly not an aspirational thing, it's very practical and pragmatic thing, saying, hey, this is the minimum we need to implement. And it focuses on very simple cases, it doesn't try to boil the ocean or say how can we secure the whole routing system. It looks at very simple cases where risks and ‑‑ brought to a minimum but, at the same time, with the hope that it goes to be deployed widely, I think we can expect positive impact on routing, obviously.

So, what are those good MANRS? There are four of them, and they might look like motherhood and apple pie, because we talk about filtering, anti‑spoofing, coordination, contact information up to date and routing policy, published, but if you look at the document itself and I hope you captured the URLs on the first slide, then you will see that those actions are scoped quite narrowly for specific configuration cases. Let me quickly go lieu them and give you some examples. For instance, when we say about filtering we are not tackling peers or up‑streams, it's only about your customer code. The same thing was anti‑spoofing, some topologists or implementing anti‑spoofing can be challenging, introduce some fragility, affect your customer's traffic but for some of the cases, for instance, your own infrastructure or single home stop customers where you know the topology, what is behind and what is connected to your network, then it's not such a big deal, it's not a big deal to implement this stuff.

Same coordination, it's really simple. The only I think that this action requires is that you publish global accessible up‑to‑date contact information. It actually doesn't tell where, because the thing is, what this document is aimed at, it is aimed at outcomes, so it doesn't prescribe a specific technology to use. But it says, hey, those are outcomes we are looking for. So, well, there is some discussion, there is some, I won't call this guidance, some discussion with some pointers to other information, other documents, for instance, saying hey, peeringDB the IRRs, if you go ahead and publish information there that is a good thing, that is good enough. And global validation is publicly document routing policy, and information about what others can expect being announced from you, so others can be enabled to do validation of prefixes that are originated by your network.

So MANRS is not only document, it's not a BCOP or BCP or whatever, it's a commitment, and I think as someone who says yes, I adhere to those principles, I think routing security is important, it's not just saying that; it's actually implementing those actions. And that is the commitment that is there.

As I said one year ago on the 6th of November, this initiative was launched 99 ISPs got together, put together this document called MANRS and has started this initiative. This is work in progress, the document itself, it's not cast in stone, so, the thing is, and I think that ‑‑ yeah, I will talk about a bit later, but what I just wanted to say, this is not something that started now. Now we are just looking for more participants, this is more to that.

So if you go to the web page of this initiative you will see growing list of participants, we start with nine, now we are slightly more than 30. It's not a big deal, compared to 50,000 networks out there, but we need to start somewhere, so right now, we are building reputation, building the brand and that is why I am doing those presentations, promoting this stuff.

So, as I said, it's not only document and the document itself is not something cast in stone once and forever, first of all expanding group of participants and here we are looking for leaders, not for everyone who never heard about routing security, interested how to implement the stuff, of course they are welcome, I think mostly we are reaching out to leaders in this industry, people that have already implemented those nums and much more and can show this as evidence and say, you know, we are security aware, we take security seriously and we think MANRS can help further that.

Another challenging thing, we are trained to build a community so other things that, people can discuss security issues, the community that can demonstrate that that is important and push this further, because website alone is nothing, so this community also community of ambassadors that can go out and convince their customers, peers, partners, friends, that this is a good thing to do and developing better guidance. We had a BCOP task force on Monday. Job Snijders presented a proposal so we are working on this document, providing more guidance for people lowering the threshold hold for people who haven't implemented the stuff. Making more clear what is expected in this action by looking at implementation guidance.

Well, there is a sign up forum so if you are interested. The thing; you don't need to implement all four, although that is certainly appreciated and welcome, but you can tick some of the boxes and sign up for MANRS if you really implement the stuff. Well, I think questions I heard discussing with some of the operators after those presentations and attending some of the meetings is, yes, we take this seriously, we implement this stuff but why we need to sign up? Why you think it is important. Yeah, it's a difficult question, but the thing is, those are kind of three reasons I manage to come up. And, well, if you guys have a better reason, let me know, I will happily accept that and use this in my kind of promotion campaign but I think it boils down to this disconnect that we are trying to bridge, without bridging this disconnect I think it's hard to expect that we will have significant improvements in the routing system. Routing security cannot sneak in with a new software update, that is too complex. It requires action, and this action doesn't necessarily have a very clear business case, sometimes. So the business case, I see, is that while you invested in the routing security, by promoting MANRS you are actually increasing return on investment. Because if others do the name your network, your measures become more effective.

But, why I am doing this here? Because I think we also talking about interconnections and another thing is that I don't think there is such thing as global community. They are a collection of local communities, local communities united by common operational interest. And my question to you, to you, representatives or members of those local communities: Do you think MANRS can also be useful not to you as individual member, I mean, we covered that, I guess, but to you as a community, to take this as a reference point, to take it as a baseline, as a platform that you can also, in your local community, create some security aware activities? So, please think about this and maybe some of you would like to respond in questions. And that ‑‑ I think that ends my presentation. It's kind of some of the testimonials of existing participants running on the screen, but I am done and open for questions. Thank you.

REMCO VAN MOOK: Thank you, thank you. Any questions for Andre? I see a couple of people walking up.

MARTIN LEVY: Usual suspect at the microphone. Last week a major Internet backbone in southern Asia announced a whole bunch of routes anchored by its ASN that it did not own. Clearly this looked like an IGP to EBGP link circa 1977 when we used to see stuff like this. Before I sort of ask the question, who here in the room saw those route leaks? Who here in the room saw their route in that route leak? And I know there are others that did that are in the conference. Actually, I am going to ask the general question: Who here is motivated to do something about route leaks if this style and others aka what MANRS is talking about, because of that type of event? And I am waiting for everybody to put their hand up. And I will wait a long time. Thank you, Randy, for that.
(Applause)

RANDY BUSH: IIJ. That was a wonderful scroll there of all the testimonials from people who actually don't practice what they preach. And in fact, the big names on the front of MANRS, and I can name them if you want, like Level 3, have had major problems in the last months. So, words are cheap, OK? What are we actually doing, technically, in the community, to improve routing?

REMCO VAN MOOK: Thank you, Randy.

ANDREI ROBACHEVSKY: Yes, it's ‑‑ well, thank you for bringing this up, it's an important question. Let me respond to that in a way that we are looking at more checks on compliance, we are also looking as a community of MANRS participants, how can we address those things, because, well, bad things happen, we know that. No one is perfect, especially big networks. I think support is important, I think intention is important, but I agree that checks and balances and commitment and verifiable commitment is also important. There are challenges in checking these things, while they are obvious when incidents like that happen, it's not obvious when nothing happens, were they compliant or not but we are planning to improve and make it more public, so participants can respond to those incidents and give us confidence that this is an incident that not just lip service they are paying to MANRS. So this is work underway but it's not easy.

RANDY BUSH: I am not asking you to remove their logo. I am asking you to give them the tools to do ‑‑ to put their money where their mouth is, right? We need to give the operators the tools to easily do this. I pick on Level 3 because they used to be one of the best, they descended from MCI where royal ‑‑ I don't need to name names ‑‑ two key engineers did really good route filtering based on the IRR, OK, and two, three years ago, left for Google and Apple, I think, and Level 3's ability, because ‑‑ and not because they are evil or stupid, because it was very hard to maintain and was taking two very, very senior engineers, who had decades of experience doing this, to maintain it. How do we reduce that barrier so idiots like me can do it?

ANDREI ROBACHEVSKY: It's a very important track how you actually implement MANRS and more, right? So do you have any suggestions?

RANDY BUSH: Oh, I put my money where my mouth is. I am trying to do validation based on the RPKI. I have been the poster child for the IRR since 1995. But, you know, the IRR is getting old and I will talking about it in routing. RPKI deployment is increasing and could significantly help, but it's not a panacea; it would not have caught the problem ‑‑ no, no, it would have caught the reorigination, it won't catch what are known as routing leaks. And actually, I would like to have a taxonomy of what is going on, a routing leak is when somebody misannounces it with their origin, with their AS as the origin, somebody else's space, right? A re ‑‑

ANDREI ROBACHEVSKY: That is a hijack, right?

RANDY BUSH: Well, that ‑‑ the word hijack assumes intent. I am sorry to be picky about my own language in an audience for most of whom it isn't, but hijack means bad intent, so misorigination is what I will use. Anyway, I have a taxonomy of four classifications of what is happening and neither the IRR‑based filtering nor RPKI will solve half of them. All right?

ANDREI ROBACHEVSKY: I mean there is some work in the IETF going on now in trying to figure out taxonomy and provide some solutions, but more importantly, I think ‑‑ I totally agree with you. One thing that we came up is starting this work on BCOP providing more clear guidance more tailored to MANRS ‑‑ also addressing smaller networks that maybe have less clue in how to implement those things, making those solutions more easily deployable. We hope that that can ‑‑ is diffusion of those technologies in these networks and therefore, support MANRS.

AUDIENCE SPEAKER: Blake Willis. So speaking as someone ‑‑ operator, also equipment vendor and do a lot of customer managed type stuff and so forth. Your router when you buy it comes with a manual, and that manual says lots of nice things about how to configure BGP and how to configure your IGP, but it doesn't say if you want to connect this to the Internet, you need to register your routes. It doesn't say if you want to connect this to the Internet here are the best common practices and so forth. And I can say the majority of my customers ‑‑ that at least, we are fortunate enough they look to us, but we have several customers in the, for example, in the example that Randy gave, southeast Asia, also several north African, head down, stuck in the documentation. Last year we had a customer that we said all right we are ready to configure BGP, what is your ASN and they said we will ask our router vendor. OK? So the point I am making here is that this is great for all the people in this room but I think all the people in this room are probably either trying pretty hard to do this or at least want to and/or at least aware of it, for areas like the most of the developing world which are slowly coming down the line which is where we see most of these, Malaysia telecom, like the example Randy gave, they are not aware of this stuff and their primary source for information about how to make their box work and get themselves on the Internet, is their vendor.

ANDREI ROBACHEVSKY: Right. Was it kind of implied suggestion to reach out to vendors and ‑‑ so one thing, I think that is an excellent suggestion. One thing is, and to also another point that many people in this room are trying hard to do this stuff and maybe implementing MANRS, I would love to see all people in this room on this website. The thing is, the more people are there, the more visible this community says MANRS is the way to go, that is the minimum baseline, that is the new norm in the routing. The more visible ‑‑ it's as a quality mark, so the more aware are the parties that have to play this game, or contribute to this game, the more seriously this MANRS is taken, so please support this and point taken.

REMCO VAN MOOK: OK.

AUDIENCE SPEAKER: Alexander curator labs.  Hijack, route leak ‑‑ poisoning, different things hear from the discussion and not the same with the discussion with Randy even us doesn't hear our vocabulary right, let's first define the problems define the vocabulary that industry uses and then in terms of explaining to five years old pictures, diagrams, explain to the industry why it's bad, just last week ago we mailed one well‑known operator in Scandinavia with examples of his leaking ‑‑ routes to this other uplink, affecting our, for example, and NOC didn't get it right, they are like, we don't see problems there, it's OK. And then after extending discussion, they, oh, yeah, this might have affect performance, we see it now. And it's not in Africa, it's not Eastern Europe, it's Scandinavia it still happen, first class engineers. So education is really important, but before the education we should have our ‑‑ straight. It's really important.

REMCO VAN MOOK: Thank you. So, from my side, who in this room has already signed up for MANRS? A show of hands. One, two, three, four. Wow. Who is going to look at it again after this? That is more promising. You have got your result, thank you for setting up here.

RANDY BUSH: We do all that stuff, we are just not interested in the marketing part. Our logo is not there, right, we do filter, we do blah‑blah‑blah, we do spoofing, blocking 38, etc. It's not the marketing effort; it's the delivery of the packets.

ANDREI ROBACHEVSKY: That's right and I would say this is not, at this stage at least, it's not promoting your company; that is promoting this effort. And IIJ has joined, by the way, Randy, thank you for that. I think, yeah, we are building the brand and your support is needed to say, this is a repeatable effort. I understand model of what Randy said that we should be very much careful that it doesn't translate into lip service.

REMCO VAN MOOK: OK. I am going to close down for now because next up is Thomas and I do want to hear what he has got to say. Off you go, Thomas.

THOMAS KING: So, it's me again, from DE‑CIX. I gave this presentation already as a lightning talk on Monday so this will be the extended version. But you might already have heard some of the information I am going to share with you. So this presentation is about traffic volume dependencies between IXPs and what is very important for me is that this is not about blaming, this is about understanding and that we learn what is going on. And please keep that in mind during the rest of the presentation because, yeah, it's really important.

So, from time to time we have discussions about how robust the interconnection system is in total and especially if you look at IXP world because I am from DE‑CIX and that is what we try to make sure that works well. So, on the right‑hand side you see the map with IXPs in Europe and you see that there are a lot of them outside have small ones, big ones, all different kinds of IXPs are out there and the question; what happens if a large IXP and this includes of course all the DE‑CIX, if a large IXP fails and what happens if this is the case, what happens to the interconnection system?

Luckily, there was an incident which we investigated and this presentation is about the results on this incident.

The incident I am referring to is the incident AMS‑IX Amsterdam experienced last year during RIPE, you probably are aware of it because the RIPE meeting was also in Amsterdam. The incident happened on May 13th and as you see on the traffic graph, more or less all the traffic dropped at their place and, yeah, that is what happened, but what really started us as DE‑CIX looking into that is that we saw traffic drop at DE‑CIX Frankfurt, at the same time as the incident happened in Amsterdam and if you look at the traffic levels you see we dropped about 240 gigabits within five minutes of the incident and we see that it's quickly recovered, starting ten minutes after the incident started.

And to get an understanding, you know, how this two incidents relate to each other, I show here a time flow of the two sides on the left‑hand side you see AMS‑IX and on the right‑hand side you see DE‑CIX. And the information I present here is what is available from public sources provided by AMS‑IX. So around 12:22 the AMS‑IX technical staff traded a loop with four times 100 gig which resulted in basically a blackhole where they dropped all the traffic. And if you compare that to the traffic graph at DE‑CIX you see around that time, we see that the traffic volumes drop quite fast. Then three minutes into the incident we see that 500 out of the 600 BGP sessions at AMS‑IX were dropped also, and around that time we see we reached the lowest low on the traffic graph at DE‑CIX. Then eleven minutes into the team the ‑‑ reacted and deactivated the loop ‑‑ which is actually quite fast from my point of view, eleven minutes to figure out what's wrong and then resolve the problem is good. And you see around that time the traffic volumes at DE‑CIX start increasing. And at 12:40 the ‑‑ most of the BGP sessions at AMS‑IX were back on line and you see we reached a normal traffic levels we would expect at that time.

So, we wanted to understand why are these traffic volume dependencies between AMS‑IX and DE‑CIX and we did some measurements we discussed the early findings already with quite people in the industry, I was talking to a lot of you guys to get more ideas what was going on and I will present the three main reasons we could identify throughout our ‑‑ our research work in the next couple of slides.

One thing we identified is that remote peering routers were overloaded, so you probably are aware of the remote peering router concept but let me quickly recall it. So, usually if we have a remote peering router that means the same router is connected to more than one IXP, and if this router is actually connected to more than one IXP and there is a traffic storm or a broadcast storm or whatever, it can happen that routers overloaded and then it might restart or drop out BGP sessions or whatever it does to handle the extra load. And if this happens so let's say at one IXP you have a loop which generates a broadcast storm and the remote peering router shuts down all the BGP sessions because it restarts then the BGP session other IXP where it's connected to drop also and we looked that that and we could find at DE‑CIX Frankfurt, that we are effected by this, and this forecasting was responsible of about one gigabit per second of traffic and if you compare that to the 2140 you see there is an impact but it's compared to the overall traffic drop, it's a small impact. So we looked for what else can be reasons for that, and we cannot exactly say how much traffic this second reason was carrying but we know that there is an impact and it's quite obvious ‑‑ it's quite big. The thing is that we ‑‑ the second reason is what we call asymmetric routing part we mean that if you have a laptop which is connected to AS 1 and sends a request to the server that it's connected to is 2, then the request might traverse IXP A but response back from the server to the laptop might traverse IXP B and if we have such scenario we call that asymmetric routing path. And we wanted to know how that commonly, especially between AMS‑IX and DE‑CIX to get an understanding of the traffic volume dependencies we have seen in the graphs. So we carried out a measurement study using RIPE Atlas probes and to identify the relevant ASs and customers that are connected to DE‑CIX and the reason for this traffic drop, we looked at AS to S path with traffic drop of more than 200 megabits per second at DE‑CIX Frankfurt during the time when the incident happened. And we could identify 183 of these AS to path, I mean customer that is connected to DE‑CIX so it's just passes that went through DE‑CIX and it's here, just the passes of our customers.

So, then wave second set, we were looking into ASs that were connected to DE‑CIX Frankfurt and, at the same time, to AMS‑IX Amsterdam and we could identify 323 ASs that are connected to both IXPs and because we wanted to do RIPE Atlas probe measurement so we had to look into the ASs that are connected to both IXPs but also host RIPE Atlas probes and you see nearly half of the AS host RIPE Atlas probes. That is actually a good point to ask people to host RIPE Atlas probes because it makes it easy to do measurements. If you do not have RIPE Atlas probe in your AS please find some of the RIPE guys and they will help you.

So, we took this to that and overlapped them and then we could identify 50 AS to AS routing paths which saw traffic drop of more than 200 megabits per second and which host RIPE Atlas probes. So we did a trace route measurement study on that and we looked into the IP addresses that showed up in the trace routes and mapped them to the different IXPs of AMS‑IX and DE‑CIX. We know the IP space that is used within the peering line.

And we could identify that 38 of all the AS to AS paths we were looking into asymmetric in the terms I described the slide before. So, which means that if one IXP experiences a problem, the traffic volumes on the other IXP might be impacted by that. And interestingly, 8% of all the AS to AS paths we identified which means both are connected to AMS‑IX and DE‑CIX at the same time, did not traverse an IXP at all even that they are connected to an IXP, so that is interesting to see that there is ‑‑ besides using the ISP infrastructure also private interconnect going on with, is a good thing in terms of robustness.

So, that is the second reason I showed you. The third reason we could identify but it's also hard to tell how much traffic this third reason ‑‑ or the third reason, how much traffic it was responsible in terms of the traffic drop we were seeing, is that users experienced connection errors and usually especially if the users are not geeks like us they stop using the broken Internet and do other things like getting a coffee. Less users usually means less traffic and this is also why we assume that this is reason also contributed quite high on the traffic drop we saw.

So that's pretty much it from my side. Let me quickly summarise: So the reasons for the traffic volume dependencies between IXPs, we have identified is remote peering routers that are overloaded to the incident, asymmetric routing paths and of course annoyed users who do ‑‑ who switch to other activities. But what we would like to do is get a better understanding if this traffic volume dependencies we have seen between AMS‑IX and DE‑CIX is that common all over the IXP industry so will we see same things between AMS‑IX and LINX or DE‑CIX and Netnod so we want to extend our results here, and we will also come up with recommendations in all ‑‑ that we can reduce the impact of traffic volume dependencies in terms of, you know, if you run a remote peering router please do this and that so that if there is a problem with one IXP it does not traverse to the other one. So, that is what we would like to do next and I probably will keep you posted on the updates. So thank you very much.

One final note: I am a nominee for the RIPE PC so if you haven't voted yet please go to the election website and vote for your next RIPE PC members, thank you.

REMCO VAN MOOK: Thank you, Thomas. Any questions for Thomas.

RUEDIGER VOLK: Deutsche Telekom, still. Thomas, in the list of contributing factors for traffic loss, I think there is one bullet missing, one shouldened stand that the BGP convergence process that is triggered if some connectivity is lost, is actually fairly slow and kind of the reaction time of the AMS‑IX knock is almost just ‑‑ is not really in in order of magnitude slower than the BGP convergence. Of course, the impact is very different from when, say, you have peerings that are old style STM connections and well, OK, drop of the connectivity means immediately the BGP session recognises something has ‑‑ is gone, on a switching fabric like the IXPs, losing the connectivity between parties, actually has a long lag time until the incident is reacted to and actually propagating just within a multihomed network, takes a while, as long as we are not standard by default working with S path and all the interesting new stuff that, well, is not really around at the moment. And the asymmetric ‑‑ if you have asymmetric traffic, of course the impacts are a little bit more surprising and unpredictable anyway.

THOMAS KING: That's correct. Let me quickly comment on your first hint about the BGP stuff, yeah, that's right, we didn't look into the BGP convergence time and what was going on on BGP yet, we might do that in the future, also. But there is a very interesting article from Emile on the RIPE Labs ‑‑ sorry ‑‑ on the RIPE Labs website; he just published, I think, yesterday, and it exactly goes into that direction, he looks into what happens on the BGP side and what was the reachability and where we have seen the different BGP path and things like that so, probably you should have a look on that or talk to Emile after the talk.

WILL HARGRAVE: Isn't one of the conclusions here that that product we have the ISPs have built is bad for the Internet?

THOMAS KING: No, I think that is bit fast to come up with that conclusion. I would say we have to do it right. It's good but you have to do it right. As always right ‑‑

WILL HARGRAVE: Do we need some guidelines or working out we do not create these kind of problems?

THOMAS KING: I think we should come up with best common practices ‑‑ recommendations to make sure that doesn't happen too often.

RUEDIGER VOLK: Are the customers of all services actually evaluating what impact they are buying?

REMCO VAN MOOK: I think this is a great follow‑up discussion for our next session. So if anyone feels like they could stand up and start talking about that at the next Working Group session, please come and talk to me or send an e‑mail. Thank you, Thomas. Big round of applause, thank you.
(Applause)

And next up is Emile.

EMILE ABEN: Not the address policy Working Group, I am Emile, presentation about country's ISPs and RIPE Atlas. And some you have might have seen some of this content before, it was presented at several Knox but not here at RIPE meetings.

We all know RIPE Atlas, 9,000 probes, at the moment on line, so if you start looking into this on a country level, some countries are well covered and some not so well. In yellow here, these are Africa is not as well covered as we would hope. And some countries are really well covered, so I started asking questions about can we create specific Internet measurements from RIPE Atlas and can we use these to make things better, can we detect problems?

I know this is important mantra here keeping local traffic local, so, the question I ask: Is Internet traffic local and in arbitrary simple definition for local, does stuff stay within a country? Of course, I know a country is just an arbitrary boundary and you can repeat these studies with arbitrary boundaries and you will see an example of that here.

And questions I was wondering about are: Do you see IXPs in the path between RIPE Atlas probes in a country and do we see these countries looping out of a country and then coming back to the country and in that case, does it matter, do these need to be fixed, so I would like to provide some insight into that? What I thought of is just do a measure trace routes between public probes, because you can target these.

So, in the latest iteration of the two I developed for this, I used maximum two probes per ASN because that makes stuff manageable so for instance, if you take France, which is something I did in September, half million trace routes, if do you a full mesh, that is quite a lot of trace routes. And quite a lot of waiting for analysis, so if you do one to two probes per AS and try to and get some diversity in the probes that you select you end up with 23,000 and that's manageable. You can do analysis around that.

Then, trace routes, so I have IP addresses and because existing geoIP for routes is not very good, we have open IP map where we crowd source this information so that is, I think is better. So use that to actually see if we see IPs out of a country, and for IXPs, I make it configureable in my tool so you will actually have to configure peering LANs and the tool will tell you if trace routes traversed peering LANs in a country or not.

Very important, if do you measurements and measurement study like this, look at the limitations of what am I actually measuring. So we measure path, this is not traffic volume, so this is not representative traffic volumes in any shape or kind. But my expectation is that the large volume path are already optimised. This is the stuff that everybody looks at, your top partners in net flows so you have already optimised that, so I am interested in the stuff that isn't ‑‑ that is not optimised, how automatic is it to keep it local. Furthermore, RIPE Atlas vantage point, some networks are easy to get into and some are hard so my expectation this is biased towards clue. And trace route is a limited tool, there is ICMP rate limiting, blocking, you don't seat Layer 2, you just see IP addresses, you don't see what happens in between IP addresses, keep that mind when looking at all of this. First case study was presented at Netnod, just take probes in Sweden, do a mesh, how many out of country path addresses. Do you see? I was shocked by these numbers because I think Sweden is a pretty nice market, nice IXP, clueful people, yet 12% out of country path that we measured and 21% in IPv6 even, so tried to get some insight into that, created a matrix, so this is source destination pairs. So each row is a source probe, each column is a destination probe and I coloured them by the two attributes, two times two is four, and I coloured them suggestively because I like IXPs. Green is traffic or path go through an IXP and we did not detect out of country, so that is from the IXP perspective, that is the good stuff. Orange still good, no IXP in the path. But not out of country so it could be direct interconnect or via some transit that stays in country. Blue a bit of a weird case and I don't see it that often, is where goes through an IXP but looks out of the country and back. And the red is where it goes out of country without an IXP involved, and these are the cases that would you want people to at least look at. So I also create interactive versions of this so you can hover over the cells and see the trace routes, does it only ‑‑ does it go to the US and back. URL is here if you want to read about this.

So, let's define our country border or our locality border a little differently. What if we include Oslo and Copenhagen? These are really close to Sweden and they they could be considered part of the market. If you repeat the same study, you will see this, a lot of green and hardly any red any more. So, well, keeping local traffic local is not keeping all traffic within a country, you probably already know that and I think this is a nice case that actually shows that. Sometimes it just makes sense to loop stuff outside of your country.

This one was presented at France IX, and here I have to explain the probe selection a little better so you understand why this is shaped like spider web. So these are all the trace routes, just connecting the locations that you see. This is due to probe selection because if I have more than two probes I select the probes closest to a capital and furthest from a capital to get diversity both in geographic difference and typically the capital is a big city and the further you go from there you get some rural versus city diversity. So you get a nice spiderweb shape, that is the cause of that.

But here you see quite a lot of traffic ‑‑ quite a lot of path go through Paris, of course, but you also see London, Amsterdam, Frankfurt, it's quite normal, I guess. Is this bad? It's five extra milliseconds, for some stuff it might be, for others, not. But it also shows some resiliency so if stuff within a country breaks there is a lot of stuff out of country that provides connectivity so that is good.

Here is another nice case. You can actually do this for multiple countries, so you select all the probes with this methodology for one country and do it for the other country and do a mesh between the countries and, well, Chile and Argentina are separated by the and days but there are fibre path and they must have been laid there for a reason but they are not always used and, well, Miami is a big interconnect market and we see a lot of path there, going there and you can see in the matrix here the red dots ‑‑ the red dots are these out of country paths so this can provide some insight into that and especially if you want to do business in both countries but want to ‑‑ put one data centre in, you have to be careful on where to put stuff.

And of course Romania as host country, I wanted to do an example about here. The same geographic but let's just put all these trace routes on a map. In IPv4 and the same set of probes in IPv6. In IPv4, I did not detect any out of country path but in IPv6 I see Frankfurt and Amsterdam, well, I contacted people involved in these different and they ‑‑ each said to me, yeah, we peer in v4, not in v6 but I guess we could. So you see there is still a lot of IPv4 and IPv6 non‑congruency. So, what happens if all of a sudden a lot of v6 traffic appears? In this case, happy eyeballs may not use the IPv6 connectivity, so you will not see any significant IPv6 traffic. If for this path, where it's very different.

You can also do an AS graph from trace routes, just wanted to show the example. You may notice ‑‑ and this is Romania again ‑‑ only inter LAN is here and there are multiple ISPs in this country, I am automatically getting peering data from peeringDB and some have and some not and that is the explanation for that, you can see where stuff is connected, what international parties are involved in interconnecting Romanian ISPs.

So, how to do all of this, it's a couple of scripts I created, I called it IXP country Jedi, because Star Wars is cool. It's five scripts and they prepare, measure and do the analysis and configuration is simple, you just have a single JSON file and simplest config is what I put up here, country, Romania and it will just do all your stuff. It's Git hub, that is the URL. It's quite expensive to do all these measurements and you may know, I work at the bank that gives out all these credits, so why not do all these measurements for you? So, what I have been doing for a couple of months and this one is the first one that I am reasonably satisfied with, is do runs for all countries with at least three covered, so latest run from 1st of November, has 105 countries. So I invite everybody to check these out, check your local country out, bug me with bugs. And again, IXPs are automatically filled from peeringDB for a given country, so if you want your IXP to appear in these, please list them and list the peering LANs.

Interesting related work from a former colleague of mine, Sebastian, he did this for New Zealand and he has much nicer visualisations on top of this so we intend to work closer on actually making the visualisations for the IXP country Jed die at least as cool as they are.

So action points: Network of interest not covered, like the previous meetings, apply for a probe, become ambassador, here are the URLs, please do that. For network operator, I invite everybody to explore these graphs and the visualisations and see if you can identify stuff where things can be improved. IXPs, find network operators you can bring together, peer locally or peer. And for programmers, check out this code and see if you can improve it and if you have future requests these are more than welcome. So that is it from me. Any questions?

KURTIS LINDQVIST: I was one who encouraged you to do this in the first place. I love this ‑‑ could do this for all countries on a continuous basis, completely different topic earlier this week. What I think would be really cool if you show the countries that have the biggest change over time, where IXPs actually makes the most difference and put a number on that because I think that would be cool to see what happens over time and if you go in and dig deeper into that.

EMILE ABEN: That is one of the things that we indeed could do, and for instance Flatistaf in the room? Macedonia doesn't have an IXP but will likely, hopefully get an IXP soon and we can see the difference, yeah.

AUDIENCE SPEAKER: Freddy from E net 7. How do you manage the PNIs? How do you make them ping within a country or outside of a country?

EMILE ABEN: Well, I just look at IPs and look at if they are geolocated to a specific city or not so. That is ‑‑ I don't do a map to an AS and map the AS to a country, it's just the IPs, so ‑‑ does that answer your question?

AUDIENCE SPEAKER: Well, I was just thinking, we peer PNI with most other networks or with most relevant networks within the country, and so they don't appear to ‑‑ trace routes don't go over the Internet Exchange, so I just was thinking ‑‑

EMILE ABEN: Then you would just show up as orange here, which is perfectly fine. And I have been thinking for the networks that are directly connected I should add another colour, well, then it becomes a big rainbow of colours and ‑‑ but just showing the direct ‑‑ the direct connections as an additional visualisation I think is a good idea, I can distill that from this.

RANDY BUSH: IIJ. Kurtis, the problem is, in time, the topology of probes will change and I have been scratching my head about removing the bias created by that and have not come up with any pixie dust if you have got some send it my way.

EMILE ABEN: The selection and method might, at least, keep the bias the same.

RANDY BUSH: The same probes?

EMILE ABEN: Yes.

REMCO VAN MOOK: OK. So, that's it, I think. Thank you, Emile.
(Applause)

So, before I invite Martin on stage to talk about his new toys and playing with RIPE Atlas probes and data. I am happy to announce that Matthew has been found and he will be presenting after Martin. So we might eat a couple of minutes into your lunch, but I hope you won't mind too much. I also, I am really sorry, Filiz. Can you stand up for me. When we started I completely forgot to announce that Filiz is co‑chairing the session with me. Thank you very much, Filiz.
(Applause)

And now Martin, go ahead.

MARTIN LEVY: I am very happy not to be in Address Policy. That is RIPE hack‑a‑thon project that didn't exist Saturday morning it's an idea, one of those things that sort of comes to mind, the RIPE hack‑a‑thon which was Saturday and Sunday prior to this meeting, organised by Vesna and others from the RIPE community interested in Atlas, I sort of came to it and said I have got this idea, somewhere in all these trace routes there are IP addresses that jump an AS boundary, and those ‑‑ an AS boundary is a peering, a transit, an interconnect and I am really interested in that, have been for a long time, so, here is an example that you may or may not have seen of just a listing of measurements, and they are random trace routes done by random people, and you will see where the randomness comes in later. But those trace routes, they are there, there is a lot of them, there is a lot of data, so, I pitched this ideas and four other folks thought it was a good idea and joined me so Alexander deem tree, Christian from RIPE. And we all got to sit around a table and come up with a database schema and with an idea and we got help from Emile and other people as they wandered by. Dimitri kept bringing espresso coffees. So there is lots of stuff done. We wrote code, I am going to present this and why and what we did. The motivation is what I said: Wouldn't it be nice to know where one backbone ‑‑ one ASN connects to another backbone. This data should be there. There are thousands of trace routes that RIPE app lass does, some of you have kicked them off, some of you have done one‑off, some of you have done continuous trace routes, they are just there, there is lots of them. This is mining data. They are from random sources, if you look at what Emile just showed, he said OK. Fine, I am going to select a bunch of probes in France and I am going to trace it to a bunch of other addresses, some here let's say at a major content network may have done a lot of probes in their area of interest. My goal was not to care about that. My goal was to.convince this team, and I did this, that in fact actually we don't care, let's just randomly capture trace routes and see where they go. And here is our classic sort of trace route diagram we are used to and the Reverse‑DNS gives us a clue, this is request, this is NTT, and they both have AS numbers, and although there is no reverse DNS here, there is in fact actually programatically a way of detecting the AS number being swapped.

So normally when do you a RIPE Atlas project you sort of go, oh, I need credits and I need to run tests. We did the exact opposite. We didn't run a single test or add to that pool. We just sucked in interest that pool, so thank you all for using your credits, we used your data. So, this is the simple methodology. Stand for measurement IDs, start from today and work backwards. Look at individual measurement ID and grab a ton of those trace routes. In some cases, we grabbed one or two or 20 or 100 trace routes, in some cases because of the early code, we grabbed about 20,000 trace routes from an individual measurement. We repeated this and in fact what we did was wrote this as batch code in the background. And we scanned sequentially down the trace route mapping to ASN, again a very easy thing to do because there is sets of tables ‑‑ sets of calls from RIPE and other players to do this and we placed the pair of IPs that ran over a trace route in a pairs table. We threw away the rest. Latency, didn't give a damn. Begin point, end point, couldn't care less. The time, however, was very important. So we saved away the time and that time will be useful to us in the future. We repeated this, we ran this stuff in the background. And then we went separately to match these IPs against a geodatabase, in this case Emile's crowd source and IP database because it was different than MaxMind, it had a certain focus to interconnect because people had been looking to trace routes as it was the end users and we displayed it. A quick review of the tables and the pseudoschema we used here, the measurements is the measurement ID from RIPE, by the way the address family was rather important, we realised we wanted to optimise v4 and v6 so place placed them in several tables. We saved all the IPs we found because we wanted to map them toists, we knew we would see a lot of the same IPs. We did quite a bit. We collected the pairs, the location database which at this time we batch uploaded from Emile but we want to improve that and you will see why at the end. And we saved away every measurement so in theory we can actually reverse check this and go back to a measurement ID at a particular time for a particular point.

And this is some of the data, numbers. I mean, we scan ‑‑ scanning for measurement IDs turned out to be very light wait code, ran that too much, but we grabbed a whole bunch of measurement IDs. These are active or active at some point in time, trace routes. Didn't care about pings or anything else. They could be one‑offs, or continuously running. We collected nearly 80,000 trace routes. There is a lot of map reduce work going on here because these numbers get smaller as you go down the list. Let's look at the IP pairs, when we finish the code we ended up with 11,000 IP pairs and 3300 ASs found in the end and because of the fact that we had to rely on geolocation we ended up with less dots on a map because if we didn't' have a geolocation for a particular IP seen in a trace route, we couldn't see anything.

We found a lot of coincidence dots, what that means is the AS is in fact actually the same at both ends of a link, which is great. Here is a couple of stats we found. You will notice we found a lot of, if you look on the right‑hand side, a lot of well‑known back bones with large, this is actually the sort of pseudotier one list and how many of those we found. On the other side here is a count of the number of interconnects we saw in particular cities. That may just be a function of the measurements that we found, and of the trace routes that we found, and if we run this longer we should see different locations.

So this is what it looks like, and by the way, yes, the website is there, AS number Tryst.com, Tryst, a word. It means the sort of secret rendezvous of lovers? Who thought of your backbone as a lover? Well, we are in the interconnect group, we are in the connect group, and so this was an idea to expose where those interconnects were. Now that you see the map you sort of get the idea, and guess what, this map looks right. You see sort of thicker stuff where you expect to see interconnects. We know there is lots of interconnects in Frankfurt and New York and Hong Kong and Singapore, but, you know, there aren't in answer other places and we see these dots on the map. This is sort of cool. So this is like everything we found. Here is everything we found with a bunch of lines. So, one end of the line is one hop in the trace route, the second end of the line is another hop in the trace route where an AS boundary has been found. We don't care about the insides of a backbone, it's only the edges. And this needs more work but it's interesting.

Here is a zoom in on some stuff we found. In the UK, we see a finite number of locations where interconnects occur. Yes, that makes sense. If you look at some of the ones that you can click oranges you can draw down on the information, so here in Huston, Texas, we see a Mexican Internet ‑‑ Mexican ISP connect to Level 3. That makes sense. Here in Kazakhstan we see a connection between Kazak Telecom and Trans Telecom in Russia. That makes sense as well. Here is the bay area where all the locations that interconnect show up and Emile's database is city‑based not sort of street address based so we see interconnects in here, this is 200 ‑‑ this is Paul lowality toe, this is he can Quinn wicks, Hurricane Electric, I have got to stop, I have got cool stuff to show you.

So, I will stop in two seconds. This is where ‑‑ this is where Level 3 encogent seemed to interconnect for the places that we have found trace routes, run the code longer, generate more data and more dots. Maybe. Here is a bunch of lines, here is where all the Tier1s interconnect in some way. We have a lot more work to do. It's on Git hub, ASN Tryst.com. If you want to help let us know.
(Applause)

REMCO VAN MOOK: I am not going to allow for any questions or comments because people have to run off to other things. I will allow Matthew to get up and do his lightning talk. So if you can bear with us for a few more minutes, here is Matthew.

MATTHEW WALSTER: Hi, I will make this quick. Essentially, I am annoyed with a lot of the state of our documentation in the community. I looked at a few project that is came out of NANOG and projects that came out of, say, MANRS, it's deeply technical, the words always have more than 7 characters in each word, which you have to be an English graduate to understand some of these words and I am just to foster an open approach to documentation, so that when a new network arrives at an IXP you can hand them this document and says if you want to be a smart pier with less downtime, better, less downtime and less faults, with better service for your customers this is what you need to do on this talk is just about filtering. Today, the EBGP the config BCOP which was launched, you try and read that. I understand every word in that and I get bored after about 15. Lines it's impenetratable. MANRS, I love what they are trying to do, every word has more than seven characters, it is too complex. And if you try and use a tool like IRR toolset you try and make that build, you have to have used for LENIX for five years to make it configure.

Go have a look at that site, it's something I am trying to to, try and use something called basic English, it's the thousandish most basic words used in the English language. You can contribute to it and as soon as you create a pool request, there is ‑‑ you will see exactly what I am trying to do and unfortunately, I haven't got the time to see that. You can see that, peering.readthedocs.org. I want to get people filtering so we don't get another bart or whatever happens in the next six months. Was that quick enough?

REMCO VAN MOOK: That was brilliant. Any final questions, comments? No. I do have a final general question: We have done these connect Working Group sessions for a couple of times now. Are we generally happy with the format and content? Is there stuff there that is missing or that we paid too much attention to? How are we doing? You can put in the e‑mail on the mailing list as well so that gets some use because that has been slightly under‑used as well. I would like to thank you all for your attention and enjoy lunch and the rest of the RIPE meeting. Don't drink too much and drink water in the morning.