Archives

Connect Working Group

Wednesday, 13th of May, 2015, at 11:00 a.m.:

REMCO VAN MOOK: Good morning, everyone. If you could all find your seats, I know they are not very popular but still seats in the front somewhere, I don't bite much. Good morning. Welcome to the second session of the connect Working Group. I hope you are all having a good time in Amsterdam this week. Great social last night, I am glad people have started turning up at 11:00. So, we have quite a full agenda today, we are a few minutes late but there are still people walking in.

Let me go walk to the place where I am supposed to stand.

So, my name is not ‑‑ my name is Remco, we're the lovely co‑chairs of this lovely Working Group. The agenda for today, this is the opening, obviously. Going through the agenda, scribe, introductions. This is the agenda as it stands in front of you. Any changes, recommendations, things I have missed out, anything if I did, then tough luck on you. Scribe, who is the scribe this time? Amanda. Thank you so much for volunteering. Introductions, so this is me, and that is Florence, anyone else feels like they should be introducing themselves? You are awfully quiet, I don't like ‑‑ I don't know ‑‑ I don't like a quiet audience. Then we move on to minutes from RIPE 69, I suppose you have all had a very careful read of them. There is going to be a test. Any comments? Well, thank you RIPE NCC for doing an excellent job on absolutely flawless minutes, clearly.

Now we move on to agenda item number 3 which is an update on the IX member list scheme in a by Nick and he will is a

NICK HILLIARD: Hello everybody. And I have been working with Alesia on this JSON export schema. To give a quick overview of where we're coming from, there is two points of view, the IXP point of view and participant point of view. A whole pile of IXPs export data, it's usually the same sort of stuff, but because, you know, IXPs don't always talk very well together, except in the bar after midnight, they are all sort of HTML based on CSV and it kind of all sucks really badly, which was bad from a participant point of view because all of these formats were pretty awful and some organisations who felt that they had a requirement for really large scale peering, actually had to resort to screen scraping HTML output which is really pretty ugly when you are trying to do some large scale automation.

So what we did was define an export schema based on JSON, which was based actually on previous work done at Euro‑IX and we had set up a data task force to look into this. Which defined basic information types and the work we did was to pull this all together, put it into a format and actually write code. So this is what it does. It's basic IXP information via VLAN switches, information and addressing and that was released at the last RIPE meeting in London, and we were pretty blown away, actually, by the support; it was about eight IXPs after a week with three separate code base implemenations. And that was really good.

We good a whole pile of feedback about this. The first obvious thing was, well, look, you actually have different types of connections coming into Internet Exchanges, you have route server, you have the IXP itself, you have differentiation between different types of participants, so you might have regular members, if it's member based IXP, you might have pro bone no connections, so we needed some way of dealing with this, in particular what do you do with the IXP itself, is it a member of itself or is it not, so we had to figure out what to do there.

Mac addresses, it turns out that plenty of organisations do clever things with MAC addresses. Some Internet Exchanges have multiple IXP infrastructures, I think there is some pretty good examples of this, France IX has VLANs in Paris and it also has separate Internet Exchange In Marseilles, but you can connect to the two of them, so that would be a good example of separate IXP infrastructures.

Another one would be, for example, LINX where you have LINX LAN one and two in London, and then you have some of the other organisations, or at least Internet Exchanges around the UK like IX Manchester, Cardiff and other places.

So there was a requirement to fit in that sort of thing, and the idea was that you would have a common member list for all of these particular Internet Exchanges. More contact details, and there was an open question about how to deal with the statistics.

With that in mind, we took all of this feedback on board and created version 0.4 of the JSON schema which was released to GitHub on February 2015. There was a little bit of data reorganisation, data structure reorganisation. There were two things in particular, one atom name was renamed, which is, we kind of had to do it, so I hope it's not going to cause a major problem.

Some other data structures were reorganized in terms of how they presented the data. And we took quite a clinical approach to how to structure the data in terms of making the data as atomic as possible, because if you have multiple copies of sort of things like member information or IP address information or VLAN information or any of that sort of stuff, well, then, there is, from a client side point of view, there is not going to be a clear way of assessing which is the conimical data atom inside the presentation format. So we took a clinical approach to the schema and I think it's actually going to work a lot better from that point of view.

And just in terms of the member list, you can now have peering organisations, you can have the IXP itself, you can have the route servers pro bono and there is an option to put in other types of connections in there. So this means you can do things, you can go in there and have a look, pull out all of the data, all of the route servers, build your ‑‑ build your own internal databases set up separate policies for route servers and it's all pretty convenient.

So that is what we have done. I would like to give a quick idea of what the grand vision is here.

So, I have put up a few URLs on the screen here. And if you look at the second one, if you pull it up there, it will give you a break down of the number of ASNs at the Euro‑IX member IXPs so there is 8300 ASNs, of which 4200 are unique, and that is 2.2 million peering sessions, it's a lot of peering sessions. Probably most peering sessions, you aggregate, it takes about 20 minutes so that is maybe, these are figures pulled out of the air, so maybe 350 man‑years, it's a lot of time spent on peering. And in terms of effort categorisation, this is kind of the breakdown, as I see it, that most of this is a complete waste of time. We should be automating all of this. So from the end point ‑‑ end user viewpoint, what we want to do in future is to enable organisations to have easily accessible peering automation, to that end we are looking at code libraries, so we are going to write php, Python, maybe some perl and other libraries as contributed, so what we need to do is, sit down, think about a common API structure for all of these libraries and write a bit of code. None of this is going to terribly complicated but it will give the opportunity for third party organisations to come in, pull a Git repo down into their own systems which will give them native access to all of the data in the IXP ‑‑ all of the IXP that is they are connected to.

There is also an additional advantage from the Internet Exchange point of view. PeeringDB 2.0, like some of these games due any day so this is what we are looking at. We have all of these IXPs which send data up to the various Internet Exchange organisations, which is effectively the IX‑F federation. And there is going to be an import and export mechanism to find to and from peering dB. This is a way of getting that data out to the general public. In a way which is fully automated and is actually credible. You can actually go into peering dB and get that data in a fully structured format. That is a general overview of what we are doing. The code is up or at least the structure is up on GitHub, if you just pull down that URL you can just check it out. So any questions on what we are doing?

FLORENCE LAVROFF: No question, anybody? OK. All right. Well, then, we are going to move to next topic.

REMCO VAN MOOK: Thank you, Nick.
(Applause)

FLORENCE LAVROFF: Thank you. We now have a presentation from Kurt about IXP topology.

KURTIS LINDQVIST: I am going to do 18 slides in ten minutes so there will be plenty of time for questions. So, in ‑‑ at the peering at NANOG 62 ‑ from Google made a presentation where she compared population density in the US or statistical metro areas in the US to the existence of an IXP, where if an IXP existed or not. Me and John saw this presentation and thought this was a really interesting idea and we should copy this over to Europe and do the same for European data set. We also had, when we started, we had some ideas that we could do further analysis and look at other things such as relevance to language and a few other skills you have in Europe that doesn't necessarily exist in the US. This analysis part turned out to be a bit harder than we thought, maybe.

So, what we tried to do was we wanted to analyse IXP proximity to major population centres in Europe. We also tried to develop a classification system for IXPs where we tried to group them depending on what type they are. And then we tried to, we wanted to try and combine those two data sets and see what conclusions or analysis could be done and maybe also see where we thought that the market was going and where we saw the development was going. The latter wasn't that successful, but we tried.

So, this is the map of all the IXPs in Europe and it's quite a dense number, quite a high number of IXPs, much more dense than many other rural parts if not than any other parts of the world. There is a history ‑‑ we gave this presentation at the URX forum in Marseille and it was pointed out that historically, why the first IXPs emerged was either the academic institutions where the Internet in that country was first established or where national fibre routes on landing stations happened to be and it followed those fibre paths of fibre interconnects. And that is certainly true, we are seeing in recent years trend with much more IXPs being established in much denser mesh of IXPs.

So, we went through all of these IXPs that we could find and tried to classify them, based on where they come from, what type of organisation they are, and type one was a membership based IXP, where the members actually owned the IXP. Type two was other non‑for profit IXPs, they can be owned as trusts at Netnod or industry associations or similar but still operated in a non‑for profit manner. We then had type three, which is IXPs hosted by academic institutions or service organisations. Type four were commercial data centres or ‑‑ yeah, data centre IXPs. Five, commercially owned IXPs. Six, and we had joint ventures or some sort of combination of the above.

And those were the classes we thought that broadly outlined the types of IXPs we have in Europe. If you look at this, and this is how we classify them, we see of the European IXPs and this is Europe excluding Russia, and I will come back to in a second, we saw that the type one is by far the largest group of IXPs is 39%, followed by the academics and then by non‑profits. And the interesting part is that if you combine mutual and academic, that is 65% of all the IXPs in Europe. If you add the other nonprofit organisations, 76, a very high percentage are operating some sort of non‑for profit or member‑owned forum.

So, the classification, In Marseilles, in URX, we went through this slowly and people could get feedback on the type of the organisation. The slides are available on line and so is the reports. I am not going to go as slowly through this if you find something where you have a feedback you can let us know. This is the IXPs we identified and this is not all the IXPs in Europe, it's 46 of them. I will come back to how did the selection.

So, when she did the stud see she looked at where they existed in this major are statistical population centres in Europe. We tried to find something similar in Europe to define a metro area or population centre and we turned to the authoritative source of all information, i.e., Wikipedia, and Wikipedia have a definition of the European metro regions and that's what we used, it's called European population centres. If you look at them, you can read this better than I can, but they are ranked by the population size and we mapped out the IXPs to these regions and you find some interesting facts here that on the 9th place, ninth largest which is nape he wills in Italy, there is no IXP. The second one ‑‑ fourth in largest region, which is Birmingham or West Midlands, there is no IXP. I am not sure what is the black square ‑‑ I don't know. And then we have Cologne ‑‑ if you go to the fourth largest ones, we have Valencia, Glasgow and Seville that are all missing IXPs. And less and less we have.

When we started doing this we for a while thought we should look at also capitals or major centres that are missing this but then you end up looking at places like Malta and so on which is not so large areas to look at.

So we did this mapping and you see there is quite a few of these that are missing IXPs and the largest regions that you find is the rural area, whether it is missing or not. If go strictly of the population cetre as described Wikipedia, we decided to stick to the same definition of the areas, followed by Naples, Birmingham, large cities in some of the European countries and few of the large Spanish cities except Barcelona and Madrid also missing IXPs.

This is why? A very obvious one, there is no national ‑‑ natural network topology in the cities so it might not make sense to have an IXP there and again, as we saw the historical reason why the largest IXPs ‑‑ they could be market conditions, either very, very dominant local players or formal Monday op lease, regulatory environment, etc., there is a number of market conditions that might affect this. And the last is proximity to other existing IXPs and if you have a very large IXPs, very close by, it might not make sense to establish a second one just kilometres away.

So we did look at this and went and looked at the largest areas missing an IXP and the distance to other IXPs. Nape else, for example, is not that far away from Mcex in room, 225 kilometres. Birmingham is relatively close to LINX and loan app in London and Manchester. We have the Cologne area and Dusseldorf and and DE‑CIX in Frankfurt, 54 kilometres. Valencia is not that far, it's a bit further away from CATNIX. And this is just an observation to these numbers we have, and again, for example, 54 kilometres could be inside a metro area, so it's fairly close to the Glasgow. But that is the idea ‑‑ again distance might not be everything in this, either. Some of the other factors playing in like market conditions and ‑‑ might have a much more dominant role in this.

So how is Europe comparing to the study ‑‑ US study. The 19 largest CSAs in the US all but one had IXP. Eleven of them are major peering hubs that are quite defined in this presentation as major peering hubs. The smallest of the 19 would be the eleventh largest in Europe so there is clearly a scale shift as well here in terms of population centres.

So, Europe, one conclusion or one of the few you can make, Europe seems to have a much denser mesh of IXPs but we are missing more IXPs in large centres than the US did.

So, this is how far ‑‑ wrote a report about this, that you can go, last slide has the URLs for it, you can go download and read this and when we presented this last time there was a few ideas of how to develop this study. One was to investigate IXP proximity to GDP population and that doesn't necessary indicate high economic growth or activity. And we put that on the to‑do list and complement the study with that.

The other suggestion that was raised was to correlate this to neutral data centre availability. It's an interesting idea, just really hard ‑‑ how do you find this data set of the neutral data centre and what is it? I think you all agree what the neutral data centre is but finding the authoritative list is much harder. We will look into it but at first glance wasn't so obvious to do. You can find a report either at the LINX website or I didn't get the full URL for Netnod website but I will update the slides later today and give the report there as well to download. That is the last slide. Any questions?

REMCO VAN MOOK: Any questions? Did you have your coffee this morning?

KURTIS LINDQVIST: Plenty of coffee this morning, yes.

REMCO VAN MOOK: Anyone? In that case, thank you, Kurtis, and John for this.
(Applause)

Up next is Florence. We have CDN best practices, an update from the previous session.

FLORENCE LAVROFF: So good morning, everybody, today I would like to follow up on the talk that I did at last RIPE meeting in London at RIPE 69, about recommendation for ISPs to interact better ‑‑ so you will certainly remember that, the objective of our last talk was to actually create and enhance our win‑win relationship with the ISPs providing more recommendation how to optimise content delivery within the ISPs networks we work in.

So, what has happened since then? So a couple of things happened since last November when I last talked to you. On the Google perspective, what changed is that we launched some new features via our ISP ‑‑ delivering many to our notification page, we addressed via some recommendations how to make things work better with the ISPs we work with. And I could see actually that the feedback that we have from this new feature actually worked pretty well so we had some responses, it is helpful for ISP and it really encourages to go deeper in this direction. So, it was good for us to see that they are responses that this is helpful and we thought that we will definitely continue this, and I personally think there would be value to enlarge this to OSC to other CDNs and content providers.

So I think that we go the structure right and this is the structure now that I would like to share briefly with you. So I will go very quickly through that, so as you can see this is work in progress. Still, for now. So the first category of recommendation that we would like to address are BGP prefix announcements and traffic engineering. So no big surprise that this is what we talked about last time, so please announce all your prefixes and use traffic engineering wisely in a way that you know makes sense for you and your CDN content provider as well. That is the use of prefix specificity, AS path length etc.. if you know that your content provider give high preference to specific prefixes, we really encourage you to advertise via peering and your caches, prefixes which are at least as specific as what you advertise via transit, just an example.

When you get your BGP prefix announcements right, it's already a good thing. Then if you have a enough capacity and diversity within your network to carry this traffic to your end users this is even better. So please talk with your content provider and CDN about your scalability plan, diversity, failover plans, etc. So it could be a idea when you see that your PNI are getting close to 70, 75% capacity, to start discussion to upgrade them. It can also be a good idea if you have different PNIs in different locations.

On caching point of view, of course consider the capacity that you have with your caches but do noter get to consider the uplink to the caches, this is also important.

Please talk to us, with your K D N, with your content providers about that, that is really important, about what you really want to achieve in terms of strategy and where your network are located. If we can help new some ways, if we see that it hurts we can help you.

The other categories, I think that you all expect this, IPv6, please dual stack for sessions to peering and caches and please, please, please keep your peeringDB records updated.

So, this is me standing in front of you presenting this but I would like to be very clear on this, this is no Google initiative. This is a project with other content providers and other CDNs. I am only one member out of our project support team. So I just would like to give some names here of the people who are helping to work on that. So we have Neal Becker from Akamai. We also have Ren Provo from Apple. We have Nena from Netflix. Thank you. We have Joshua from Microsoft. Stand up. Thanks.

We have mike Hughes from Limelight. Thank you. And we have Mars from Facebook who is not there with us, unfortunately, today. So I think that this is pretty much it for today. So our next steps, of course, are to discuss this as a first step, and then after we have synchornised we would like to get back to you with the outcome of our conversation via the mailing lists. And then we will invite you to comment on this. That is it. Thank you.
(Applause)

REMCO VAN MOOK: Right. Any questions for Florence? Oh, come on. Honestly.

Nena: I am Nena and I am too tall for this one. From Netflix. I just want to stress that this is, the way we presented it right now is very much, hey guys, we are going to come here and tell you guys how to do this and that is not really the point. So I would like if people who are interested, who would like to say I need this from the CDNs because I really don't know this, don't know what to do if would you use the mailing list already now and tell us what do you actually need, with a would you like us to tell you in order to get this framework together. So that is just my call to the other side, the ISP, please tell us what do you want to know, because we have a lot of it information and we can talk for hours about this. And we can write big huge documents that nobody will ever read and we really don't want to waste time doing that. So please, use the mailing list and telll us what you need. Thanks.

REMCO VAN MOOK: Thank you, Nena. Any ISPs or essentially the receiving end of this best practice who want to say anything? Or are ISPs going to write their own best practice document on how CDNs should connect with them.

AUDIENCE SPEAKER: We could have another document and take a vote which one will go.

REMCO VAN MOOK: And definitely send it to the FCC. That sounds lovely. OK. No more questions, comments, remarks, tomorrow ate toes, bricks? All said and done. Thank you, Florence. And up next...


(Applause)

REMCO VAN MOOK: Up next is Keith Mitchell talking about Open‑IX and how hard it is to set up a new community event.

KEITH MITCHELL: Good morning. So, today I am wearing my Open‑IX hat, I have been a board member there for about a year‑and‑a‑half now. As you are probably aware Open‑IX is a North American initiative and to a large extent it's a consequence of the fact that within Europe we manage to get the peering and data centre co‑location ecosystem working quite well. Some of that was as a result of hard lessons learned and some was as a result of luck but a lot of the founding principles with Open‑IX are principle that is you probably know from the EIX Working Group which is probably /PRED certificates of this Working Group and also of Euro‑IX. We have a model where there are neutral Internet Exchanges and they are separate from the data centre operators. No in North America it's very different, there are a lot of exchanges where there are very ‑‑ their services are very closely locked into particular data centre requirements and part of the Open‑IX mission is to try and receipt row actively apply some of the things that have worked in Europe, to try and fix some of the challenges in the North American market.

I am going to skip over stuff that is common knowledge. Within the US market the data centres tend to be differentiated into two places, extremely large ones postally for ‑‑ hosting content but aren't necessarily very richly interconnected and then there are the large delivery ones which tend to be in the major met poll tan areas and closely associated with particular commercial co‑Lowe providers and that is where the latency applications are happening.

Open‑IX has had its origins two or three years ago. We have been functioning as a fully qualified legal entity with nonprofit status for a bit over a year now. Three main activities, defining standards for interconnect, whether that is the actual data centre or how you run an Internet Exchange; enencouraging the em /PHREGS and a certification to /PRO* /SES where Internet Exchanges and data centres can opt‑in and demonstrate that they meet the standards.

So, why was it necessary? There was a number of players in the ISP market in the US who felt they didn't have a lot of choice and felt forced into particular concentrated facilities and that was making interconnection more expensive and difficult than it needed to be. The idea with certification if you are an Internet Exchange operator and you are in a particular metro you can have a choice of which data centres you work W you know that these are going to be up so certain technical standards and openness standards. And the idea is that once you have the competing ecosystem of data centres within a given metro like in Europe without forcing you to have your Internet Exchange in a particular place, there is going to be competition and improvement there.

So, the idea is that there is less concentration and density, and more choice and more resiliency, costs are lower and the Internet is a better place.

The governance model for Open‑IX is perhaps slightly different than in some of the Internet Exchange bodies that you have come across. It's open to anybody as an individual you can basically pay your 50 dollars a year and you get to vote in the elections, you get to potentially become a board member, you can serve on the volunteer committees and you get to review or provide input on the certification applications.

Certification, anyone can apply for certification. There are two standards, the certification is separate for each. There is one for the Internet Exchange technical requirements, there is another one for the data centre requirements. And there is ‑‑ we are working on the cost, because there is some issues with people who want ‑‑ bodies who want to certify a large mm‑hmm of sites. It's 5,000 per market share per year.

And the other thing is you can ‑‑ the standards are there, you can aspire to adhere to the standards without necessarily going through certification process, obviously we would prefer if you do.

There is a bunch of community, usual /S‑RPBGTS I am sure many of these familiar are familiar to you (suspects) I am one of the legacy boot strap directors but most of the board is elected and the rest will be elected at the AGM this autumn.

And then co‑opted on to the Board are the Chairs of the various standing committees, and I am sure many of you know Henk who is on the IXP standing committee and there is ‑‑ a membership committee, which is ‑‑ does the mechanics of making sure that members are on board, a marketing committee to do outreach particularly into the data centre market and a Programme Committee that was used for the ‑‑ on the IS one interconnect conference we had in San Diego last year. So two standards, one for Internet Exchanges; a lot that have will be very familiar to people in this room in terms of what is good ‑‑ what is a good way how to run an Internet Exchange. It's about being contactable, it's about keeping your peeringDB information up to date, having transparency on your website. Data centre standards is obviously more about infrastructure, how is your building constructed, where are there the power and data feeds, is there appropriate redundancy there, do you have all the operating procedures in place, do you have service levels, what are your power and cooling arrangements? And I think a very important one is open non‑discriminatory access on pricing. It would be a gross mistake if Open‑IX ever got involved in saying this is what you should be paying for your data centres. It's rather about insisting if you want to comply with your standard you have to price, you have to publish what your prices are and offer the same prices to everybody so there is no back room secret deals, and we believe that that transparency promotes competition.

So in terms of certified bodies, of course the interesting thing is most of the Internet Exchanges in North America that have been certified are in fact operated by European1 Internet Exchange operators who are expanding. So, there are a few up and coming US‑based Internet Exchanges in Florida, Phoenix, we are still working on certification with them. And then there is huge take‑up within the data centre industry, I mean ever since I have been on the Open‑IX board I get phoned up by analysts who are in the data centre investment business and, you know, I am not going to tell them which data centres they should invest in but I think it's really encouraging that all of these independent neutral players have applied and got certification. And you know, the other thing you will see the most recent certification was jaguar networks so that is our first European certification. Open‑IX's focus is very much on North America but we are completely open to operators in, anywhere in the world, participating in the standards process, becoming part of the organisations that comply with the standards or indeed apply for certification.

So, this just gives you a map of where they are, obviously there is an interactive version of this on the website. And that is pretty much it. You know, it's an open organisation, we have official nonprofit status and we are trying to both execute and encourage best practice in the Internet Exchange. So I will be happy to take any questions.

REMCO VAN MOOK: Any questions for Keith? Anyone? No one. It's very quiet. I don't know, I am shocked, really. I don't know this audience like this.

ANDREI ROBACHEVSKY: I have a question. So you referred to ‑‑ on your website, could you elaborate maybe how the standards were developed and how people came participating in the development of this?

KEITH MITCHELL: They were developed in a very sort of bottom‑up community based way. The starts committees are entirely volunteers, they were ‑‑ at the start we listed who wants ‑‑ who from the members wants to be on the committee, and you know since then there has been a lit bit of vetting and refresh to make sure we have actively engaged people, not just people warming their bums on the seats and essentially, they came up with the standards and these standards were then written by the committees, put out to the members for review, once the members were happy with them then we have a public mailing list and were put out to that and then once all the feedback came in from that then the board endorsed the standards. There will be probably be some iteration of the standards but once we got them set the first time we don't want to tinker too much with them, but there may be more, for example, there is some talk about an Internet Exchange lighter standard for start‑up Internet Exchanges, this is something that people keep something for. But, you know, basically, we are open to more proposals to standards or for discussion or revision of the existing standards.

ANDREI ROBACHEVSKY: And the processes document on website, the ‑‑

KEITH MITCHELL: Yes, the standards are on the website.

REMCO VAN MOOK: So Open‑IX organise the America's interconnections summit, as far as I know that is the first time there is actually been a community let attempt at setting up peering event. How did that go? What are the lessons from that?

KEITH MITCHELL: Obviously we are biased but it seems to have been successful, we had nearly 200 people participating. There was two days worth of a combination of presentations and also meeting maker sessions where people could set up, there was an automated tool to let people set up one‑on‑one meetings. Very good sponsor support, you know, and whether we do another one or not is going to be as a result of community feedback, so if you found it valuable and would like us to do it again, then we will certainly consider doing it again. It was ‑‑ we are nonprofit, it was effectively cash neutral event for us to do.

REMCO VAN MOOK: All right. Anyone else else, questions for Keith? In that case, thank you very much, Keith.

KEITH MITCHELL: Thank you.
(Applause)

REMCO VAN MOOK: Now, we move on to the next part of the connect Working Group session, we are going to have three presentations on the topic of IXPs in developing markets and afterwards a short panel session, write down questions for the three presenters. The people will be Bijal, Jane from ISOC and Andrew from Tereco. First up is Bijal. Go ahead, Bijal.

BIJAL SANGHANI: Hello. Thanks for the intro. Nice to see a full room here. I am going to be, today, talking about some of the support work we do for IXPs. So, I think everyone in here ‑‑ does anyone in here not know who Euro‑IX is? Thank you. That is good to know.

REMCO VAN MOOK: They have been very quiet all morning so don't take anything from this.

BIJAL SANGHANI: What do we do. Euro‑IX is a membership organisation for IXPs and one way that I have heard it been described before is it's a self‑help society, and really, that is exactly what it is. We ‑‑ the IXPs come together and they support each other and help each other grow, share ideas, experiences, and I am just going to run through the different ways in which this is done.

So, we have two forums a year and these are now growing fast as well, we have had the latest forum which was In Marseilles, we had 125 attendees and over ‑‑ it was actually 42 Internet Exchanges attended the forum. So, that is definitely growing, and also it's not ‑‑ it's not something that you see all forums or all conferences that you will go to, such a large number of Internet Exchanges all in one place. We also do various different ‑‑ going back to the forums, it's a two‑day meeting and there is lots of different topics that are discussed, which can help the IXPs so things like regulation, we had a really interesting session on regulation at the last meeting, we talk about tools, some of the work that Nick has talked about on the JSON schema that has been developed at Euro‑IX. There is always information shared about route servers and what are we going to be doing with these. So it's a lot of information and for new IXs coming to the meeting they learn how IX is actually run ‑‑ already are running their own operations.

We do various types of communication: You know, there is ‑‑ I do presentations, explaining what Euro‑IX does and the different projects that we work on. We have mailing lists and that is another great resource for members and we have different mailing lists for people that are specifically interested in regulation or route servers or tools. And we have newsletters, so we have been writing newsletters for over a year now and it's great to see that the subscription there has grown as well and we have information about what is new things that are being done from the IXs. We have blogs from the forum and also information about past events that we have been to.

Another support item that we do is exchange programmes and this is where IXPs within the membership can actually visit another IXP, and this was ‑‑ one of the IXPs that have done this is AMS‑IX and PTT and Netnod have done it as well where the Kenyan IXP and it's kind of a two‑way thing. You may think that it's the one that is going over to learn is the one that is going to gain the most from it but the feedback I have had is also that the IXPs that are already well‑established are also learning because they are getting asked questions that they may not necessarily ask themselves during ‑‑ when they are working normally. So that has been really good and it's a great way for IXPs to learn and support each other.

We have a benchmarking club and we run this once a year. It's done by volunteers and again this is getting ‑‑ we are getting more and more participants on this. In 2014 we had 42, which meant the results were a lot more interesting and there was a lot more information and data there for us to share with the members.

Something that we launched in the last forum In Marseilles was a fellowship programme, and this is ‑‑ this is a new funding programme that we are ‑‑ we have launched and the ‑‑ and it's actually the members that contribute themselves into a funding part and then this will be used for either members who need support attending the Euro‑IX forums or for emerging IXPs so that they can get introduced into the community and it's a really good opportunity ‑‑ way, for them to learn and see what is going on in the IXPs scene.

So, that is a summary of the kind of support things that Euro‑IX does.

But it's not just ‑‑ the support things that the IXPs are doing are not just through Euro‑IX, there are many IXPs doing their own support and I am going to go through a few of these.

So, for example, we have ‑‑ for starters, we have got AMS‑IX who provided hardware and training and also they have Bastion, who is doing a great job in providing some regulatory support for the IXPs.

DE‑CIX, again they have done training in Angola and provided operational support, and they are also taking part in the Euro‑IX twinning programme.

So the twinning programme is a programme which is developed to help support IXPs that need help, so IXPs that are kind of just newly established and want to get on to the ‑‑ a standard level or being able to support each other and be self sustained, they can join the twinning programme and we twin them with IXPs who can help support them grow.

France IX have done a lot of good work in the ‑‑ in Africa, we are doing some work with ‑‑ that was done through ISOC on the axis programme. Nick from IX and I am sure if you give him a drink he will tell you a fantastic story of when he went out to Gambia and this is an ‑‑ and also INEX are doing a lot of support with IXP Manager, and I think from my records there is over 15 IXPs that are now using IXP Manager. So obviously INEX are providing the support there as well.

LYONIX, again with the ‑‑ in Africa they have done a lot of work with ISOC and the axis programme in Cote de ivore and Brindi. And one thing that Samuel, actually, when I mailed him and asked him for this information, what he said it's not actually the technical part that is disks it's actually the community building, bringing everyone together in one room and sharing the ideas and educating people on what an IXP is and how the community can come together and really make this work.

And Netnod are busy as well giving training in various different locations and also provided a lot of support and they are also twinning with /PHOZ a.m. beak, the Kenyan Internet Exchange and in Congo.

And that is it. Thank you. Are there any questions?

REMCO VAN MOOK: Any questions specifically for Bijal. If you have more generic questions we can save them for the panel session. Thank you, Bijal.
(Applause)

Next up is Jane.

JANE COFFIN: Good morning, thanks to RIPE, and particularly Bijal also at Euro‑IX and some of the IXPs in the audience for all of your help with what we are doing around the world.

I am with the Internet society, if you want to know what ISOC is just ask me later, I won't get into that right now. We are doing work globally with our teams that live in countries, speak local languages, know the local environment, around the world, and that means Latin America, the Caribbean, parts of Europe, this is not done by ourselves of course, it's with partners. Again with the people who live in region but also with Euro‑IX, RIPE, other RIRs. We do this work with partners, as I said, but also through grants. We have our own budgeting for this but we couldn't do what we do without some of the equipment grants, some of the human donations of time. We have done work with the African union and I will talk about that in a minute. Cisco have given equipment, both of those have given lots of equipment to us to help dough mate to level that out.

Comcast is doing work with us in Latin America.

A tipping point for us have been these grants to level up and boost what we have been doing. We have created an IXP tool kit which we would love information from any of you in the room, and feedback. This is on the website here. There is a portal, obviously. Partnering to do training in the region, we recently were in month neglect row, working on measurement studies in Latin America and Africa and to see what ‑‑ the way we do a lot of this work is through layering of discussion, workshops and meetings. It's the human, the technical and the governance infrastructures that we are strengthening so it's people and training, equipment and probably a little jet bag on my part, it's talking to governments about training to lower the duties on equipment coming in. It's amazing, we sent six months, six months later two are still in customs. That is big tricky but the bottom line is they don't really know what they had, they wanted more money from us to liberate this, we had to work through that. Egypt it was 10,000 dollars to get a switch in and that is after the donation. So what we do is try and forecast how much we are going to have to spend to try and get the gear in.

On the management and operations side, this is important, Bijal mentioned twinning, which is partnerships, bringing in companies and IXPs and what they are doing, have been through certain processes but can help what they want. I am almost done. There are about 15 more slides, I am going to click through a couple. You can see through this later or ask me questions. Here is some of what we are doing in other regions. In Latin America, this just shows you some of the workshops, the launches, the levels up, the case studies, again, thanks to the different grants we have got.

You would be surprised in Bolivia where we were trying to do some of that work, Rasberry pies ‑‑ something simple, the government thought they were spy devices and we couldn't get them into the ISP networks, it was a little tricky, we went through three ministers in one year. There is patience entailed in all of this and lots of work on the ground to explain and reexplain what it is we are trying to do. We are not trying to do anything fancy just trying to get some data, prefixes, routes, things like that but that is new in certain places to obtain that data. So it takes a bit more time than you would imagine.

Europe and the CIS, this is through partnerships. Middle East, tougher climate, working to try and do what we can there. Pictures of some of the IXs and some of the partners we have, Nigeria down there, a bit more sophisticated, engineers 24/7. IXPs are also for many of you looking at different environment to invest in, a perfect way to host your routes, to put in also ccs, certs and other. You can see here in the picture, it's a small IXP, trying to help boost it, not much activity right now but you stay in for the game and you try and work with them for the longer term. We do often use examples from Europe obviously, and as Bijal mentioned there are a lot of partnership going oranges different things we have seen and the impact of IXPs I think you can see here, there was 100 dollars megabit per second after the ‑‑ after that a dollar per megabit per second. Some of the impact is huge. When come bass is a comes into a city in late inAmerica and average enty, costs come down, it's before they have installed the IX.

With Africa, you can see here, of the 31 IXPs that are operational, less than 50% are publishing their stats. So one of the best practices we are trying to work on, which is something Euro‑IX has also identified, is the importance of putting up your data on your sites. So a study that we are going to be looking at and working with RIPE to try and help them get the Atlases deployed as well, is to try and gain more data working with some coders to put some servers in the different IXPs, five this year we hope, it will be available to everyone that is interested in it. We gave this presentation at Open‑IX and I had been asked by a certain consent delivery networks, how many in Africa running more than 10 gigs of traffic. We said maybe three out of 31. So there is a completely different environment that you are looking at, and if you are talking about a gig to 10 gigs, what is in between? Why are we not talking about two, three, five? Why is it there is this barometer of one to 10? We have often heard that people in some environments want to sayer get one, we want to focus on 10. You are eliminating half of Africa if we do that. We can give you data oranges just ask us, we have got lots of information. (Data on).

That is it. All right.

FLORENCE LAVROFF: Thanks. Does anybody have questions specifically for Jane right now? Will, I see you. /HAR /HAR I think when you people are asking about looking at traffic routes and asking about the value, the problem you are having is the traffic routes have the wrong thing on the Y axis, the mega bit figure needs to be replaced by a dollar sign and suddenly an IXP in Africa doing 20 gigabits in a market where IP transit is 20 bucks a meg or more, looks very different in terms of that traffic volume and people tend toor get this, especially people from the west.

JANE COFFIN: It's a really good point and actually, I had interfaced with one of the fellows doing MP R G graphs and indicates some of the information they require and are not requiring would have been more helpful to have up there as far as different statistics. They have money to try and do this. But this might be something we could work on because there is definitely ‑‑ you have to come at this in advantage of what is in it for you, right, in the country. And it's obviously quality of services latency is hopefully going to go down, traffic will go up, we have seen this time and time again. And it's the he can could system that is being strengthened as well, the human ecosystem, better engineers and people who could work on some of these problems.

FLORENCE LAVROFF: Anybody else? If no further questions specifically for this presentation, then I recommend that we move to the third speaker.

JANE COFFIN: One last thing for content delivery networks that I would stress, when you are trying to work with an IXP to encourage them to bring in your cache, it's really helpful to us that we often donate equipment to those IXs through our grants (cash) help them understand why you are suggesting a certain level of equipment that is needed. Sometimes they don't understand and we'd rather make sure that the configurations we are giving them is going to last more than a year. Ghana they went from 800 MEGS to 1.5 gigs gigs of traffic, we will be donating more to them, it's something to think about with standard configurations.

FLORENCE LAVROFF: Thank you. We have our third speaker which is Andrew Owen.

ANDREW OWEN: Good morning. Thanks for the comment about the graphing, we don't call them megabits, we call them mega bucks per second. I want to talk quickly today about the state of interconnection in peering in Africa. It's sometimes difficult to speak about Africa outside of Africa because we don't always have very good story to tell. But it's getting better. One of the things that was mentioned earlier was IXPs starting to work together and we finally have an established Africa Internet Exchange forum which we call Af‑IX. And one of the primary tasks of Af‑IX to begin with was to get data about exchange internets in Africa because for a long time there wasn't an authoritative source of IXP information in Africa, for a long time we just didn't know how many IXPs we had, so the data collection process is, it's an ongoing process, so, you know, some of the stats are not 100 percent complete. What we know is that we have ‑‑ actually in our 31 operationally active Internet Exchange points in 22 countries out of 54. And we have managed to count, so far, a daily combined peak of around 52 gigabits per second of traffic or gigabucks, as I mentioned. The continent has got large region networks so there is a lot of networks that can carry you anywhere in Africa with over 700,000 kilometres of terrestrial fibre reaching 40% of the continental population. Multiple cables with over 24 terabits per second of landed capacity into major centres. And what we are seeing is neutral facilities are a reality now, but the concept of neutrality is starting to gain traction as well, which is great. And the community is working together, the exchange point community is working together and it's thanks to the establishment of /A*F IX. So multiple IXPs in key hubs, I think South Africa we have got six Internet Exchange points, and the amount of traffic that I mentioned earlier, probably around 80% of that traffic is in South Africa. Kenya, I think, has got two Internet Exchange points and Nigeria, I think, that is 3. We are seeing larger traffic numbers and member numbers in the capitals now instead of being concentrated at the landing stations and yes, one of our exchange points are up and to the right.

So, some more stats that we got from Af‑IX and the data collection that we are trying to do now is there is a nice balance between industry associations and non‑for profit organisations that own the Internet Exchange points, which is great. The peering models, majority of them follow a sort of European model where multi lateral peering or route serves are operational at the exchange point but they are not mandatory. And we have seen very high penetration of route server use. I know that in South Africa we have got upwards of 96% of around 192 members, peering members that are peered with our route servers.

One of the most interesting stats that came out of this data collection process is that 65% of exchange points in Africa do not charge any fees. And, you know, 30% of those don't have any /PHRABS to ever implement any (plans) to ever charge any fees, so that was quite interesting for us.

So what is holding us back? I am sure you have all heard this before but bandwidth costs a hell of a lot of money in Africa and it's different all over the place. To give you an example, the most recent example that we saw was in Swaziland, which is inside South Africa. We saw bandwidth pricing of around 1700 dollars per megabit, that is via the incumbent telco. You can't buy from anyone else. But you can understand the limitation that that puts on market.

There is, in certain areas there is a lack of skill, but it's not just necessarily a lack of skill but also a lack of understanding of what peering is and how it can benefit people. People tend to get stuck in the comfort zone of sending everything via transit. So, you know, convincing people to do otherwise becomes a bit of a challenge. And then of course, age old problem, we have many countries still operating government owned incumbent Telcos, and this all leads to not necessarily a limited market, I mean there is definitely a market but it becomes a challenging market because, instead of just having to explain to people or convince people of the benefits of peering and interconnection, you now are having to convince them to invest in resource, in skills, in prefixes, in ASNs and that process is quite ‑‑ it's quite daunting for new entrants and more importantly, it's also quite an expensive process, especially for the smaller guys that are trying to get in.

And then of course, everybody knows about our interconnection problems. It's just ridiculously regulated and very expensive to get interconnection going. We always get the example of Jabouti where it costs you 18,000 dollars a month for one gigabit to cross‑connect. And then we also have ‑‑ there is some government involvement in Internet Exchange points which is not necessarily a bad thing but, you know, we have some people sort of promoting the idea of establishing regional Internet carriers, so a sort of Monday op lieisation on a continental scale.

And then on to some of the bad things. One of the unseen prohibitors of growth is power, you know, nothing that we do is possible if we don't have power. So just some examples: In South Africa, which is widely considered as one of the economic hubs of Africa, we don't have any reserve capacity at the moment so for the last couple of months we have been experiencing almost daily load shedding or rolling blackouts where some areas being without power for up to five hours a day in South Africa. And, you know, it's a difficult process, because building power stations is not something that can happen overnight, and, you know, we are in the process of doing that but we are about five years behind schedule because of labour unrest and union involvement, so it looks like South Africa is probably only going to break even in 2018 which is ten years after we ran out of power. So it's getting there; it's just a very slow process.

Kenya is, looks a lot better, they have still got 20% reserve margin but they are forecasted growth at around 5%, going forward, where their real growth is 8.3% so they are definitely going to run out of power.

And in Nigeria they have given themselves or they have forecasted they need to provide around ‑‑ 40 gig watts of power by 2020, and realistically, the analysis is they need about 200, so it's ‑‑ it's just very far behind. I mean, they are way behind their current peak demand at the moment so it's quite a difficult situation.

So, yeah, just in summary, Africa is a very, very big place but there are certain centres where it is very easy to come into the market, and, you know, what I tell everybody, it's quite an open canvass at the moment so it's great that everybody wants to be part of building that. Thank you very much.
(Applause.)

REMCO VAN MOOK: All right. Thanks, Andrew. Any questions specifically for Andrew? It's still very quiet. John, please go to a microphone. I will bring you a microphone and you can introduce yourself.

ANDREW OWEN: Yes, the lack of questions is disturbing so I have got one for you. We heard two previous presentations about what is being done to try and help fledgling ISP, if you could have a magic wish what is it you would like the more well developed IXP community to do for you, what would be on that wish list, you can interpret the question any way you like?

A. ANDREW OWEN: It's different for different areas. We have seen a trend of people just saying well here is a small IXP, I have got a hell of a lot of kit I can give to them, this should get them going. It's not about that, you can give anybody a switch and say start an Internet Exchange point. It was mentioned in one of the other topics, was, you know, a lot of the places need help in understanding how to build a community, how do you get a community together, how do you get people to start seeing each other as trusted sources. So, yeah, you know, the hardware donations are never a bad thing but there is a lot of ancillary stuff that needs to go with that, that is training, sort of keeping in contact with people and making sure that they are doing the right things and I think just most importantly, building a community, teaching people how to work together and giving them an idea of what their goal should be.

REMCO VAN MOOK: Thanks. Does that answer your question in any way? OK.

JOSHUA: Joshua from Microsoft. With the predominant use of route servers in Africa is there any sort of consistency or effort to do them better? The route servers in general are administered ‑‑ odd trips are used, Microsoft has come out and said we won't go to a route server unless we absolutely have to because it just ‑‑ I mean, because it causes performance issues and routing problems. So, with the use, is there effort to try and get them on the same version and try and get them doing things that aren't going to negatively impact the routing?

ANDREW OWEN: This is a discussion that has come up amongst the community, so what we found is that even though there is a lot of route server usage, you know, a lot of the infrastructure was put in a long time ago and it was put in to get everything up very quickly, and back in those days when you had, three, four, five, members at the exchange, it was a case of, I know all of you, so I am not going to do any filtering on the route server; so you can pretty much receive anything and send anything from the route server.

So yes, we are aware of that in those specific instances, and certainly there is ‑‑ there is work being done to try and make it better.

You know, because a large part of the problem is that, remember a lot of the entrants that are coming into the market now are very new at this, so, you know, if you go to a large exchange in Africa and this is a very new wet behind the ears guy it's going to be difficult for him to contact 100 members as a new bee to turn up at all of those sessions so it's kind of seen as a very low barrier to entry, you know, because he only has to deal with getting one session up where he ‑‑

AUDIENCE SPEAKER: It does make it simpler if they are doing things like rewriting next hops or inserting ASs, then they are making it worse on themselves in the long run.

ANDREW OWEN: Yes, but, at the same time ‑‑ I understand where you are coming from and it is something that we are aware of, certainly, at this stage I think people are seeing it as a quicker way to grow.

AUDIENCE SPEAKER: Right. That is what I am saying, is there a recommendation that saying you want to do route server, great, here is a simple way to do it, don't insert an AS or rewrite next‑hop, have communities and ‑‑

WILL HARGRAVE: Do we need a route server like a quality, stamp to say it's any good. Best practice document, yes. Talking about quality and filtering and stuff like that, some of the status even in Europe is terrible.

REMCO VAN MOOK: So given the time now I would like to open up to questions from others as well, I still have your question, Keith. And I do know at RIPE 71 we will have a presentation from Microsoft on why route servers suck. That is good. If I can ask the other two presenters and come forward and go to a microphone and ‑‑ Keith.

KEITH MITCHELL: You use the term council power in your presentation, which I am not familiar with. Is that what Americans would call utility power?

ANDREW OWEN: I am not sure. So we have one provider which is S com, and S com sells power to the municipalities who are then responsible for distribution.

REMCO VAN MOOK: So the answer is yes. Thank you, Andrew. So, one question from my side to all three presenters. So, we are seeing development in ‑‑ of IXes around the world and we are also seeing challenges on where you can develop them. How is the development of IXPs related to the deregulation, does deregulation of telecoms help? And is the first one to deregulate in a region the winner when it comes to interconnection? Who would like to take that on?

JANE COFFIN: One thing we are seeing is that there is a shift in emerging markets as to the what the role of the regular layer S they have gone from a telco environment, where they were looking at tariffs, fibre, maybe infrastructure sharing. We are trying to promote the idea/suggestions that as a regular layer you are trying to promote investor, bring in better connectivity, remove some of the barriers, i.e. do you have ‑‑ landing stationings, are you looking at trusted fibre costs, do you have infrastructure sharing regulations, if you are putting in new roads or rail roads or whatever with liquid, are you putting fibre in or allowing it to be put in. It's complicated with some of the regulation because they see a nail and think they should regulate it. What we have been trying to do is suggest the IXPs reach out to some governments or some data centres to say would you like to be on a board, can you give us some idea, that way they can be part of the IX and learn a lit more and hopefully back off and not try and regulate. There are some knee‑jerk reactions we are seeing. With snowed enthere there are a lot of people think IXP is a place to monitor traffic. When we talk about measurements and statistical analysis that can get super tricky so we are just trying to balance it out.

REMCO VAN MOOK: Andrew, as I understand it South Africa got deregulateed from telecoms perspective in an all mighty hurry. How did that impact the market while you were in there?

ANDREW OWEN: So interestingly, that is around the same time that we started our operation. And yeah, as Remco says it was a court decision, you know, a specific group took the regulator to court effectively because they just weren't willing, they kept on stone walling the deregulation and I think this was probably around 2008. And you know the court made this decision and they had to deregulate and it was overnight, you had ‑‑ tens, almost hundreds of registered providers that were suddenly digging their own fibre into the ground. I mean, you couldn't drive anywhere in South Africa without seeing somebody digging up a sidewalk somewhere and it's just amazing. As I said, the Internet Exchange traffic that you are seeing in Africa, 80% of that is coming out of South Africa and it's sort of become, you know, one of the entrance hubs into Africa is through South Africa. So it was the reason, it's the reason that our business exists today.

REMCO VAN MOOK: Thank you, Andrew. Any other questions for the panelists before I close off? Anyone else?

Salam Yamout: Salam Yamout, member of the RIPE NCC Executive Board, but I am talking with my hat as found are of the Beruit IX in 2007. I want to echo what Jane is saying, we are not at the same level of discussion as Euro‑IX. I mean, our IX has been founded by the community, bottom‑up, in 2007 and our biggest problem are our government. The regular layer, the government and the fact that Internet is not run in a bottom‑up or multi‑stakeholder, whatever you want to call it, manner. So, I don't know what to say, I mean what to do, so we can at least get to the same level of technical expertise or even at the same level of bandwidth and of capacity so we can start talking to each other.

REMCO VAN MOOK: OK. Thank you. So, with that, I would like to close this part of the session. I would like to thank all three presenters for their input. And yes, thank you very much.
(Applause)

It is 12:28 and we are at point number 8, which is where I ask this room, since it's only the second session of the connect Working Group, are we heading in the right direction? Was this all a waste of your time? You have been very quiet this morning; I get slightly uncomfortable with that. Is there anything we can do to improve this session, this Working Group? I also noticed the mailing list has been mostly silent except for some e‑mails from me and Florence, is that a good thing, a bad thing, a feature, a bug, let me know. Anyone, please? Oh, well. I will look forward to ‑‑ ah, I get two people standing up, lovely.

PETER HESSLER: Peter Hessler from Germany. We are a smaller player in the market, mostly with content and from our perspective it's not entirely clear in how we can best cooperate with this Working Group. It's interesting and we definitely want to have more connections but we are not running IXP, we are not ‑‑ we are not founding anything, the German market as you know is very well established. So, from the smaller players' perspective, what else can we do to to encourage communications and what should our expectation be for it?

REMCO VAN MOOK: Well, let me turn that question around, what would you like the others in the room to do for you when they connect to your network? Would you like a pony, a unicorn, maybe? You can come and stand here at the mic phone and tell me what you would like to see. You see the CDNs telling them what they would like to have, we have had our presentations about voice‑over IP and other stuff. So, it's really, this is an open platform to talk about anything that is related to connecting stuff together.

So, I hope that sort of answers your question. I am answering a question with a question which is a very diplomatic way of doing stuff. Yes. No, maybe, OK. Somebody else?

AUDIENCE SPEAKER: From the Danish engineer exchange. I just say that content is good, it's interesting. And I think we are going in the right direction with this.

REMCO VAN MOOK: Good. Positive feedback is also welcome, absolutely, thank you very much for that. Anyone else? I have one final request for a single slide from Marco, so I am going to ask Marco to come over here and say what he has got say, which is about this.

MARCO HOGEWONING: I am in between you and lunch, disclaimer, RIPE NCC although involved in organisation we have no particular involvement in this session or these efforts. There is a net neutrality session for up copping euro dig which will be in June in Sofia and the organisation group behind that is preparing a statement to come out of regarding net neutrality, a draft text is available on the URL which is now on your screen and the organisers are very eager to receive your feedback, which can be left on that website so if you can spare sometime, please have a look, review and leave your comments on the website. We thank you to participate. All right. Thank you.

REMCO VAN MOOK: I thank you all for your presence, enjoy lunch and sigh next time.
(Applause)

LIVE CAPTIONING BY AOIFE DOWNES RPR

DOYLE COURT REPORTERS LTD, DUBLIN IRELAND.

WWW.DCR.IE