LeaseWeb’s annual Amsterdam TechSummit took place on June 2 at the Pakhuis de Zwijger, an old warehouse converted to a high-tech multimedia event center. The summit was sold out with over 315 attendees who came to hear a variety of presentations from professionals focusing on this year’s theme: Designing for Scalability.
Those who attended were a diverse assortment of software developers, operations engineers, and managers from companies both large and small. Many of the attendees were local but a good percentage of them had traveled from other countries including Germany, Spain, and even as far as Liberia. All of them were looking to learn about ways to help them grow not only from a technology perspective but how to scale up their engineering teams and how to anticipate and deal with the issues that result from that growth. The summit also provided a good opportunity to network with peers and learn about the challenges they face and what they’ve learned from past mistakes.
Terrence Ryan, a developer advocate at Google, gave a talk entitled Containing Chaos With Kubernetes at LeaseWeb’s TechSummit in Amsterdam on June 2nd. We sat down to find out a little bit more about his thoughts on the topic.
Interviewer: What issues are facing engineering departments who have just moved to containers?
Terrence: One of the large issues I’ve seen is how you manage and keep track of them all. Containers are ephemeral, so there is the switching over to the dev practices that supports that.
Having applications and architecture that is fault tolerant in the sense that these containers go away and that should be ok because the data is stored persistently somewhere else. All the app is doing is computing stuff and sending it back to the users. One of the big challenges we’ve seen and one that Kubernetes tends to solve is, “I have all of these containers, how do I keep track of them?” Those are the two problems we see come up. Kubernetes solves the management of the containers.
Read the rest of this entry »
In this blog post I’m going to cover how we are scaling our engineering department at LeaseWeb: where we started, the lessons we’ve learned, and how we are hoping to move forward. LeaseWeb was founded in 1997 and we currently have over 350 employees throughout the world with the majority working at our headquarters in Amsterdam. The engineering department currently has about 100 employees and manages 65,000 servers in seven locations (with more to be added soon).
In the past development was based on shifting priorities rather than planned business cases. Developers weren’t able to concentrate their efforts on thoroughly building, testing, documenting, and presenting a demo for one project before another more urgent one was moved to the head of the line. Operations and Development were working on different schedules and weren’t able to meet each other’s requirements to their mutual satisfaction. We wanted to change this.
Last year, we merged our existing operations and development departments into one Product Engineering department. Since then we have been focusing a lot on coaching all 13 teams and improving their effectiveness
In october last year, we attended an excellent talk by Bol.com at Velocity Amsterdam. In this talk they explained their ongoing transition towards DevOps. One of the concepts they introduced, was a maturity model to measure and incentivise continuous improvements within a team.
Inspired by the Bol.com talk, we have since developed and implemented a maturity model within our Product Engineering organization which consists of a matrix of four levels in four categories.
In this 2-part mini-series, Joshua Hoffman examines some of the common issues companies face when designing for scalability. Read part 1 here.
In my previous blog I looked at what I call the first three sins of web scale – pride (the refusal to use tools not invented here), envy (the desire for a more exciting project) and gluttony (ignoring scope and capacity). Today I’ll discuss the other four sins you need to be aware of when building and deploying your app or product. So without further ado, let’s check them out.
Throughout my career I’ve had the opportunity to work at a variety of different companies both large and small. They each had their own set of unique challenges regarding growth but one thing I noticed with time and experience was that the solutions to the problems they faced were not specific to the company itself. The approaches that were taken and the lessons that were learned could be extrapolated and applied to many of the situations facing a company looking to expand and grow technically.
There is a concept in some religions that before you save a sinner you have to tell them how they have sinned. In other words, if someone doesn’t know what the problem is they won’t be able to change. For a company just starting out, there are no wrong ways to build and deploy your app or product. Once you begin to grow however, you realize there are things you didn’t know and that some or all of the decisions that you made at the beginning were mistakes. This is the point where you need to decide how to address these issues. New companies are started all the time so I decided to draw from my experience to put together what I call the Seven Deadly Sins of Web Scale using seven real world examples from my career.
Recently I was reading this article in the New York Times about Minecraft. It’s a story about how Minecraft is changing the way children play, learn and create things. It does so by bringing them into a digital environment that provides the freedom to let them fully design their own world, complete with houses, vehicles and more. Players start mining and expand their environment by chopping trees, mining blocks and creating their own tools. In Minecraft, the article goes, you’re provided with a toolbox to do so, which allows you to be creative and build things. The physical equivalent of Minecraft is somewhat like Lego.
The theme of TechSummit 2016 Berlin and Amsterdam is “Designing for Scalability”. But what do we really mean by the word “scalability”?
Well, we’re all familiar with the standard definition along the lines of Wikipedia’s definition: “The capability of a system, network, or process to handle a growing amount of work… A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.”
In the past two years we’ve witnessed various events that have had an impact on the open character of the Internet. In October 2015 European Net Neutrality rules were published, providing guidelines for regulation, but they were criticized by many as being too open and leaving too much room for uncompetitive behavior (here’s an example). In June 2015 the FCC published its US Open Internet order along the line of “no blocking, no throttling, no paid prioritization”, driving a significant change in the IP Interconnection landscape especially. In parallel, we saw ongoing consolidation on the side of the ISPs, with large ones absorbing their smaller competitors or other players in the digital value chain (e.g. cloud hosting services, “Over-The-Top” – OTT – video services) or even merging with mobile providers. Another trend we saw was the launch of services for which the related Internet traffic is not counted towards the “monthly data budget” of the customer, typically referred to as “zero rating”.