Success at Apache: The Leaseweb Fork

I joined the Apache CloudStack community in 2012 and became a committer in 2013, eventually becoming a PMC (Project Management Committee) member in 2017. My journey to becoming a PMC was both physical and literal and included several forks in the road. The forks presented themselves in all aspects of my journey – although the literal forks came later, mainly because my journey began in China.

In 2010 I was working at China Mobile, the world’s largest mobile network operator, in Beijing as the manager of a cloud project based on OpenNebula (another Open Source project). A year later, my partner received her PhD in the Netherlands and began working in Belgium, so I started looking for new work opportunities in the area.

In 2012 I visited Europe and had a few interviews, but it was difficult at the time as my English was quite poor.

I was committed to finding a good job and moving to Europe, which meant I needed to improve my English quickly. I studied the language, left China Mobile, and moved to Belgium permanently. It took me seven months to became fluent in English. I re-interviewed at the companies that had rejected me seven months prior and landed a position in Amsterdam at Leaseweb as a Cloud Innovation Engineer. At that time, we had two public cloud platforms based on Apache CloudStack. I was mainly working on the research of Apache CloudStack.

In the first two months, I fixed some bugs we found in our productions. Thanks to the CloudStack community, I also received tons of help as I began contributing my changes to the mainstream. In 2013 was invited to be a committer, 3 months after my first submission. It was a huge surprise and a massive honor for me, and I began pushing my changes for new features and bug fixes much more quickly.

A year later, in 2014, Leaseweb released its first private cloud based on Apache CloudStack. It was very welcomed by many customers. As this was happening, we began finding issues with CloudStack as customers were requesting more features and functionalities. The same year, Apache CloudStack moved code repositories to GitHub.com and started using GitHub pull quests to review and merge commits. While all commits should be reviewed and approved by other commits before they are merged into the mainstream, we had already made many changes at Leaseweb and could not wait for the next release. Because of this, we created our own fork containing all our changes and bug fixes.

We developed very quickly, and our process was much faster than the review/merge process of Apache CloudStack. The gap between our fork and the community was getting bigger and bigger. When we decided to upgrade from CloudStack 4.2.1 to CloudStack 4.7.1, we had to spend half of a year just to port all of our changes in our fork based on CloudStack 4.2.1 to new fork based on CloudStack 4.7.1. The same problem happened again when we tried to upgrade to CloudStack 4.14, and we had to spend around one year to port all of our changes. The lesson we learned from these two upgrades was that we needed to contribute more to the community and maintain a fork as small as possible. After realizing this, we contributed all of our features and bug fixes to the community by creating many GitHub PRs. Some PRs have now already been merged into mainstream, while others are still in review.

My colleague recently asked me, “If you could go back in time, would you still make the Leaseweb fork?” My answer is yes, I would do it again. A fork makes us more flexible, as we can offer more stable production and more functionalities to our customers. However, if I could go back in time, I would have spent much more time contributing our changes to the community. I’ve learned that the gap between the fork and the community should be less than 100 commits.

We learned so much from these two painstakingly long ports and have implemented the above advice. From now on, the Leaseweb fork only contains features we have developed in the past and bug fixes. For new features, we will always contribute to the community and deploy to our production only if it is merged into the mainstream. By doing this, we will be able to upgrade to the next CloudStack release much easier and will benefit more from the community (e.g., more bug fixes, more features by other contributors). When we contribute to the community, we also benefit from knowledge sharing and the contributions from others.

Wei Zhou has been an Apache committer since 2013 and a PMC member since 2017. He has a Masters in Computer Applied Technology from the University of Science and Technology of China, and a PhD in Computer Organization and Architecture from the Institute of Computing Technology, Chinese Academy of Sciences. Wei specializes in all things computers and has over 10 years of experience in software development. He is a Principal Cloud Engineer at Leaseweb.

This article originally appeared on the Apache Foundation’s ‘Success at Apache’ blog, found here.