Playbook covers our best practices and gives you the best insight how we deliver successfull long-term software projects.
Hello! We are NECOLT. We have been working with product teams around the world to build large scale long-term software projects since 2006. Our clients range from large international organizations to startup founders. We also contribute to open source projects, run community events and grow talents at universities.
Through our ten years of operations we’ve been able to achieve a level of technical excellence which has then allowed us to hugely contribute to the success of our clients. After working with us, most of these clients have gone on to attract millions of users and considerable investments, and became partners with such top global players as Apple, IBM, and Electronic Arts, with most eventually getting acquired or going IPO.
The majority of our clients have very little engineering staff in their offices. They prefer to hire us instead of an in-house engineering team because by choosing us they can:
We have spent the last ten years perfecting our software skills using only the best practices available and we are always striving to develop and improve. This book is the result of that learning. We have put it together for everyone who is interested in how we have been able to consistently deliver successful long-term projects. For potential clients, it’s a window into our process, and for potential new hires it will show them how we can help them grow. While for other companies and specialists, there might be some insight or tip that they can apply to improve their processes. All this information can be found in the following 9 chapters:
We grow and learn new things every day. This book is a live document that will change over time. Feel free to read it again later and find something new. If your interest lays more in the technical details, feel free to read our Guides.
This book was written by Saulius Grigaitis. He is the Founder and CTO of NECOLT, an associate professor at Vilnius University, active community member, and open source contributor. He focuses on software process and systems engineering at NECOLT. This book is the result of the NECOLT team’s ongoing quest to find the best way to build successful software projects.
We love to work with projects that suit us perfectly because our goal is to provide the best experience for our users, clients, and team. Through our more than 10 years of practical experience we have been able to isolate and identify the attributes that all our best projects share. For us, it is the opportunity to build lasting partnerships with our clients through rewarding long term projects that drives us forward and defines who we are.
One of our biggest strengths is our ability to deliver large scale projects. Since the launch of our company in 2006 we have focused on what software process improvements we could make that would enable us to deliver and scale 5 000 - 50 000 developer hours projects. We also work on smaller projects, but bigger projects fit us better, because they employ all the best practices we have learned. While there are a lot of companies that can handle small projects, only a few can handle larger projects. We want to stay on top.
We typically work with a client for 4-5 years. This process could potentially last even longer, but most of our clients are simply acquired by top companies within about 5 years of commencing collaboration with us. It is really challenging to deliver and scale high-quality software over such a period for such fast growing and dynamic companies. This is the main motivation for us. Only some companies can satisfy clients over extended periods. We pride ourselves on being one of those.
We mostly build new long-term products. These products have one shared attribute - changing requirements. In most cases, it’s not possible to fully design a final product upfront for a number of reasons. It could be because it’s not clear what the users really want, or because business rules have changed since the delivery of the first version of the product. For this reason, the ability to support fast changing requirements is at the core of how we have designed our process. Our entire process is change driven.
We move fast, making use of a lot of the experiences we have already gained. Our best scenario is when a client has previous experience with software projects, as this helps us to maximize efficiency. We ask new clients about their previous experience with software projects. This allows us to understand better how quickly we can move with a particular client and how good a fit the project is for our team.
We prefer to work with clients who have at least some overlap with Central European Time zone. Our “No Managers” culture gives our developers direct access to clients, so at least some business hours overlap is preferred. We also prefer real-time communication, which is also why some overlap of business hours is needed.
Web applications backend. We mainly use Node.js and Ruby on Rails for web applications and the development of APIs with fast-changing requirements. Django / Python is a better fit for web applications which demand first-class Python ecosystem integration. We also use Deepstream and SocketCluster frameworks for real-time applications and subsystems. We prefer GraphQL for the APIs we build. A lot of other technologies such as Nginx are used in our backend stack.
Mobile applications. Our clients demand the best mobile application experience. This is the main reason why we have developed a lot of native iOS and watchOS applications with Swift. We tend to choose React Native for cross-platform applications when both iOS and Android support is needed. This approach still needs a significant amount of native extensions for each platform, but the majority of the code is shared across iOS and Android platforms. We rarely do native Android only applications with Java Android SDK because development costs are usually significantly higher than iOS apps.
Augmented Reality applications. We use Wikitude is most of our Augmented Reality projects. If it’s not the best fit for a particular project we use ARKit or other SDKs. We also implement custom augmentations and other graphic solutions with SceneKit and OpenGL ES.
Databases. Most of our projects use a combination of relational database and key-value store. PostgreSQL is a reliable relational database choice for most of our projects. It also has a lot of excellent features such as full-text search, native JSON support, and many useful extensions. PostgreSQL can be used as a key-value store in some cases, but almost all of our projects use Redis as it has a very extensive feature set. We also use Memcached when it’s a better fit than other technologies. If a dataset is too large for the afore-mentioned technologies, we use Cassandra or another big data solution. Mobile applications have special database requirements, and we mostly use SQLite and Realm.
Search. Elasticsearch is our preferred technology for more complex search cases when the databases mentioned above can’t handle what’s required efficiently. Some of our clients prefer 3rd party solutions such as Algolia. Usually, 3rd party hosted solutions aren’t flexible enough, so this approach works well when a project needs only those features that a 3rd party solution offers.
Artificial Intelligence. The software we build gets smarter and smarter every year. Most of our machine-learning solutions are implemented with TensorFlow. It’s the most feature-rich library for deep learning with neural networks. It also integrates very well with the other Python ecosystem components we use.
Infrastructure. The majority of our large-scale web applications runs in AWS because it provides an extensive set of reliable infrastructure components. Smaller projects run well with less sophisticated infrastructure providers such as Vultr and DigitalOcean. Our infrastructure is automated with Chef and Terraform.
This is just a taste of some of the technologies we use. The technology world moves fast, and we are constantly extending our stack to incorporate the best new technologies as and when they appear.
If you want predictability, you need planning. Good planning is essential for both our clients and us. Planning helps us to set weekly goals. It also gives the client a clear idea of when what they have requested will be delivered.
We have weekly Sprints. Every Thursday we close the previous Sprint and plan the next Sprint. Our team lead does quick Sprint planning:
Planning meeting takes around 10 mins.
We don’t usually receive complete specifications ahead of development. That’s good because full specifications can soon become out of sync once development has been started. Instead, we create User Stories as the project progresses. A User Story is a use case description, preferably from the perspective of a user. Some User Stories may be defined in other ways if this works better for a particular case. That’s ok as long as both the client and the team understand what needs to be implemented.
The User Story is set of changes that need to be performed in the current system in order to improve it for a user. This attitude helps us to welcome any amount of changes because the intention of every change is an improvement for the end user. Our entire process is designed for such change driven development.
There aren’t many restrictions on User Story content. The Product Owner can submit any type of content (description text, images with UI elements, prototypes, etc.). The only requirement is that the provided information should clearly describe the desired change so that the developers can implement it and testers can verify it.
We use Story Points to estimate Stories. We used to use an hour based system, but this never worked as well as Story Points. Developers are too optimistic and underestimate the time needed to complete the task, and this means that an hour based estimation never works. Developers, however, are considerably better at understanding a task’s scope and complexity. And it is for this reason that the Story Points system works much better.
We use a Point sequence that is similar to the Fibonacci sequence for Stories:
Most of the Stories are estimated during Sprint planning meetings. Planning Poker is used for collective estimation. Every team member individually estimates each Story one by one. Estimations for Stories are then discussed. Much of the attention is dedicated to analyzing why some team member has assigned considerably more or considerably fewer Points to a particular Story. The final amount of Story Points is assigned to Stories and saved in the Redmine planning system.
We also perform a quick Estimation meeting after our Daily Standup if some new tickets have been added to the current sprint. In such cases, we return Stories with the lowest priority back to the Backlog.
We deploy to the production environment multiple times each day. Client and users do not need to wait for new features until the end of the Sprint. This approach has benefits for both our users and our team. Users receive new features quickly. This also encourages us to maintain our software process at the highest level so that we can ensure fluent deployments multiple times a day.
There is also another important benefit here. We don’t need to plan releases. We simply deploy each new feature when it’s ready. This approach saves a lot of time and other resources.
One of the most interesting things about our company is that we have no managers. Instead, we have such a transparent and automated management process in place that there is simply no need for managers. We have employed a set of practices that enables us to manage large long-term projects without managers.
We have a short Standup Meeting every day at 11 AM. Every team member briefly presents what she/he has achieved between the last Standup and now, and what she/he is going to achieve between now and the next Standup and what obstacles, if any, there are in the way. The Standup usually takes 5-10 minutes for the entire company. The point to each presentation is that they need to be as focused, dynamic and brief as possible. This helps to concentrate the other team member’s attention and by doing so optimize the effectiveness of each presentation. Daily Standups work as a quick way to synchronize who works on what parts of the system and who can help to complete more complex Stories.
We have two types of retrospectives. The first is a quick internal retrospective every Thursday after the standup meetup. The second is a little longer with each Product Owner (usually a client).
During our internal retrospective, we provide a quick overview and highlight the most important aspects and progress in every project. Also, we discuss what new improvements we should introduce. This retrospective works as a weekly status refresh at the company level. This retrospective takes 5-10 minutes.
Retrospectives with a Product Owner are usually done via Video call, or a face to face meeting when possible. Every team member provides an overview of what she/he has achieved during the last Sprint. We also discuss the next Sprint plan. All other important aspects such as newly signed contracts, raised funding rounds, or usage increase is highlighted too. This retrospective takes 5-20 mins once a week depending on the team size. This retrospective works as a weekly status synchronization at the project level.
We don’t need a manager to track team and developer performance. We do this automatically by tracking Velocity. Velocity is the most important metric for individual and team performance. Velocity is simply the sum of completed Story Points per Sprint. We track both team and individual Velocity.
Team Velocity works as a great indicator of how the entire team is performing. We discuss reasons for Velocity change during Retrospective Meeting if there is significant Velocity change. Team Velocity also provides a strong and consistent base for prediction for future deliveries, because it tends to be stable in the long term.
Individual Velocity works as a great way to figure out personal performance for every developer in the team. This both helps us to identify individual performance changes early and motivates developers to complete planned Stories in the Sprint.
Our team speaks fluent English and communicates directly with clients via text chats, video calls, emails, project management tools and other channels. We eliminated managers between developers and client. This decreases possible miscommunication and quickly builds trust between client and the team.
Open Space encourages communication between team members, so there is no need for a manager whose role is to eliminate gaps and join loose ends. The team naturally solves such cases. Open Space also naturally encourages open discussions involving other team members, especially those who have more experience with the subject. Working together in a common space also provides numerous other benefits, such as a feeling of equality between the team.
There are numerous benefits to simplicity. It’s easier to understand simple solutions. It’s easier to extend simple solutions. It’s less expensive to maintain simple solutions. The complex is the opposite. So why are most implementations complex? Because while it’s hard to build simple solutions, it’s easy to build complex ones.
We try to make the design as simple and elegant as possible. We use a few rules to help us achieve a simpler design. The first is to choose the simpler design if there are a few options. Then, you need to choose a testable design. Try to verify that design before implementing it. Do not over-engineer. Choose good metaphors. Use appropriate design patterns. Discuss your design with the team and improve it according to their feedback. It’s essential to simplify the design if others find it hard to understand. Refactor to a simpler design when it becomes too complex. Avoid premature optimization. Never forget that your ultimate goal is the simplest possible design that meets requirements.
The Rails Way and similar approaches work well only for smaller projects. We rarely use such approaches. We dedicate special attention to architecture from the beginning of development because most of our projects successfully grow and become large long-term projects. We prefer architectures that allow us to succeed in the long term. Our Guides cover the great architectures we use for successful long-term software projects. Such architectures bring numerous benefits: it becomes easier to extend and maintain functionality, it’s easier to swap old components with better new alternatives, it’s easier to scale, and it’s easier for new hires to understand code base. We also tend to use approaches such as Domain Driven Design that can be applied to all the platforms we work with from web backends and frontends to iOS and Android applications. All these things help us to reduce risks and costs, not to mention increase team member satisfaction.
Design patterns are an essential part of our software design. This doesn’t mean that we apply design patterns to every piece of code. It means that we try to structure our code in the best possible way with the help of great design patterns. We maintain a great balance of simplicity and more complex design patterns.
NECOLT Guides has an extensive design patterns section. We actively maintain this section and everyone in the team follows and applies it. This set of patterns is used mostly in the software we deliver. We also use less common patterns that are not added to our Guides.
Software rarely remains unchanged for a long time. New features and improvements are constantly being implemented. Old features are removed. Optimizations are performed. Sooner or later, even the simplest or most elegant solutions become too complex. This increases costs, complexity, and the risk of introducing bugs. When this happens, it’s time to refactor and get back on track.
We do a lot of small refactorings often. Such refactorings are part of the new functionality. Developers are encouraged to make such small isolated improvements whenever it’s necessary. Small refactorings usually don’t influence the entire code base, and as it doesn’t take a lot of time, there is no need for a sophisticated process.
Sometimes a system needs to be changed significantly. For example, when a business model has been significantly altered, and some of the entities have fallen out of use or been changed significantly. Large refactoring is necessary if those entities were used heavily in the system. Such refactoring is not included in User Stories and is scheduled as a dedicated refactoring task. We prefer to dedicate time to such refactorings when no other refactorings or big features are under development. Otherwise, this leads to a complex integration process.
We value high quality, easy to read and easy to maintain code. It’s not easy to produce such code, so we’ve got a set of tools to help us.
We also use code quality analysis tools such as the code smell detection tool Reek or the structural similarities analysis tool Flay. These tools help us to identify where some pattern or some other structural improvement should be introduced. We also try to stick with default configurations as much as possible for higher quality.
We take these tools seriously because most of the time, they provide huge value and help us to maintain our high standards.
When it comes to bigger projects you often find that there are multiple people working in parallel on multiple new features, such parallel developments present challenges. One of the biggest challenges you face is the compatibility of new features with each other.
We have tried well-known branching and integration models, but none of them have worked well for high-quality releases multiple times a day. Git Flow’s release branches require a lot of effort to maintain the entire release process when features are released multiple times a day. Github Flow is suitable for releasing multiple times a day, but it’s very inconvenient for QA teams to test separate feature branches and at the same time test compatibility with the other features under development. We ended up with our own model and named it Git Double. The main concept behind Git Double is that it has two main branches develop and testing, which hugely simplifies the integration and testing process.
This approach has many advantages:
Bigger projects are too big to be held in one person’s mind. Some teams choose to delegate particular parts of the code to particular developers. This is a very risky approach. Knowledge gaps develop over time. Code becomes inconsistent and of lower quality.
We use an entirely different approach. Any developer can change any line of code in the codebase. No one owns any part of the code. This leads to numerous benefits in the long run. More developers know more about the entire system. Everyone is encouraged to improve any part of the system. It also highly reduces the risks of knowledge gaps.
Code reviews are one of the most important contributors to high quality code. This process also helps you to spread knowledge about the project code. It develops developer skills too - during reviews developers learn new things from each other.
The entire review process is very lightweight and supported by GitLab. The author pushes changes to a remote feature branch and opens a Merge Request in GitLab. A continuous integration server runs tests and code quality tools, and reports build results to GitLab. At least two people review every pull request and leave comments in GitLab. The team lead is one of those reviewers. The author then improves the code according to both the developers and automated reviews. Merge Request is closed only when build passes and accepted suggestions have been implemented. Finally, the feature branch is merged using Git Double.
We rarely perform exceptions to this flow; it works great both for features and small changes, so there is no reason not to follow it for every change.
Some issues are way harder to solve than others. It’s better to have extra brains for such issues. Pair Programming works very well in such situations. Everyone in the team is free to ask another team member to join a Pair Programming session. We don’t code in pairs a lot, but we are happy to do this when it’s really needed. We found that Pair Programming leads to much faster and higher quality solutions for the most complex issues. We also found that it’s not necessary to use Pair Programming for all code because most problems aren’t usually that complex.
Another reason why we don’t perform Pair Programming full time is that we perform a code review during the Merge Request process. The code here is reviewed by at least two people. This process has the likelihood of even greater efficiency as two people are involved in the review here as opposed to the one reviewer during the Pair Programming session.
We write a lot of automated tests. We maintain very high test coverage because it’s one of the most important things for the success of large long-term projects. Such projects are just too large to test quickly manually. Developers need to get fast feedback for their code changes, not after the days or weeks it takes for a manual tester to verify a new build. That’s why we automatically build every commit to give fast feedback.
Backend unit tests take up the largest portion of the test suit because they are the best ways to write thorough tests for isolated units. Also, these tests run faster than higher level tests, so we can afford to have a lot of unit tests even when we need fast feedback for code changes. We also test a lot in other levels including automated acceptance testing. New tests are written when a bug is found, and this helps us to avoid regressions.
Our test suits contain thousands of automated tests. It’s too slow to run this on a developer machine, so we have a shared cluster. Everyone in the team can run the complete test suite on this cluster and get test results for their changes in a few minutes. It’s a very convenient way to test local changes before committing them.
All tests must pass both on the testing cluster and the continuous integration server before they are merged to the main branch. The continuous integration server builds feature branches, and this means that all build issues are solved before they are merged to the main branch.
High code coverage is not enough. The test suite would be able to check only a small part of the code base even if the code coverage tools report full coverage. One of the best ways to verify test quality is through Mutation Testing. This technique slightly modifies the program then runs tests against a large number of slightly mutated code bases. If mutations still pass the test suite, this indicates low-quality tests.
Mutation Testing is very resource intensive. Fully Automated Mutation Testing consumes too many resources to verify each Merge Request. We use a semi-automated process to make the most of Mutation Testing. Our manual tester creates an optimal Mutation Testing scenario for each Merge Request, executes it and reports feedback to Merge Request authors via GitLab. This technique allows us to optimize the results of our test suites.
Automated testing does not guarantee that the software meets the highest level of quality that our high-profile clients expect. It’s hard to catch details such as element alignment with automated tests. A human can identify such type of imperfection much easier than a computer. The tester quickly scans the UI and can quickly tell if everything is ok. It would be very hard to specify an entire UI in automated tests and then maintain it for a fast-changing system.
The original developer is not the best person to find edge cases for the code she/he implemented. That’s because a developer tends to use the software in the way she/he implemented it. It’s hard to hit edge cases in such a way. A manual tester didn’t write that code and will most likely try to use it in a different manner than its original developers. Also, a manual tester will try to use it as a regular user, not as the individual who developed it.
We found that a combination of both automated and manual testing is needed to provide the kind of high quality software that our clients demand. We will continue to perform both until something better is invented.
We grow talent. We believe that growing talent at NECOLT is the best investment we can make. This is in line with a few other interesting findings:
We use no external or internal HR staffing solutions. Potential candidates reach us organically. Most of our great candidates come to us via:
We have a lightweight and highly efficient hiring process, which helps us determine the answers to the following:
The process looks like this:
The candidate only moves on to the next step if she/he has achieved significant results in the previous steps.
Our sales are purely organic. New projects organically arrive in three main ways:
We don’t need sales people, because:
We may change this process at some point, but so far it has worked very well.