DevOps and behind the scenes

av | feb 19, 2019

Eser Gözcü is one of the DevOps Engineers at Purple Scout. And he is quite sure that the world doesn’t need yet another article explaining what DevOps is. Instead, he wrote one on what the DevOps approach can provide to your business. Consider this article the truth of the IT business and a glimpse behind the scenes of DevOps.

Background story

I have been traveling around the world for the last decade, mostly for business purposes. Most of these visits were for field engineering; solution deployment/design, project management or high-risk change management during nightly maintenance windows.

In most cases, a business traveler is on a tight schedule and runs from meeting to meeting, therefore does not have the time to discover the city itself. However, I find myself very lucky to have had such an experience. It is not only because I had a chance to meet with many people and got to taste excellent food but also because I saw how a similar task is done differently in different regions/companies. Moreover seeing how a product is being used at a customer site was enlightening. This is because developers generally do not see how the product is being deployed in the field where the customers’ expectations might be different than the original design purpose.

 

BEST EFFORT != BEST PRACTICE

Deploying a project might take a couple of months and requires a good plan. A Very high-level project plan can include such steps as

  • Identify the technical requirements/use cases
  • Check the feasibility of the use cases in a lab environment. Prepare the configuration.
  • Meanwhile, check the release guide (if applicable) and see if there are any known bugs that might affect your setup.
  • Prepare a high-level design document (HLD) before moving forward. This helps the team to be on the same page.
  • Prepare a test plan (functionality and smoke testing).
  • Execute the test plan in a lab environment and document the results.
  • Plan a maintenance window.
  • After getting approval for the maintenance window, prepare a MoP (Method of Procedure). The MoP should include every step/command that will be executed during the live deployment. It also includes a plan for a possible rollback.
  • Monitor systems for a day or two.
  • Prepare Low-Level Design document (LLD) which is suppose to include every small detail of the deployment. (specs, ports used by the services, diagrams, where to contact in case of an issue, used firmware, supported features, system management tool details, configuration details etc) This is to handover the project to the client so it can be maintained.

 

Now I would like to tell you how both customers and vendors are failing to follow up such a checklist, or even if they do, why things get complicated.

  • Identify the technical requirements/use cases.

Things are generally over-promised. Presales sells a use case together with the relevant hardware with it, and a field engineer is supposed to deploy it. Sometimes they say: “yes” for a non-existing feature and letting field engineer and the project manager find a workaround for it, Or asking Research & Development team to develop that feature not to jeopardize the whole deal quickly. This makes everything very complicated and existing priority list for the feature requests becomes completely invalid. Not to mention that it also demoralizes the team.

  • Check the feasibility of the use cases in a lab environment. Prepare the configuration.

This is the one I love talking about the most; Best effort is not the best practice. The same use case can be deployed in many different versions. Especially if it requires some coding or complex configurations, the customer is depended on the field engineer’s skill-sets. Most of the vendors out there, unfortunately, does not have the best of class internal training materials for their engineers. Some of their field engineers pick up much faster or does the job more optimally while others’ configuration may not be the best. Make no mistake; both can fulfill the goal; however, in the long term, the non-optimal one can cause some issues such as when scaling is on the table or additional hardware is being added. Things can get broken, and generally, they escalate pretty fast.

  • Prepare a test plan (functionality and smoke testing).
  • Execute the test plan in a lab environment and document the results.

Sometimes the timeline is too tight, and the customer is under pressure because, for example, high-level management is expecting a faster deployment, or maybe it is a request from the regulator, and they need to implement the changes as soon as possible. Test phase might be skipped, and generally, the result is devastating enough to spend even more time to fix the configuration and go for the new maintenance windows. Meanwhile, the damage is already done, and it can be hard to get your customer’s trust back.

  • Prepare Low-Level Design document (LLD) which is supposed to include every small detail for the project. This is to handover the project so your client can maintain it.

I think we engineers, in general, lack the skills to document what we do. We generally forget to prepare a low-level design document and hand it over to the customer. Documentation is the most tedious part, and people don’t bother to ask for it at that phase since the project is deployed and everyone is happy. A lack of documentation makes a possible issue very hard to troubleshoot, or even for vendor technical support team it is hard to understand what the root cause might be at first glance.

Each of these failures can delay the completion of a project for a couple of weeks or even months.What is THE Root-Cause?

Many of those failures are because departments within a company are very isolated. It might sound like an easy job, however for different development teams together with Quality Assurance, Documentation and Sales departments, moving forward to a common goal is a real challenge. Nowadays enterprise products are the combination of multiple technologies and the components, and larger the company, the greater the lack of collaboration for these different groups. DevOps approach is a must for business owners and the development, operations, and quality assurance departments to deliver the software in a continuous manner that enables the businesses to adapt to the market requirements faster.

Adapting DevOps

Adapting DevOps is not putting the commonly used tools to the table such as Ansible, Terraform, Kubernetes, Puppet, Gerrit etc.

It starts with a real communication. Moving towards a common goal requires social engineering. How can a development team be the most productive? What code collaboration tool fulfills their expectation. How should a ticket for an issue or a feature request be created? What would be the workflows of these tickets and how does the issue tracker react when relevant code is pushed to master branch? Who evaluates the feature request? What are the existing in-house tools that can possibly be replaced with open source ones? How is a code snippet being tested and deployed on the lab environment? Is the test being initiated automatically? If yes, how it really works and does it put more burden to the development team to test their code or make their life easier which can result in better product quality?

DevOps teams are here to provide you the ability to identify business needs and adopt the widely known DevOps practices such as;

  • Continuous Integration,
  • Continuous Testing
  • Continuous Delivery
  • Continuous Monitoring

CI, CD practices allows developers to work together even if there are a large number of cross-functional teams. Continuous Integration also requires Continuous Testing where each new feature or a bug fix can be tested by using test automation approach and revert back to the team if need be, or integrate the ready-code with a specific branch and fire up another automated test to verify the build.

Continuous Monitoring, however, is my personal favorite, this approach can be used in the test or and the production systems. Figuring out how a product behaves after being deployed in the production environment is another challenge. My experience shows that even at the companies where an IT Operations team is allocated to monitor the systems don’t know before an issue becomes a problem. When things are broken, everyone will be involved to solve this, and possibly a small blame game is around the corner. With Continuous Monitoring approach, you can check the metrics and provide real data to everyone including shareholders. Does your network pipes are being saturated? What is the utilization rate and do you need to invest for a possible expansion at the close future? How many times or how often a certain error occurs and can we visualize this? Can those metrics be used for different goals rather than troubleshooting? (marketing maybe?)

Data provided by monitoring solutions can help businesses to react in an agile manner and change their business plans if need be.

 

 

Pin It on Pinterest