NetDevOps: Introduction and a working reference model

Disclaimer: – This note was written by me (Mayank Nauni) in my personal capacity. The opinions expressed in this article are solely my own and do not reflect the view of my employer or my preference towards any of the OEMs.

 

“NetDevOps”, another buzz word in the town, the real question is, is it yet another unicorn trotting on a stage with confetti flying around it?

Let us ponder on some fundamentals questions from the network engineers.

My network works just fine, why do I need to change my ways of working?

Well, there are a few points I would like to share before we even start discussing what is NetDevOps.

What are our current network infra challenges?

• We have a never-ending tussle between Agility vs Availability, the application developers have adopted the DevOps methodologies long back and hence are much more agile in the application development & operations now but the title of weakest or slowest link in this end to end delivery chain has been passed on to the IT infrastructure. If we push changes to infrastructure quickly without performing a proper evaluation, we end up breaking production. Rings a bell?
• We have been Copy-pasting and hand-crafting configurations at the command line, which: –

•        is extremely error-prone

•       Leads to inconsistencies across the network

•       Validation / Assessment on CLI configs takes a long time

•       Does not scale and not meant to support Agile methodology

• Matching pace due to Agility at other component layers (application) can result in massive outages due to less time for review

In short, with the existing ways of working, network changes will fail because of the growing scale, complexity and rate of change.

 

Because of all these agility related demands, more of the enterprises decided to go towards the cloud only to realize later that cloud is not an answer to all IT infra woes. With partial workload migration to cloud, application owners had experienced the benefits cloud brings to them, agility was noticeable difference besides cost and scalability. As I have mentioned in my earlier articles that not all applications can go over to cloud so now we have two infra components in most of the network, cloud and on-premise infra, there is a visible difference between both, on-premise infra and the associated OEMs had no choice but to step up their game, bring the same level of agility to the on-premise infra which always had better control than a public cloud.

Okay, I know the challenges now, but what are goal of our future Networks?

Most of the changes in the network today are recurring in nature, in most of the enterprises seldom engineers build something which is totally new and had no relevance to the earlier changes in the same network; so, the future network should:

• Use templates to ensure standardization, accuracy and predictability
• Handle bulk changes at faster pace that too error free, reduce the chances of human error by following an “idiot-proof” stencil
• Help us to achieve configurational consistency in repetitive changes
• Enable engineers to bypass CLI
• Be able to run repetitive automated tests

 

Good news, software world had similar challenges and they have already addresses these issues by using DevOps methodologies which is: –

•       Continuous Integration and Delivery (CI/CD)

• Frequent individual integrations into master repository
• Automated build, test, deploy
• Identify errors as quickly as possible
• Many tools already available like Git, Jenkins, Ansible etc.

 

Hold on, are you asking me to start my network engineering journey afresh again, from scratch?

In a way, Yes, but it will be quick and safe, how about following a working model rather than reinventing the wheel?

Just because we have learned to do things a certain way doesn’t mean we can keep doing it the same way to eternity, the usual ways of doing network changes are failing because of the growing scale, complexity and rate of change.

Different teams have been speaking different languages in an enterprise environment, but not to mention they all have a common goal (though road towards the goal may be different), but disparity in the ways of working will always be a hindrance which often creates confusions and misunderstanding between the application and infra world.

Wouldn’t it be fantastic if we could find a common platform for all teams to collaborate, so that they could understand each other better? Well that is what I am going to talk about in rest of this document.

 

 

Can you tell me what is NetDevOps already?

Well gone are the days when we had fixed definitions in the IT industry, now experts coin new terms and rest of the world (another bunch of experts) interpret those terms the way they want, hence it is very difficult for me to define NetDevOps in a standard way, but the best possible standard explanation is: –

“NetDevOps brings the culture, technical methods, strategies, and best practices of DevOps to Networking.”

NetDevOps builds and manages a network that enables network services to be consumed in a DevOps approach, it is adopting “Continuous Development” approach to network changes.

What skills do I need to learn to become a NetDevOps engineer?

NetDevOps is so easy that even an average network engineer like me could set it up easily with little help from Google, I would like to simplify it for my fellow infra friends. We can start reading about these tools: –

·       Git – Source Control Management

·       Jenkins – Workflow Management

·       Ansible – Configuration Management

·       EVE – The Emulated Virtual Environment for Network, Security and DevOps

·       Jinja2 – Configuration Template

1.     Git – source control manager

Git can be simply defined as a source control manager (SCM) or simply known as revision control. It manages changes to the configuration files, it can manage versioning for collections of information as well.

The real question is that why is it so relevant to network engineers. The answer lies in the current practice of dealing with config files, left in some local directory, some remote disk or external disk and that is it. Refer to these fantastic blogs: –

http://ipengineer.net/2015/04/git-for-network-engineers/

http://onenetworkguy.com/2016/05/17/git-for-network-engineers/

Besides these wonderful blogs, I saw a bunch of videos on YouTube to understand how Git works (my favorite one is this by Kevin Wallace https://www.youtube.com/watch?v=chzW0Zrlg44) .

The crux is: –

1.     We have a master repo containing all production configurations

2.     We create branches which are analogues to sub folders and test config changes in the branch

3.     Once network engineer has tested his branch repo, he can merge it with the master repo which may be pushed to network devices later

4.     Team can collaborate efficiently with Git

5.     Git can be on cloud or on-premise, you get free public repo on GitHub and you get private repo by paying a bit every month; practice caution while you are using public repos, the whole world can see it.

 

2.     Jenkins – Workflow Management

So, we have two kinds of workflow in every environment i.e. documented and undocumented. Irrespective of reactive or proactive change, we must follow some process to implement the change and get the desired outputs.

Workflow management software like Jenkins help us to achieve Continuous Integration, which is a pillar of DevOps culture. It is very helpful to automate repetitive tasks (i.e. firewall changes, routing changes etc.), and create a feedback loops, so that future repetitions can be done more efficiently while insuring less errors. With the plethora of awesome plugins on it, Jenkins can interact with other vital components in the DevOps like, Git, Ansible, Spark etc.

The tool is very intuitive and quite easy to install (I personally used a docker for it); I would recommend free videos on YouTube to understand Jenkins better.

https://www.youtube.com/watch?v=Lxd6JMMxuwo

A good read on Jenkins: –

https://www.infoworld.com/article/3239666/devops/what-is-jenkins-the-ci-server-explained.html

 

3.     Ansible – Configuration Management

Ansible is the most popular network configuration automation tool, it is very popular among the network engineering ecosystem since it doesn’t need an agent to operate. It can run playbooks, which are essentially new way of representing network configuration in YML format.

In short, all repetitive work in your day to day tasks can be automated by using Ansible playbooks. Very hard to explain Ansible in words, here are some very interesting videos to learn more about Ansible: –

https://www.youtube.com/watch?v=lUBPAGI9W6U&t=1063s

https://www.youtube.com/watch?v=6oG7SCX-q4A

https://www.youtube.com/watch?v=VYEVjKvMKqU

https://www.youtube.com/watch?v=_3XIp6bw58s&t=471s

 

4.     EVE – The Emulated Virtual Environment for Network, Security and DevOps

EVE can create virtual networks which are very similar or almost equal to production networks. This has huge advantage for those who want to ensure that their solution will work before they go online. EVE practically supports nearly all device’s virtual OS images.

Testing your configuration prior to production deployment is one of the main pillar of DevOps culture.

http://www.eve-ng.net

 

 

5.     Jinja2 – Configuration Template

Using Python and Jinja2 to automate network configuration templates is a useful way to simplify repetitive network tasks. In using this alternative method to automate recurring network changes we can remove the common error mistakes experienced in the copying/pasting of commands into the CLI (command line interface). You can find more details in the links below: –

https://blogs.cisco.com/developer/network-configuration-template

 

Okay, thanks for the overwhelming theory, but does it work?

 

Well, yes it does. I am using a simplified NetDevOps deployment in my lab and would be more than happy to answer any queries regarding the same.

•       All config playbooks (.yml) are on a repo at GitHub

•       Network engineer does modification in Dev branch

•       Dev branch modified code can be tested on EVE / VIRL / GNS3

•       Merge into master branch post validation

•       Merge triggers the web-hook to Jenkins, triggers pipeline

•       Jenkins downloads the repo from Github

•       Ansible plugin on Jenkins fires the .yml file to deploy the playbook across the environment

•       Jenkins Spark plugin sends success / failure notification to Spark chat room.

 

That was cool, so are you telling me I am going to have NetDevOps in my organization without any major challenges?

Just like any other new methodology NetDevOps has its own bunch of challenges: –

·       Cultural change may not be easy

·       NetDevOps requires network engineers to think differently, new set of skills are required i.e. Familiarity with source control, Ansible, Jenkins, Git etc.

·       The value of NetDevOps is only visible when it is 100% templated, so we have to be patient enough to see the value NetDevOps brings to an organization.

·       Network Platform APIs not standard yet, so it is quite challenging in a multi-vendor environments. Most of the products in the network may not have all functionalities exposed via APIs which can be quite challenging from templating’s perspective.

 

Are there any NetDevOps learning portals I can refer to, to begin my NetDevOps journey?

Yes, plenty of them.

Cisco

https://devnet.cisco.com/

Ciscolive Presentation

BRKDCN-2602

Plenty of hands-on labs and videos by NetDevOps Genius, Hank Preston.

F5

https://super-netops.askdelphi.com/

GitHub

Ready to use codes from Hank Preston

https://github.com/hpreston

 

I hope it was helpful.

Mayank Nauni ( [email protected])