It’s not hard to debate that sausage has conquered the culinary world. The global popularity of sausage is undeniable. Chorizo, linguica, luganega, wurst… cocktail, weiner, franks, brats… Lamb, pork, chicken, goat *gasp*… Broiled, fried, grilled, smoked, cold, hot… You get the picture? Lots of sausage, lots of varieties, and lots of methods.

Sausage has a primary defining characteristic. The meat must be chopped up. That’s it! This is where I draw an initial parallel to a network. A network moves data. After that, anything goes with sausages, or networks. Maybe you have a network of sausages? Maybe you’re a network engineer at a sausage company?

Sausages are also very regional, sort of like networking vendors. Your home tribe may consume just one type of sausage. Maybe your tribe likes a wide variety of sausages. There is nothing wrong with either food choice, that’s just your taste.

The networking world can easily draw many parallels with sausage making and I could go on for hours about how the “sausage gets made”. Our presenters at Networking Field Day 21 talked a lot about one specific aspect of sausage making — network sausage automation.

So, if you like sausage, do you prefer sausage from a package or fresh sausage from a butcher? Network automation parallels this quite well. I like sausage from a butcher, but if I’m cooking for 500, I may use sausage from a package. It doesn’t mean I’m going to eat sausage from a package 100% of the time. I want both, but for different reasons. So give me network automation for the time I want to do something 500 times, but I don’t want automation for the design. I’m looking at you, [insert vendor here] that’s going to “green field” my data center with your “automation”. No thanks click-button-get-fabric.

Is network automation going to take your job? That depends. Are you a sausage stuffer? If you’re going to stand there and stuff someone else’s sausage all day long, then you ARE replaceable. According to Replaced by Robots – Sausage Stuffer, there is a 98% chance that “Sausage Stuffer” will be replaced by robots. Those automation machines exist and those packaged sausages are likely never touched by human hands. Life pro tip – don’t be a sausage stuffer. Create a recipe, be the chef, get your hands dirty, touch the sausage.

Also, according to Replaced by Robots – Network Engineer, there is a 3% chance of Network Engineers being replaced by a robots. I believe that statistic is very accurate. There are to many different kinds of sausage to believe a machine is going to automate your recipe. Unless you build the sausage automation machine. However, that isn’t an easy thing to do. Networking vendors spend millions on scalable platforms to perform all the necessary layers of automation. I hate to say it, but your *nix box with a few playbooks is not a ‘platform of automation’. It will help you package some sausage, but you made that sausage, created the recipes, and hold them dear, and call them George.

Some Networking Field Day 21 presenters showcased their automation platforms specific to their own solution. Some presenters showcased their multi-vendor approach to a platform. Either way the point was very clear, automation platforms are incredibly hard to build, maintain, resource, and pay for. Entire companies are being built around platform creation and very few, very very few companies are internally well equipped to build a platform. Some companies think they can make their own sausage, package it, sell it, and eat it, but that is a long way from reality. If you’re not already talking the language of cloud native application architecture, maybe stop now and budget a purchase.

I once received some advice from a rocket scientist, “Configure, don’t customize, a boxed product. The moment you customize then you become the owner, the developer.” Wouldn’t that also translate to trying to automate all the ‘boxes’ on your network? Engineers in the field are barely able to keep up with the rapid pace of change from one vendor and you’re going to create a platform that keeps up with the API changes of one vendor, two vendors, three vendors??? Seriously, don’t choke on your own network sausage automation.

I’ve always enjoyed seeing different visions of the sausage making future. Networking Field Day 21 showcased several of those visions and I have some more specific posts on the way. Will network sausage automation become a reality? Well, again, that depends on your tribe, your tastes, and your budget.

Alright, I need to wrap this up. 😉 I hope you’ve enjoyed networking sausage automation. Is wasabi sausage a thing? #sooory

—– A Networking Field Day 21 Reflection —- #NFD21 @TechFieldDay

A little bit of technology buzz is being generated around Firefox and the “default” implementation of DNS-over-HTTPS (DoH). Mozilla has decided that at the end of this month (Sept. 2019) they will start rolling out DoH as default in the browser. I’m assuming you’re familiar with DoH.

Let’s address the most obvious problems for enterprises. DNS represents a wealth of information gathering within the four walls of a business. Content control, security protections, and split DNS are just a few things to mention. Breaking host level DNS resolution of browser is a threat against these protections. The security team should already be accounting for this in their SWOT as malware has already adopted this for C&C.

Mozilla has a plan for this and has set up ways Firefox will fall back to host DNS resolution, per network session (coffee shop, home, vpn), with a few specific checks.

When does it fall back? (I don’t know all the methods and I need to get back on nightly)

• A canary domain listed in content control systems. If the lookup is triggered Firefox realizes the host is within a content control DNS system. OpenDNS/Umbrella immediately comes to mind with their operation of exampleadultsite.com. Not actually an adult site.
  • The problem for users- blacklists are a mess, standards, and are they really going to get worldwide participation to have this canary domain created? You could write a whole blog about the gaps in these systems.
  • By the way – DoH will not be enabled in the UK because of this exact issue.
• Safe search redirection is triggered, in the example www.google.com resolves to forcesafesearch.google.com
• Non standard TLD detection and RFC1918 responses. This isn’t a bad attempt at triggering the fall back, but many enterprises focus on split dns which is a valid TLD. IPv6 also throws a wrench in this logic.
• Windows and macOS, detect parental controls enabled in the operating system

So what can you do inside your enterprise?

Blocking Cloudflare is not a good solution even though that may be the initial reaction. Cloudflare obviously does a lot more than DNS and keeping up with blocking a CDN is just a painful management point. Don’t do this.

Today, Firefox may not be included in your desktop system management and this will likely become a future requirement. Users are able to manually enable DoH and specify a custom provider that isn’t Cloudflare.

You can signal non-managed Firefox installations with the canary domain, which would work while machines are on-network. This canary domain “use-application-dns.net” needs to return NXDOMAIN instead of the A or AAAA IP address. Again, to reiterate this will not block a user manually enabling DoH.

Running a proxy and/or edge decryption clearly presents you with additional options to block this configuration.

So what’s the most logical step? Welcome to Firefox for Enterprise!
… Sorry, more management, but that isn’t all bad. Consider adding Firefox as an allowed package alongside an enterprise policy.

If you’re doing psexec checks to block Firefox or SWIM then you’re controlling things differently. If you want to allow Firefox then you need to give it a policy. Firefox will check for enterprise policy to disable DoH.

• Is the Firefox security.enterprise_roots.enabled preference set to true?
• Is any enterprise policy configured?

Also, if you’re allowing Firefox in the enterprise I hope you are already controlling security.enterprise_roots.enabled. Firefox runs their own root program and you should be controlling those circles-of-trust. Did we not clearly explain the circle of trust to you, Greg?

What would be my recommendation? Deploy Firefox with Windows group policy ADMX templates, macOS/Linux file distribution.

Firefox ADMX templates can be found at https://github.com/mozilla/policy-templates/releases

DNSOverHTTPS – Locked

Configure DNS over HTTPS.

Enabled determines whether DNS over HTTPS is enabled

ProviderURL is a URL to another provider.

Locked prevents the user from changing DNS over HTTPS preferences.

https://github.com/mozilla/policy-templates/blob/master/README.md#dnsoverhttps

I hope I covered this information accurately. If I got anything blatantly wrong feel free to call me out and I’ll edit.

Thanks!

________________________________________________

The rabbit hole.
https://github.com/mozilla/policy-templates

Source link: https://support.mozilla.org/en-US/kb/configuring-networks-disable-dns-over-https

Source #1: “If a user has chosen to manually enable DoH, the signal from the network will be ignored and the user’s preference will be honored.”

Source #2: “When any of these checks indicates a potential issue, Firefox will disable DoH for the remainder of the network session, unless the user has enabled the “DoH always” preference as mentioned above.”

Source: Pi-hole is already on it — Within the past 24 hours of this post this canary domain was merged into development https://github.com/pi-hole/pi-hole/pull/2915

If you’re looking to fall deep down the rabbit hole of DNS-over-HTTPS issues then I have the link for you. The IETF has published an extensive look at DoH risks and issues at https://datatracker.ietf.org/doc/draft-livingood-doh-implementation-risks-issues/

I rather dislike information silos and everything they stand for. I like collaboration, knowledge sharing, and that is likely evident by the fact you’re reading my blog post right now. The “information silo” cuts against my personality and if I keep writing in this paragraph I’ll tell you how I really feel.

So why do they exist? Scale and functional teams will create these silos. Which is necessary to churn productive work in a large business. Team X does X, Team Y does Y, and Team Z does Z. If every day at work, XYZ had to get together and discuss their responsibilities then they’d never get anything done. We all need to produce! The traditional management hierarchy makes this even worse with vertical up-down pay, titles, and promotion systems. So if you join X, Y, or Z then you may exist in that silo for a very long time.

Why these silos exist and grow hair becomes an accounting problem. To overcome the information sharing gaps you need to hire an X-Y translator, an X-Z translator, etc… and all those translation hires become expensive. For every layer and silo you create, without the translator you may get stuff done and grow, but agility shrinks.

So this is something that I picked up from way back when. It’s the concept of a “floater” which is used on manufacturing floors, assembly lines, or production lines. I once acted as a floater on a credit card statement production floor. Back before e-statements, the credit card processor was likely responsible for statement production. There was the printing silo, the warehouse silo, the production silo, and the mailing silo just to name a few. It was the “floater” that kept these silos fed. If the envelop stuffing machines needed more envelops or more advertisements then you’d go fetch it and share that information quite physically. The key point I’m trying to make is that each silo performed a single production focused task and that kept itself moving. The printers printed statements, the warehouse organized everything needed, and the mail room silo kept everything moving out to the postal service. It was indeed quite an impressive operation.

What would happen if we expected that production system to operate without floaters or feeders? The warehouse system couldn’t keep things organized if their job was also to accept and run materials to the floor. The stuffing machine operators couldn’t expect to keep their machine running if they had to get their own materials, discuss what they needed, or take the envelops to the mail room. The mail room couldn’t run at full capacity if they had to go get their own stuff or move their own mail.

The key was the “floater” who could move and operate within each of these silos. They understood key elements of each of these production silos and helped keep the larger machine running. The floater didn’t need to know exactly how to run the stuffing machine, or the printing machine, or the mailing machine, but knew _enough_ to keep it running.

The floater concept is lost among traditional IT silos. We expect everyone in their silo to know what the other silo is doing. The best we can hope for is people in one silo toss and spit out enough information so that things don’t grind to a halt. In full speed production, the teams barely have enough time to tackle their own work, much less help the other silos understand what’s going on.

Things get bad when things break down. If one silo starts to slow or fails to operate it can have a severe impact on the rest of the business. The floater can help pace things by relaying information and key pieces of details to keep things on track, slow things down, or speed things up. In the credit card processing line this was a very in-the-face approach. Quite a bit of yelling was involved.

In a small to medium sized business this is usually handled relatively easily within IT. The IT team may be just a few people and the day-to-day of working and meeting together is enough to keep information sharing afloat. When we start getting into small enterprise is where the problems begin to surface. We expect the producers in the security team, network team, and compute team to both produce and float. This results in inefficiencies and the production line starts to get messy.

Hire IT Production Floaters, seriously. It’s a real job title in manufacturing and assembly. I think floaters can help elevate production and at the same time destroy the information technology silos. As the benefits of cross-team collaboration erode the information sharing barriers each silo becomes more efficient. Functionally move someone between security, network, compute, edge, mainframe, and the PMO on a regular basis and I’ll bet it has a positive impact. Let those individuals report back as an outsider looking in. Do they have something negative to say about how the network team deals with the security team? Having been on both sides they should be listened to. Help them pace production and make changes where needed.

For fun — What would that job description look like?

Essential Functions

• You will be responsible for performing varied tasks including network, security and testing of applications.
• Work will be performed in different functional areas depending on the highest need at the time.
• Must be comfortable and willing to have varied work that may change each day or potentially several times throughout the day
• Demonstrates capability in configuring network devices and securing computing devices
• Capable of reading and comprehending network architecture diagrams and routing protocol diagrams
• Responsible for checking deployed devices to meet XYZ security compliance standards

Minimum Qualifications

• Not be an idiot, willing to learn, change skills, seek out new life, go where no one has gone before
• Able to read an OEM manual, use an IP subnet calculator, text and walk
• Be at least 18 years of age
• Ability to work independently

Equipment/Machinery Used

• A web browser
• A terminal
• Linux, Windows, Mac networking tools

Personal Attributes

• Ability to speak intelligently, to others smarter than you, in their own area of expertise.
• Very strong written and oral communication skills
• Keen attention to detail and not afraid to call anyone out
• Ability to effectively prioritize and execute tasks in a high-pressure environment
• Ability to build a team-oriented and collaborative environment
• Mentally and physically flexible just in case you need to rack something by yourself in RU 42

Hire a floater!

In case you’re still here — let me know what you think!