July 21, 2017

Build It and They Will Come

There’s so much talk about DevSecOps I’m hesitating to write this and add to the noise. Most of what I read is about Continuous Integration (CI) and Continuous Delivery (CD) and that’s great. But many of us are dealing with a lot of legacy code and configurations, along with 3rd party vendors and products that don’t easily fit that mode. And yeah, we still have a lot of that silo mentality around that makes it hard to “un-separate the duties,” so to speak.

The best acronym to represent DevSecOps is CALMS. It’s been said many times that DevSecOps is first and foremost a culture. And that is as true as it gets. Without a commitment to that culture change, there is little chance of success. So how does true culture change happen? One way is by showing little points of success and sharing the joy. Once we satisfy the WIIFM question, most of the cultural barriers can drop.

Agile and DevSecOps rely on breaking work into the small, deliverable pieces, the MVP. So let’s make sure we’re practicing what we preach. Small chunks that deliver value.

So what could be a little point of success for DevSecOps ? Well, the “A” in CALMS stands for automation, and that’s the place to look first. While automation ultimately needs to be across the entire systems life-cycle (including system provisioning, configuration, build, testing, monitoring, etc.), where I’ve seen the most bang for the buck is in deployment automation. Getting that new code change from development, through the test regions and into production without things breaking. How do we make that happen consistently and error-free? Practice, practice, practice.

So, let’s focus on automating the deployment from the Development source repository to Integration region. Even if the automated tests aren't quite there yet. Even if the build breaks often. Even if the build is done (gasp) manually. Did the actual deployment work? Great. Now run that same deployment automation into UAT. Did it work? No? Ah, forgot to change that configuration to use the corresponding region’s service account! Now, run that same automation. Works! Now, run that same thing, this time with the target region STG. Getting there!

Now that might sound easy, but maybe your deployment is a complicated series of manual steps that’s a little more difficult to completely automate. OK, let’s look to the “L” in CALMS apply the Lean MVP principle and break DevSecOps down further. Maybe the deployment can still be manual for now (just hold your nose), but perhaps now we focus on deployment validation as our first success. Write some code that executes non-destructive tests against the database, for example. Perhaps it just validates that the service account works. Try to call an API/service. Validate that the file directories are writable. Look for a specific success or failure log entry. These are small pieces of code that can catch deployment errors very quickly without having to release it to users who will find those issues on their own.

Back to that culture challenge, now we have something we can show using the “M” for measurement. Look, our validation process passed! Time to “S” for share that success, perhaps in a dashboard that shows the build and deployment statuses. Even if you have to manually update that dashboard.

So we can get small wins along the way.

But, you say, “what about CD? This isn’t DevOps!” Perhaps you have to deal with Release Management because they only want monthly releases? That’s a different fight. Your focus is to make delivery to PROD a non-event. And since you’ve automated deployment across the non-production regions you know it’s just gonna work. 

Build that Easy Button. They will come.

August 8, 2016

What if I turn it up to eleven?

This is Spinal Tap is a classic in a lot of ways, but one thing the fictional band knew was how to turn up the volume.

Have you cranked up your load testing? In my last large project, there were several critical components that had to be load tested individually and as part of the overall system. We all know that it just takes one bottleneck to spoil an otherwise perfectly good transaction. In our case, there were so many components and handoffs inside of a single synchronous transaction that one bottleneck could easily kill the entire system. But where were they?To ensure we tackled each one, we first had to model all the components of the user transaction from top to bottom. Each sub-transaction needed to be instrumented and measured to help identify those bottlenecks. And there were plenty. The transaction volume (i.e. "load" per time interval for this discussion) is estimated at a "busy hour" based on estimates or actual production data. Then all the different scenarios need to be played out, using "what if" brainstorming. What if a busy hour hits and a heavy batch process is still running? What if one or more of the servers in the cluster fail? What if a database query is constraining the SAN? What if everyone tries to log in before the data reorg is complete? You know the drill, the scenario brainstorm is the fun part. Remember, it's not necessarily the obvious set of transactions that are the killers. The mixing and matching of transactions is critical too.Then it comes down to building the scripts that let you pull the levers to match the scenarios. Don't skimp here. Your test environment should match production as closely as possible. Otherwise, while you'll certainly discover bottlenecks, they won't be the same ones in production. I've had the experience that the negative impact on a transaction can be severe when a production system performs faster than the test system. Because the bottleneck has moved. And a transaction component ends up waiting in line for something completely different -- an outcome that didn't exist in the test environment.After the tests are documented, the analysis starts. In those scenarios, does the system perform within the SLA? Sometimes you have to answer the question, does it need to? Does your SLA dictate a busy hour response? Or overall average? Smart negotiators will agree on SLAs that include volume estimates as well as performance metrics. No one can guarantee response time for an unknown quantity.So, while functional testing is important, you also need to know how your system will perform under load scenarios as well. A properly constructed load test framework will pay dividends to indicate impact with any change to the system, especially upgrades. Crank it up to eleven!Originally published on LinkedIn

October 28, 2015

Where's The Problem?

We're putting the finishing touches on a 4-year project this week. This was one of those once-or-twice in a lifetime things that can either suck the life out of you or energize you tremendously. If done well, it will do both, and this one did. And how cool is this: It works!

It really did enable transformation of our company by implementing a new core system that allows tremendous flexibility in how we define and configure our processes. In doing so, we had to essentially re-engineer everything we've done with technology over the past 20 years. Fun? Yes! Excruciating? For sure!

If that was all we did, we'd probably count it a success. But we gained a lot more. By running an agile team approach, we put business operations leads with architects and IT staff together to live, eat, and breathe each others lives. Eye-opening to say the least, and definitely earned most of us a new level of respect for the others' jobs. 

It also made us challenge each other constantly. "The way we've always done it" was no longer acceptable. We had to look at problems in new ways. Having the diversity of talent and background on the teams made a huge difference as we leveraged each others experience and viewpoints. 

Agile scrum is an approach that is pretty simple. It brings together a team every day to review what we accomplished the previous day, what we want to get done today, and what barriers we face. The team's job is to crush the barriers and keep progress moving. Sounds simple, right? Barriers are the most common cause of project slippage (John's opinion), yet they are sometimes the most elusive to uncover. Why is that?

Sometimes we just don't want to admit that we're stuck. We may not want to blame someone else for the barrier. This is very common in some cultures and may need some prodding to draw out of someone who isn't comfortable speaking out. 

But barriers must be broken down to have progress. Identification (admit you have a problem?) is always the first step. Sometimes the barrier isn't what it seems and may need some root cause analysis to get by. Often it will need help from upper management to break through some sacred cows or get past the one who just 'doesn't get it.'

Challenge your team: Call out the barrier you're facing, no matter how small. Identify it. Search for it. Name it. Then knock it down. Success follows.

February 18, 2012

Face the Network

It’s been a really long time since I’ve written here. Busy, I guess. Isn’t that how we all are? Free time is hard to find, and what we have, we give to Facebook.

I won’t repeat all the research statistics here, but wow, this is getting crazy. Facebook’s IPO, scheduled for spring, should raise ten BILLION dollars.

I’m glad all this “wasted” time on Facebook is paying off for someone! So what does this mean for us as a society? I personally love it. I think all the hand wringing is ridiculous. We’ve evolved. Evolution is not always pleasing to everyone at every step. Often it is two steps forward, one step back, but you’re still further along than you were when you started. I’ve been able to connect with old friends that I never had a chance to keep in touch with in other ways. Had it not been for social networking sites such as Facebook and LinkedIn, we wouldn’t have had that opportunity. Is it the same as personal networking face-to-face? Nope, but it can certainly lead to more of that.

Yes, I still do yell at my kids for spending time on the computer when they should be looking for a job or playing outside with their friends (do kids still play outside?) But still, I can’t deny its appeal. Yes, I still get annoyed with people who share too much on their statuses, but sometimes it’s good for a laugh.

So what would I do with that extra time if I didn’t have social network sites to take my time? My Christmas lights would probably be put away; perhaps the carpet would have been vacuumed a few days ago. But I might have missed reconnecting with an old friend that I hadn’t seen in 12 years.  I would have been harder to learn about the job opening when my colleague changed positions.

I like being connected. No different than the other half billion people who seem to as well.

July 20, 2010

John as The Six Million Dollar Man

Steve Austin, astronaut, had it all. Then, a horrific accident. His injuries were grave. But “we can rebuild him; we have the technology.” Thus, the six million dollar man was salvaged, as was Lee Major’s career.

In the past eighteen months, I’ve had a torn rotator cuff repaired, a quadruple heart bypass, and just nine weeks ago, a hip resurfacing. If I were living in the nineteenth century, uh, well, I most likely wouldn’t be. Medical technology continues to surpass expectations, and it does it quietly. While most of us are excited about our new iPad computers and Droid touch screen smartphones, many of us are beneficiaries of technology that dwarfs home computing advances.

My new hip is a case in point. Made of titanium, it’s stronger than anything organic. The resurfacing surgery lasted about an hour and a half. The hospital stay was four days. Day one I was up and walking on the new parts. Painful at first, for sure. But a mere six weeks later, I was walking better than the day before the surgery. And that’s saying a lot. Without going into the gory details of the surgery, you might imagine what it takes to replace the “ball and socket.” It’s not like unscrewing the old light bulb and replacing it with a florescent one. No, there are a lot of parts to move around to get to the spot. Then there’s the sawing, planeing, cutting, stitching, and gluing! Good thing I was out cold.

Before the surgery, it was all I could do to get out of the car and walk to the hospital lobby. A couple hundred years ago I would be stuck in the cave, not able to hunt, not able to run away from the attacking hordes. In short, I wouldn’t have survived very long.

(By the way, the long wait and poor quality of this surgery in socialized medicine countries Canada and Great Britain are huge problems. So maybe I’d still be in the cave. But more about that another time.)

Medical advances are all around us. My heart bypass back in November of 2009 is another amazing example. It started with a blip on my electrocardiogram (EKG) at my primary doctor’s office (thanks for noticing, Dr. Bill!) I then promptly failed the stress test the next week when he referred me to a cardiologist. That same week, into the hospital for a quick angiogram. Lying on the table, I watched the monitors as my doctor inserted the probe in through my wrist, up through the arteries, and into the heart. It was a strange but incredible experience. Of course, I was hoping for a “simple” stent, which would open any clogged artery during the same procedure. No such luck. So, a quick consult with the chief of thoracic surgery, and into the operating room within two weeks as he opens my chest and reassembles things on my heart while a machine keeps me alive.

Wow. We are blessed with the most impressive medical system in the world, and most of us don’t even know it. Kinda puts technology in perspective for those of us who work in business data processing.

I might not be worth six million dollars, but for the huge investment of dollars and human capital in medical technology, for which many of us are grateful recipients, I say thank you America.

May 22, 2010

Is Customer Service Dead?

I seem to be asking myself this question more often lately. My latest experiences were with La-Z-Boy furniture in Amherst, and Dunkin Donuts in Williamsville, NY. These were two very similar experiences, but very different outcomes.

My Dad owned a La-Z-Boy recliner for 30+ years. It was a comfortable chair, but nobody better be sitting in it when he came into the room!

I wanted to purchase a new recliner to use during my recovery from hip surgery. I was hoping they’d have something in stock, but could certainly understand if it needed to be ordered. The problem is, as I was told, it takes 6-8 weeks to order one. That seemed like an excessive amount of time to order a “stock” recliner. After all, I wasn’t ordering a Mercedes-Benz with custom measurements for my hips and butt! (although that might be a great line of business for someone who actually knew about customer service).

I asked the somewhat-disinterested store salesman about the length of time required. Surely, I thought, another store would have this model in stock. Was there any way to transfer stock to this store? I’d even consider going to another store in the area if they could check for me.
No can do. Six to eight weeks. End of discussion. No “sorry, I wish I could help you,” or “let me see what I can do for you.” Just “nope.” Come on. I can seriously order a custom-option car in less time than that!

Needless to say, I told him that unless he could find a way to speed up the delivery, it would end his sale prospects with me. No response. Oh well, I guess I can live with my old recliner for a while.
I wanted to relate this experience to the corporation. I know if one of my staff provided this kind of customer experience, I’d like to know about it. Here’s my response:
Dear Mr. OKeefe:
Thank you for your inquiry and interest in our fine products.
La-Z-Boy Furniture Galleries are independently owned and operated. Unfortunately, we do not have access to what they have available in stock or on display. Our normal production time is 6 - 8 weeks for a custom order.
We suggest you contact store management at the Amherst location to discuss the lack of service you received.
 Regards,La-Z-Boy Incorporated
Let’s look at this one.
  • From “Incorporated.” Really?
  • “Fine products?” Maybe. A bit outdated I have to say. As much as I liked my Dad’s recliner, the new ones were almost the exact same mechanicals (clunky and loud). But that’s ok. They should be proud of their products.
  • The company line sounds familiar. They really don’t have access to their own stores? Do they know the phone numbers? Wow. And I wasn’t ordering anything “custom.” I didn’t even know that was an option. I don’t think it really is.
  • They “suggest I contact store management.” Isn’t that what I did? Maybe you could help that process along? No, push it back on the customer.
I wonder if this company has a Customer Experience Officer (CXO)? Doubtful, huh?
Here’s a contrast.  I am a Dunkin Donut coffee fan. We have to drive a bit out of the way to get coffee at one of their stores. I love cream and sugar in my coffee, but Beth takes it with cream only. She was on her own, and stopped to get a cup on the way to work. “Medium coffee with cream, please.” Again, another long story, but the customer service was less than friendly, and worst of all, they put sugar in her coffee. To her, that’s undrinkable. Unfortunately, she was well on the road before she tasted it. So she had to toss it. No coffee that morning.

She wrote an e-mail to Dunkin Donuts similar to mine. Dunkin Donuts is also a franchised organization, by the way. Here’s her “corporate” response:

Dear Beth, 
We would like to thank you for taking the time to contact us about your experience at the Dunkin' Donuts shop located at xxxxxx.We work hard to maintain the highest standards in guest satisfaction however, it appears we have let you down and for that we apologize. We have forwarded your comments to the owner of this location as well as our Dunkin' Donuts field executive to make them aware of your experience and request that the owner of this location contact you.We hope that you visit us again soon and give us the opportunity to serve you.Thank you and have a great day.
StephanieCustomer Relations Associate

  • From Stephanie. A real person?
  • Thanks for contacting us about the “experience.”
  • “We let you down.” Wow, they understand.
  • “We apologize.” Unique. “La-Z-Boy Incorporated” didn’t apologize to me.
  • “We have forwarded to the owner.” OK, saved me the step. There was no “not our problem, it’s yours” type of response as in the La-Z-Boy version.
Within a day, she had a phone call and apology from the owner, and ten dollars in free coupons to Dunkin Donuts.

La-Z-Boy, see the difference? We’re still Dunkin Donuts customers. We will never buy a La-Z-Boy again. And we’ll tell all our friends about both experiences.

Customer service isn’t dead, but it’s on life support.

February 11, 2010

Out From Under The Covers

I enjoyed the show Undercover Boss that aired after the Super Bowl. In the pilot episode, Larry O'Donnell, President and C.O.O. of Waste Management, works alongside his employees, cleaning porta-potties, sorting waste, collecting garbage from a landfill, and even being fired for the first time in his life.

As his eyes were opened on several levels, he began to understand the impact that his board room decisions were having on the rank and file. Cutbacks meant that staff were working two or three different job duties. He learned from one of the women who worked on the garbage truck that she has to urinate in a bottle along the route because there were no other facilities.

By the end of the show, Larry was a new boss. He understood. He was changed.

Of course we won't know the depth of what changes might happen at Waste Management as a result of this experience. My fear, however, is that the focus on a specific employee or employees who happened to encounter Larry during the filming of the episode won't be pervasive across the company if only attacked at this narrow view.

The reality is that empowerment (yes, that overused and under implemented word) is the only way to ensure that good (and maybe even some bad) ideas get implemented. The President got a fractional taste of the kinds of things that "corporate decisions" impact at the staff level. Sure, the one woman got the promotion, but only because she was in the right place at the right time. Who is making sure that each of the 100s of other employees, also going "above and beyond," are getting noticed? And that someone is actually listening to the women (and men) who don't have a place to urinate on the job?

Management, especially middle management, must take responsibility for soliciting, organizing, facilitating, and implementing staff ideas and recommendations. And if those recommendations need a champion, that manager must be brave enough to stand up for them.

It's a good show to watch. I'd love to see a follow-up on each of these companies 6 months later.

January 13, 2010

The Need for the Governator

The word governance in business is one of those terms that is so ubiquitous it has become almost meaningless. It can be applied in many ways, starting with simple process documentation to an iron-clad, locked-down model of authority for all things process-related.

To me, it's common sense that in order to automate something, we all better understand the current business process. That understanding is sometimes a responsibility that gets delegated to a business analyst or a process modeler. Too often, however, that critical step is brushed or skipped over completely. Then the proverbial Chris Alexander cartoon below comes fact.

"It's great, but this isn't what we wanted" the business person says.

"But it's what you asked for," the IT person says in exasperation.

Maybe it was, maybe it wasn't. The technically elegant solution may give little business value. So what went wrong?

For a while now, my passion has been on how those of us in IT can step back from the sandbox of cool technology toys and figure out what will really work to solve business challenges.

We sometimes expect the "business folks" to be separate and distinct from those crazy "IT folks." IT is part of the business. It shouldn't be off in the corner. IT needs to be intimately involved with both the strategic and tactical operations of the business. Some companies have gone as far as to define a best practice where programmers physically work inside the department for which they typically write software. For example, you might have programmers who locate in the Finance department, or in Customer Service. The programmer still reports to the IT CIO/CTO's organizational structure so they get the technical support they need, but they are involved in the business departmental team meetings and other activities. They even participate in departmental parties. They sit with the customer support reps on the phones, and may even take customer service calls.

For many IT departments, that immersion just isn't possible because of the wide range of required skills and supported departments. But the principle still holds true. Only if you walk in someone’s shoes can you appreciate their pain.

But when it comes down to building the systems, we hit “The Rub:” We often find that the business processes (rules) we want to automate aren’t really “rules,” but “suggestions.” We find two departments define things slightly differently. For example, “sales commission” may be defined by Finance as “x percent of sale plus payroll tax.” Sales may calculate it exclusive of payroll tax. The governance team needs to define which is the true business rule. They are called “business rules” for a reason. They should not be left for IT to define.

Documenting those business rule definitions are the job of the Governance Team. There can be many teams; IT Governance, Information Governance, SOA Governance, etc. The bottom line is that governance is not just about adherence to rules and process, but about alignment to the business.

This is also where the Use Case documents are so useful as they capture each scenario in detail for those processes. Only when there is agreement on those use cases and business rules should the software design start.

Business Governance needs to ensure that this alignment is taking place consistently. That communication gap that exists between the “business folks” and “IT folks” has to be addressed. Some of it is just agreeing on a standard lexicon. Yes, many of us in IT are geeks - and that gap can be a stretch for some.

The responsibility lies with the business executives to define a process that makes that alignment possible. Business managers should sit on IT committees and teams that work on technical solutions. Unless the “Business” is driving the change, we’ll continue to have to deal with the glorified tire swing that IT thought was asked for.

July 17, 2009

The Case for Use Cases

It strikes me how often technical folks like me expect our customers (aka “users”) to know stuff that took us years to figure out. A big part of my job is designing and building computer software systems that enable the business to automate routine and/or complex tasks. Those systems help increase efficiency and thereby help provide (in theory anyway) profit to the bottom line.

I’ve worked with several leading technology vendors and methodologies to elicit the real requirements from the customers in order to build these systems. The purpose of documenting requirements is that we need to have a common understanding of “what” the system is supposed to accomplish. If we’re good, we try to leave out the “how” the system will accomplish that task until we get the requirements (the “what”) fully understood and agreed upon.

We break out these requirements into “functional,” which is a feature such as “allow the report to be sorted by any column,” and “non-functional” requirements. Yeah, we really call it that. Non-functional in this context has to do with things that aren’t really “features,” such as security, usability (remember how complicated it was to set the old VCR clock?), reliability, performance, etc.

Too often, I’ve seen the technologists go off and build something majestic – a real work of art – that becomes totally useless to our customer because it is too complex. The term is “over-engineered,” and that usually means that we’ve managed to make a really cool thing unusable.

So, this “requirements definition” portion of a software project is critical. We need to understand exactly what the user expects the software to accomplish. Only then should the technology folks go to the design table. Part of our job then, is to apply a logical approach to this process of definition of requirements.

A Use Case is an English representation of the interaction between the user of the system and the computer system itself. It is usually in the form of a scenario, whereby we methodically go through all kinds of iterations on how that interaction might go, in order to produce some result. It’s a kind of mini-story that can be told over and over again.

Use Cases were initially introduced to technologists in 1992 by Ivar Jacobson in Sweeden. As part of the “object oriented” approach to programming, it is a model that has gained momentum to the point where virtually all new systems are now build on the object model. Unfortunately, the adoption of the Use Case hasn’t been as quick, even though it is the foundation of all development, according to Jacobson. So, the sparkling new “object oriented” software still doesn’t meet users’ expectations.

The framework of the Use Case is one that lets us find all those hidden “gotchas” early in the process instead of after the software is delivered. It’s a way to capture the logic of the program on paper, before a lot of time is spent on coding. If we can run different scenarios through the Use Case, we know we’re in good shape to begin the next development step.

Our challenge as technical analysts, architects and developers is to help the customer community understand the value of a Use Case, and work with the users to define and build them early in the project. It should be in the top of the toolbox for every person involved in creating new computerized systems.


March 18, 2009

I Read You Loud and Muddled

“What we’ve got here is a failure to communicate.” That phrase was spoken by the prison captain (warden) to prisoner Luke in the 1967 film Cool Hand Luke. The captain had just knocked him down, both figuratively and literally. The captain was showing his superiority over Luke’s lowly position as a prisoner, and serving as a warning to others on the chain gang that he was indeed the boss, and his instructions were to be followed without question.

Unfortunately, around businesses in America today, that attitude can be found in many bosses. And, just like that warden, communication failure is typically the reason for the failure of leadership. Fredrick Taylor’s principle in 1911 that “the workman who is best suited to actually doing the work is incapable of fully understanding the science,” is no longer the best management model. It lacks the essential communication and feedback processes. The American Heritage Dictionary defines communication as “the exchange of thoughts, messages, or information, as by speech, signals, writing, or behavior.” Key to that definition is the word exchange, which implies a 2-way path.

But it's not just a matter of crafting a clear message. That same message will be heard by different people in different ways. When I went from leading a team of technical professionals to leading a team of call center employees, I learned an important lesson of “situational leadership”. Many years later, I took a course with that subject by Ken Blanchard. Blanchard describes the relationship of the development level of the staff to the appropriate matching style of the leader.

The idea hit home with me. Leadership expectations must indeed be situational. For employees who were at the beginning of their career, much more direction and mentoring must be given, compared to those who have experience and knowledge of their craft. In addition, the ability to identify where a person was in relation to their career was a skill that is more than just a review of their resume of work. Having effective one-on-one meetings with direct reports is an important activity toward understanding the individual’s progress and building a trusting relationship. Early in their career, a more directional approach is required. As the staff member matures in their responsibilities, the transition to more of a coaching model is preferred as they learn to handle the responsibility delegated to them.

I think that excellent communication skills, including both speaking and listening, are the most important proficiency that a manager needs. Communication is always better received when coming from a person with whom you have a relationship. Using that skill effectively will help achieve personal and professional satisfaction, and will most likely propel a qualified manager to a higher level of career growth.