(Never) Always Set Up QA Before Production

June 23, 2009 by ADMIN

Share |

By Gene Kim, CTO of Tripwire and co-founder of the IT Process Institute

A buddy of mine is head of information security at a large insurance company, and we were talking about a common area of passion for us: implementing controls in pre-production.

He told me about an argument that came up between him and his QA manager. This QA manager was already getting harassed by the rest of the business for delaying needed software releases and needing more QA engineers each year to keep up.

: Learn More About Tripwire Here

To make matters worse, they found out that one of their pre-production environments was accidentally patched ahead of the production environment. Now they had to figure out what to do about it.

My buddy had made the following argument:

“My job is to protect the shareholders. A bunch of us, including the QA manager, are always complaining about how the Development and QA environments were always too lax and open, and that we needed to lock them down. That’s why we need to patch them — it just doesn’t feel right to rollback the patches. We should keep the QA systems as is.”

After my buddy told me this story, I shared my opinion, but wasn’t able to verbalize why it was true. But over the weekend, I couldn’t help thinking about this problem. And then I remembered about a problem that I’m working on in two of the largest Internet companies, and suddenly the correct answer was obvious.

On Twitter, I posted this question, with a Visible Ops book to the best answer.

“When is it acceptable to patch the QA environment ahead of the production environment?”

If you believe that the goal of QA is to test that application code will operate properly with the production databases and OSes, then the answer is… Drum roll, please… NEVER!

When the QA systems are patched ahead of the production environments, several undesirable things will likely result:

The code fails to function as designed in QA, because of differences in the OS, libraries, databases, etc. When this occurs, QA must figure out whether it is a code failure or an environment failure. Either way, QA is spending more time on non-productive work, most likely slipping the release schedule or forcing QA to skip tests to keep the schedule.
This situation is worse than the first. Here, the code works in QA, and then the code is then deployed into production, which then fails spectacularly. Now the problem isn’t that the QA schedule is slipping. Now the problem is that a potentially mission-critical service is down, and we have a potential Sev 1 outage, requiring the best Ops, QA and Development people to figure out how to restore service.

In the second scenario, it appeared that QA did all the right things, but the deployment still failed.

My long-time collaborator Kevin Behr noted, “It sure does seem like the entire point of QA is inextricably linked to production though?”

Absolutely. So, we have two winners. The first winner is Jonathan Katz (Twitter handle @katzmandu), who wrote :

“[You should] have OS patches in-sync the release cycle; as you promote code you test the code + OS + app together.”

Perfect!

The second winner is @sabletek, who had a long answer that I won’t quote exactly. But, what I believe he was stating very precisely is that you can patch QA early, if you are weighing the risk of patching production vs. delaying the patch to the next release and actively managing the QA vs. production variance risk. He notes,

“I’m making some large inherent assumptions regarding not only QA but patch management as well.”

Brilliant stated.

One honorable mention, because he pointed out a very good wording error in my question. Nicholas Weaver (@lynxbat) writes,

Wouldn’t the answer be “always”? I mean, you have to QA the patch…”

Crap, yes! That is also true. If you’re not testing application functionality, then yes, you should always deploy in QA before production.

But, I think @katzmandu gives the more complete answer: in order to predict outcomes, all the components have to be tested together.

Okay, enough theory.

Questions or comments? Feel free to send me a note on Twitter! I’m @RealGeneKim.

Gene Kim is the CTO of Tripwire, Inc. and co-founder of the IT Process Institute. He is currently actively working on a series of cross-industry projects to capture and codify how “best in class” organizations have IT operations, security, audit, management, and governance working together to solve common objectives. Gene co-chaired the “Generally Accepted IT Principles Summit” with the Institute of Internal Auditors in July 2005 to help codify how to create reasonable IT audit scope for SOX-404. In 2004, he co-wrote the Visible Ops Handbook, codifying how to successfully transform IT organizations from “good to great.” In 2003, he co-chaired two conferences with SANS and the Software Engineering Institute, and was named by InfoWorld as one of the “Four Up and Coming CTOs to Watch.” Gene is certified on both IT management and audit processes, possessing both ITIL Foundations and CISA certifications.

Tripwire helps over 6,500 enterprises worldwide reduce security risk, attain compliance and increase operational efficiency across virtual and physical environments. With its industry leading configuration assessment and change auditing software solutions, IT organizations achieve and maintain configuration control. Tripwire is headquartered in Portland, Ore. with offices worldwide.

* * *

Stay Informed With ISR News Feeds and Email Alerts Here:

The Publisher gives permission to link, post, distribute, or reference this article for any lawful purpose, provided attribution is made to the author and to Information-Security-Resources.com

These icons link to social bookmarking sites where readers can share and discover new web pages.