Testing vs. Analysis

There is a clear distinction in white box testing, between “testing” and “analysis”.  I think this distinction is important to call out.  On the one hand, we have tests, which are defined as:

any activity which results in a binary pass/fail result.

We also have analysis, which does not fulfill the above requirement but must nonetheless be performed.  These activities include:

• code coverage analysis
• static code analysis
• dynamic code analysis
• performance analysis

When examining the work and expected deliverables of a QA team, it’s important to remember that the second group of items above is not testing, and will return no pass/fail status.  The fact that it does not means that it is fundamentally different from tests, and needs to be reported to management in its own way, separate from test results.

Lean Testing

I’m currently in an Operations Management class as part of the MBA program I’ve been working on.  One of the main concepts the class introduces is “lean manufacturing”, which was popularized by Toyota.  A main concept of it is “the seven types of waste” (or muda).  They are:

1. Transport (moving products that is not actually required to perform the processing)
2. Inventory (all components, work in process and finished product not being processed)
3. Motion (people or equipment moving or walking more than is required to perform the processing)
4. Waiting (waiting for the next production step)
5. Overproduction (production ahead of demand)
6. Over Processing (resulting from poor tool or product design creating activity)
7. Defects (the effort involved in inspecting for and fixing defects)

Related to this, I discovered “Lean Software Development – An Agile Toolkit” by Tom and Mary Poppendieck.  This book, published in 2003 (there is also a sequel), tries to marry lean manufacturing with the world of software development.  It is an area I’m especially interested in, especially in the realm of software testing.

I’ll be exploring the above areas of waste as they relate to testing in upcoming posts.

Why which scripting language you use (sometimes) doesn’t matter

In discussing languages for testing, often the discussions can turn into debates, and the debates can turn into “religious” wars.

Ruby!

Python!

Perl!

And the list goes on…

But in some cases, the scripting language doesn’t matter all that much.  At least, not for testers.  All the scripting languages have some very basic functionality which makes them ideal for testing:

• test frameworks
• exception handling
• regular expressions
• vast module libraries to deal with things like database access, operating system functions, network connections etc.

And, sometimes it is more important what you’re trying to do than how you’re trying to do it.  Let’s think of, to pay homage to Brian Marick, an example.

Below is a basic example of a web application, and a web services interface to it.

As you can see, the application can be accessed via the browser.  However, there is also a web services interface (accessed via HTTP basic authentication) where functionality can be exercised simply by passing a URL to the server from a scripted web client.

This allows a tester to write automation in virtually any scripting language to automate actions on a web application, without having to go through the user interface.  This could render which language you use to automate your tests (almost) irrelevant. 

However, I still like Ruby.

“Best Practices”…Awful…or just irrelevant?

I’ve been having a discussion on Twitter with some folks about the “best practices” term…many people hate it.  The argument against it is:

• “Best” compared to what?  Best that’s ever been?  Seems fairly high-minded to call something “best”
• There’s no context.  Practices are more or less suitable depending on their context
• Calling a practice “best” invites contention because it is also a statement about the inferiority of other practices

Here’s my response:

• I think anytime you use qualitative terms (“best”, “good”, “practice i like”), there’s subjectivity involved, and deifnitions are imprecise.  So unless we want to throw out all subjectivity (which seems impractical), we have to live with the ambiguity
• I generally use “best practice” as shorthand for “the best practice we as a team can devise, given our current situation.”  “Best practice” is just shorter.  I guess you could just say “practice”, which would be fine.
• I just don’t think it matters much.  I’ve never, ever been on a software project where the use of the “best practice” term caused a heated debate or caused software to ship later or with less quality.  So, it seems to me to be a quest for preciseness for preciseness’ sake.

Those are my thoughts…I’d be interested in hearing others…

My Presentation for PNSQC

Can be found here.

Playing with Celerity

I’ve been playing with JRuby lately.  One of the big reasons we picked Ruby for our test framework instead of JRuby (even though we’re testing a Java application) was Watir, and the fact that JRuby didn’t support it (or it didn’t support JRuby, depending on how you look at it).  I’ve recently been exposed to Celerity (http://celerity.rubyforge.org/).  It looks pretty cool.  Here are some features of it:

• It’s written for JRuby, so you have access to Java and any Java classes you might need to interface with
• The API is based on Watir, so it should be pretty familiar to test developers used to working in Watir
• It lays on top of Java’s HTMLUnit, so it works a little differently than Watir.  Instead of driving an actual browser (like IE), it communicates with a web server via HTTP – so it basically emulates a browser.

So far, I like it alot.  However, one big issue I’ve been having (which is actually an issue with HTMLUnit) is that in a browser, if I go to a site that uses SSL I get a warning page and I can tell it to ignore certificate issues.  However with HTMLUnit/Celerity, you can’t do that.  It expects SSL sites to have a certificate.  This causes Java exceptions if you don’t have one (like on a testing site).

I’m still playing with it so maybe there is a workaround.  I hope so – I could see Celerity really filling a niche in the testing world.

Why you should version-control your tests (and why they should be stored with dev code)

You should absolutely version-control your tests.  Alister Scott talks about this in a recent blog post on his WatirMelon blog.  He also goes through the steps to easily set up an SVN repository on a shared Windows drive.

I’d go a step further, though.  I think it is a no-brainer that the automated tests need to be stored in the same VC system as the code.  Here’s why:

So, let’s say you’ve built a set of tests for a build.  You’re ready to ship, and the code is branched.  If you have your automated tests in a separate place from the code, you also have to branch the tests, and maintain the branch and (hopefully) have everything aligned properly.

However, let’s say in an alternate (happier) universe, you’ve simply created a directory off your /trunk called qa, which is at a peer level with ‘dev’ (or whatever directory your source code is located in).  So, your directory would look like this:

trunk
╚ dev
╚ qa

Now, all you do when you do a release is tag and branch everything under \trunk.  Then, all the tests appropriate for that version of code will be branched along with the code.