Jim Knowlton's Software Testing Blog

Testing vs. Analysis

2010-12-30T13:48:00.001-08:00

There is a clear distinction in white box testing, between “testing” and “analysis”. I think this distinction is important to call out. On the one hand, we have tests, which are defined as:

any activity which results in a binary pass/fail result.

We also have analysis, which does not fulfill the above requirement but must nonetheless be performed. These activities include:

code coverage analysis
static code analysis
dynamic code analysis
performance analysis

When examining the work and expected deliverables of a QA team, it’s important to remember that the second group of items above is not testing, and will return no pass/fail status. The fact that it does not means that it is fundamentally different from tests, and needs to be reported to management in its own way, separate from test results.

Lean Testing

2010-09-15T13:38:00.001-07:00

I’m currently in an Operations Management class as part of the MBA program I’ve been working on. One of the main concepts the class introduces is “lean manufacturing”, which was popularized by Toyota. A main concept of it is “the seven types of waste” (or muda). They are:

Transport (moving products that is not actually required to perform the processing)
Inventory (all components, work in process and finished product not being processed)
Motion (people or equipment moving or walking more than is required to perform the processing)
Waiting (waiting for the next production step)
Overproduction (production ahead of demand)
Over Processing (resulting from poor tool or product design creating activity)
Defects (the effort involved in inspecting for and fixing defects)

Related to this, I discovered “Lean Software Development – An Agile Toolkit” by Tom and Mary Poppendieck. This book, published in 2003 (there is also a sequel), tries to marry lean manufacturing with the world of software development. It is an area I’m especially interested in, especially in the realm of software testing.

I’ll be exploring the above areas of waste as they relate to testing in upcoming posts.

Why which scripting language you use (sometimes) doesn’t matter

2010-02-24T09:51:00.000-08:00

In discussing languages for testing, often the discussions can turn into debates, and the debates can turn into “religious” wars.

Ruby!

Python!

Perl!

And the list goes on…

But in some cases, the scripting language doesn’t matter all that much. At least, not for testers. All the scripting languages have some very basic functionality which makes them ideal for testing:

test frameworks
exception handling
regular expressions
vast module libraries to deal with things like database access, operating system functions, network connections etc.

And, sometimes it is more important what you’re trying to do than how you’re trying to do it. Let’s think of, to pay homage to Brian Marick, an example.

Below is a basic example of a web application, and a web services interface to it.

As you can see, the application can be accessed via the browser. However, there is also a web services interface (accessed via HTTP basic authentication) where functionality can be exercised simply by passing a URL to the server from a scripted web client.

This allows a tester to write automation in virtually any scripting language to automate actions on a web application, without having to go through the user interface. This could render which language you use to automate your tests (almost) irrelevant.

However, I still like Ruby.

“Best Practices”…Awful…or just irrelevant?

2009-10-30T09:16:00.000-07:00

I’ve been having a discussion on Twitter with some folks about the “best practices” term…many people hate it. The argument against it is:

“Best” compared to what? Best that’s ever been? Seems fairly high-minded to call something “best”
There’s no context. Practices are more or less suitable depending on their context
Calling a practice “best” invites contention because it is also a statement about the inferiority of other practices

Here’s my response:

I think anytime you use qualitative terms (“best”, “good”, “practice i like”), there’s subjectivity involved, and deifnitions are imprecise. So unless we want to throw out all subjectivity (which seems impractical), we have to live with the ambiguity
I generally use “best practice” as shorthand for “the best practice we as a team can devise, given our current situation.” “Best practice” is just shorter. I guess you could just say “practice”, which would be fine.
I just don’t think it matters much. I’ve never, ever been on a software project where the use of the “best practice” term caused a heated debate or caused software to ship later or with less quality. So, it seems to me to be a quest for preciseness for preciseness’ sake.

Those are my thoughts…I’d be interested in hearing others…

My Presentation for PNSQC

2009-10-27T11:28:00.000-07:00

Can be found here.

Playing with Celerity

2009-10-07T12:07:00.000-07:00

I’ve been playing with JRuby lately. One of the big reasons we picked Ruby for our test framework instead of JRuby (even though we’re testing a Java application) was Watir, and the fact that JRuby didn’t support it (or it didn’t support JRuby, depending on how you look at it). I’ve recently been exposed to Celerity (http://celerity.rubyforge.org/). It looks pretty cool. Here are some features of it:

It’s written for JRuby, so you have access to Java and any Java classes you might need to interface with
The API is based on Watir, so it should be pretty familiar to test developers used to working in Watir
It lays on top of Java’s HTMLUnit, so it works a little differently than Watir. Instead of driving an actual browser (like IE), it communicates with a web server via HTTP – so it basically emulates a browser.

So far, I like it alot. However, one big issue I’ve been having (which is actually an issue with HTMLUnit) is that in a browser, if I go to a site that uses SSL I get a warning page and I can tell it to ignore certificate issues. However with HTMLUnit/Celerity, you can’t do that. It expects SSL sites to have a certificate. This causes Java exceptions if you don’t have one (like on a testing site).

I’m still playing with it so maybe there is a workaround. I hope so – I could see Celerity really filling a niche in the testing world.

Why you should version-control your tests (and why they should be stored with dev code)

2009-08-27T10:30:00.000-07:00

You should absolutely version-control your tests. Alister Scott talks about this in a recent blog post on his WatirMelon blog. He also goes through the steps to easily set up an SVN repository on a shared Windows drive.

I’d go a step further, though. I think it is a no-brainer that the automated tests need to be stored in the same VC system as the code. Here’s why:

So, let’s say you’ve built a set of tests for a build. You’re ready to ship, and the code is branched. If you have your automated tests in a separate place from the code, you also have to branch the tests, and maintain the branch and (hopefully) have everything aligned properly.

However, let’s say in an alternate (happier) universe, you’ve simply created a directory off your /trunk called qa, which is at a peer level with ‘dev’ (or whatever directory your source code is located in). So, your directory would look like this:

trunk
╚ dev
╚ qa

Now, all you do when you do a release is tag and branch everything under \trunk. Then, all the tests appropriate for that version of code will be branched along with the code.

“What should we automate?” – A (partial) answer to the age-old question

2009-08-19T12:57:00.000-07:00

OK, I tricked you a little bit. This might not be “the” answer to that age-old question, but it is “an answer”.

As Ynigo Montoya said in the movie The Princess Bride, “Let me explain…no, let me sum up.” On my project, we have basically three types of automation (from lowest-level to highest-level):

unit tests
acceptance tests (below the UI, above the code)
black-box acceptance tests (driving the UI)

It seems to me to be a truism that the lowest-level test, all other things being equal, brings the greatest value. So, a unit test, since it is run at checkin and has low cost, has tremendous value to drive quality. That being the case, as an acceptance test developer, my first question should be, “what is the unit test not covering?”. I can find this out by perusing the latest code coverage report for the application. With this information, I can then plan acceptance tests to “plug the holes” wherever they might be in the unit test set.

Ideally, a product should be mostly covered with unit tests. However, out in the real world where QA engineers live, this is almost never the case. Building acceptance tests to plug the holes can insure that the full product functionality is covered by a combination of unit and acceptance tests, and then the black-box testing (both automated and manual) can lay on top of that to validate the UI functionality.

“Making Things Happen” – a first look

2009-08-13T13:02:00.000-07:00

I’m reading a book on project management called “Making Things Happen”, by Scott Berkun. It’s a great book. One of the things I like about it is that it shows the universality of good project management principles. Of course, software engineers can find value in the book, and that was my initial purpose in picking it up at my local bookstore. However, as I read, I got one of those “aha” moments when you realize something you already knew, but hadn’t thought about in quite that way before.

When I speak of the universality of project management, I’m speaking muuuuch more broadly than just the “engineer-business person” spread that people typically refer to. No….here we’re talking Egyptian pyramids:

The history of engineering projects reveals that most projects have strong similarities. They have requirements, designs, and constraints. They depend on communication, decision making, and combinations of creative and logical thought. Projects usually involve a schedule, a budget, and a customer.

Most importantly, the central task of projects is to combine the works of different people into a singular, coherent whole that will be useful to people or customers. Whether a project is built out of HTML, C++, or cement and steel, there’s an undeniable core set of concepts that most projects share.

I’ve just started reading the book…but I found this really interesting, because we have a tendency to think in the technology industry that our issues are so unique, so never-been-seen-before that there are no models. In fact, there are. They might not always be perfect models, but certainly there are things we can learn from the success and failure of those who have come before…sometimes a long time before.

I'm presenting at PNSQC

2009-08-13T12:42:00.000-07:00

I'll be presenting at the Pacific Northwest Software Quality Conference (PNSQC) in October on "Web Security Testing with Ruby and Watir". Come say hi! It looks like I'll be presenting on Oct 27th, in the afternoon...though schedules are fluid, so don't hold me to it.

Click here to view a preliminary draft of my paper.

Why Rake is a great tool for test run automation

2009-07-20T13:36:00.000-07:00

The more I use Rake for automation of our testing process, the more I like it. Where I work, we have a Java-based web application. However, I have found that using Rake, not Ant, is definitely the tool of choice for automating the configuration and running of our QA tests. Here are some of the reasons:

Rake uses Ruby scripts as their ‘rakefile’ – Ant uses XML files, which are great for storing structured data but have no real ability to perform logic or make decisions. Rake, on the other hand, uses actual Ruby syntax within their ‘rakefile’, which means you have full access to the Ruby ecosystem. This allows you to utilize decision structures, closures, even classes if you desire.
Rake is fully supported by Intellij and TeamCity – I am not sure what Rake’s support is in other environments, but our development/automation environment is Jetbrains’ Intellij for the IDE and TeamCity for the build automation. Both have terrific support for Ruby and Rake.
Rake has great support for RSpec – RSpec is our test framework…and Rake has great support for it.

Some of our tests in the future might be Jython tests run from the server (under the JVM), but they can still be easily launched from a Ruby/RSpec script (via a web services API). This is ideal, because it gives us the best of both worlds…the ability to quickly script tests through Ruby when possible, but also the ability to access the product’s underlying classes when we need to.

2009-07-08T08:56:00.000-07:00

Trying out my new look...tell me what you think!

Java properties files - in Ruby

2009-07-07T09:29:00.000-07:00

One nice thing about Java is its use of properties files, which are text files with a .properties extension that contain key-value pairs of data. This way, changeable data can be stored in a file separate from the code. There is a way to do this in Ruby too, with YAML files and the Ruby YAML module.

First, create a YAML file. This is simply a text file with a .YAML extension, and with key-value pairs of data seperated by a colon. For example, let’s say you wanted to store a user name and password to run your tests with (for illustration purposes – normally it’s not a good idea to store a password in a text file). You’d set up your properties.yaml file thus:

username : jim
password : mypassword

Save the file. Now, you can access those properties through Ruby. Since I love using IRB to test functionality interactively, I’ll recommend you now fire up IRB. For this example, run IRB from the directory you created the YAML file.

Once IRB is up, require the YAML module:

require ‘yaml’

Now, load the properties file and assign it to a variable:

properties = YAML.load_file( './properties.yaml' )

Once this is done, you can use the properties from your YAML file. Let’s say you had a login() method that takes two parameters, the username and password. You could at this point run your method thus:

This will run your login() method with the username and password you defined in your properties.yaml file.

Design principles:

For properties that will not change, I’d suggest you can define those in helper modules.
For properties that will change in different environments, put them in a YAML file. This makes it much easier to customize a deployment.

Urgency vs. priority – an example and an assertion

2009-04-21T12:04:00.000-07:00

QA Engineer: So, manager, which is more important, A or B?
QA Manager: Well, engineer, they’re both important…they both have to be done.
QA Engineer: So how do I prioritize them?
QA Manager: (a little frustrated)…like I said, they both have to be done.

How many times has this scenario played out for you? Many times for me. So, let’s examine this problem with an example we all experience, our own need for oxygen, water and food.

All three are essential to our survival…without any of them, we will die. There is no way to prioritize them in order of importance. However, they definitely can be prioritized in order of urgency.

Without oxygen, you would be brain-damaged within minutes (and dead soon thereafter).
Without water, you would die within days (one website I read said the average person would be dead within 9 days of 80 degree days).
Without food, you can go months, but eventually you would die.

So, although all are essential in the long run, oxygen is most urgent.

Does this mean we should always focus on our most urgent needs? No. If you focused only on your most urgent need (oxygen), and never paid attention to less urgent but equally essential needs (food and water), you would still die, albeit more slowly.

I think the lesson of this allegory is that we sometimes ask the wrong question. A better question would be “which is more urgent to test, a or b?” And if both are equally urgent, as well as equally important, then the laws of physics come into play, and you lay out how long each will take, and the amount of time you have available.

One final note: this principle will certainly not alleviate the “I need everything done, and I need it yesterday!” issue. But I do think it sheds some additional light on the questions to ask when prioritizing.

Another thing to love about RSpec

2009-04-16T13:39:00.000-07:00

I’ve been busy working on refactoring our test framework, and I ran across a gem (spoken in the usual sense, not in the RubyGem sense).

RSpec has a method for running a block of code before and after each example, before(:each) and after(:each). Its syntax is pretty simple:

describe “this is my feature” do

before(:each)
do_this_before_each_example()
end
after(:each)
and_do_this_at_the_end_of_each_example()
end
it “should be example 1” do
end
it “should be example 2” do
end

end

This is pretty standard stuff. The cool part is you can also use a before(:all) and after(:all) method to do stuff (like major setup and teardown) once at the beginning of the describe block, and the after(:all) at the very end. Since it can be tied to any describe block, it’s completely flexible how big a block of example it applies to.

The more I use Ruby for testing…the less reason I have for looking at other languages.

Watir and security testing of web applications

2009-04-07T16:23:00.000-07:00

One of the tasks I have at work is security testing. Watir makes a great tool for security testing of web apps. There are several reasons for this:

Many of the common security vulnerabilities related to web applications (SQL Injection, cross-site scripting, buffer overflow) have to do with simply posting different types of information to a web server via a client. This is pretty much what Watir is all about. It even gives you access to hidden elements, so it really is a great tool for submitting form data to a web server.
The Ruby side of Watir, being a full-service language, has great tools for querying the database, checking audit logs and the like. Also, you can generate random data (or large datasets) to throw at a web app, or even pull the test data from a CSV file.
There are some things you might not be able to do through Watir, but can certainly be done with Ruby. Again, this is perfect – because Watir is not really a test framework, it’s just a way to drive the Browser when you need to. So, tests which are more low-level (such as web service communication or network tests) can be run through Ruby and RSpec, or whatever actual test framework you’re using.

In short, Watir makes a terrific tool in your arsenal for web security testing.

Watir Design Patterns?

2009-04-03T13:11:00.000-07:00

I think the biggest thing a lot of us Watir developers need is a good guide to design patterns with Watir. I know for myself, I know the commands and basic syntax pretty well, and can design a test structure that works, but I'm sure there are things I'm doing that could be designed better (I'm a tester, after all, not a full-time programmer). I'd love a guide that describes best practices for how to set up a directory structure, set up rake for common tasks, etc.

I know WatirCraft attempts to address this to some extent, but I guess I'm suggesting that some people might need to design their own framework, but not know how to do it.

My two cents.

Axiom Wednesday

2009-04-01T15:29:00.000-07:00

I’ve decided to start a tradition (April Fools’ Day is as good a day as any to start this, right?). I often come across axioms in software development…short, pithy statements that attempt to express some greater truth. Some are terrific. Some a little less so. Every Wednesday, I’ll explore one.

Today’s axiom is: Read more code. Ideally, read more code than you write.

I extracted this axiom from James Edward Gray II’s great presentation at MountainWest RubyConf 2009. It really makes perfect sense…if you want to be a great writer, you read. Lots. None of us would respect a writer who didn’t read much. Yet it’s all too common (I’m guilty too, I admit) to not really seek out code to read. With basically all of the Ruby universe being open-source, there is a treasure-trove of interesting code to look at.

One tip James gave at the conference was to not start out by finding the hairiest, most complex code to look at. find a module that is fairly simple and straightforward (actually, Watir is great for this), and look at the source to see how it’s put together.

Some advantages James mentions of reading code:

It can show you common idioms
It’s good to practice breaking down possibly challenging code, because you will always have to work with other’s code
Understanding how something works give you insight into any limitations it has
Seeing bad code helps you write better code
Knowledge workers always need more ideas

Why I chose Ruby

2009-03-31T10:53:00.000-07:00

At work, I write automated tests for a Java-based web application. I’ve had several people ask me why I chose to implement tests in Ruby instead of Groovy. The answer is actually pretty interesting.

First of all, we started out in Groovy, and still have some Groovy-based tests. I can see cases where we will still use Groovy. It is a great way to script JVM-based applications. However, for application tests, there were some components of Ruby that, in my mind, made it better as a test framework language:

Watir – There simply isn’t a Groovy-based or Java-based tool that is as full-featured, mature and easy-to-use as Watir. Some of you may retort, “what about Watij?”. Watij is an attempt to port the functionality of Watir to Java/Ruby. It’s very interesting, but hasn’t released an update in quite a while and in my tests with it, it wasn’t nearly as stable or “clean” as Watir.
RubyGems – One of my pet peeves about Java is that installing third-party modules and getting them to work is like a toothache, earache and sinus infection all at the same time. Ruby, on the other hand, has RubyGems. Basically, all I do is go to the ruby/bin directory, type ‘gem install <module name>’ and follow the prompts. That’s it. If there are dependencies, I’m prompted to install them. It’s clean, simple and works seamlessly. I actually think it is the best package-management system of any open-source language…period.
RSpec – Other than being able to describe tests in english as behavior I expect (which I really like), a feature I love about RSpec is that I can create a slick HTML report simply with a command-line switch when I run the tests. No Rake (or Ant, if you’re a Java-phile) tasks required, no nothing else required. Simple, and to the point.

What all of these technologies have in common tells us a lot about what is important to testers in a test development environment. All these tools:

Are simple to set up
Have abundant online documentation
Are very stable and solid…they work.

None of this pertains to why I love Ruby as a language (which I do). All these examples show that when it comes to test development, the availability of simple, easy-to-set-up, easy-to-use tools is absolutely critical. And it’s why I picked Ruby.

Oh, one other thing…the community rocks. I’ve often posted questions and gotten answers within an hour. Gotta love that.

Programming through vagueness

2009-03-30T16:21:00.000-07:00

It may sound counterintuitive, but one of the tidbits I’ve gotten from Russ Olsen’s Design Patterns in Ruby is the idea of writing classes and methods in as general a form as possible. He discusses an example of different types of vehicles and writes:

By writing code that uses the most general type possible—for example, by treating all of our planes and trains and cars like vehicles whenever we can—we reduce the total amount of coupling in our code. Instead of having 42 classes that are all tied to cars and boats and airplanes, perhaps we end up with 40 classes that know only about vehicles. Chances are that the remaining two classes will still give us trouble if we have to add another kind of vehicle, but at least we have limited the damage.

In fact, if we have to change only a couple of classes, we have succeeded in separating out the parts that need to change (the two classes) from the parts that stay the same (the other 40 classes). The cumulative effect of turning down the coupling volume is that our code tends to be less likely to shatter in a horrendous chain reaction in the face of change.

This has tremendous value for testing. Instead of writing methods like:

load_preferences_page()
load_users_page()
load_security_page()

we can create a single, general method and simply pass it parameters thus:

load_page(preferences)
load_page(users)
load_page(security)

This way, if the method needs to be changed, you only have to change one method, making the code easier to maintain and much less error-prone.

Pretty elementary for you object-oriented veterans…but cool stuff nonetheless!

Running “cleanup” the beginning of a test

2009-03-24T16:44:00.000-07:00

Often a test will execute a “cleanup” method to delete items (often out of a database) that were created or used in the test, to make sure the next test has a clean environment to run in. Sometimes though, the cleanup never gets run – if it is the last thing to run (as would seem logical), if there is a test failure it will never execute.

A better option is to run the cleanup method at the beginning of a test. If the method simply executes SQL scripts to delete records, if the records for some reason aren’t found, it won’t generate any kind of error, so the script will run normally. However, doing it this way (putting the cleanup at the beginning) insures the data cleanup will get executed on every test, insuring a clean environment for every test that gets run.

MountainWest RubyConf 2009 videos starting to appear…

2009-03-18T14:56:00.000-07:00

Get them here

Some quick hits from MountainWest RubyConf 2009

2009-03-18T09:44:00.000-07:00

I’ll write more on this later, but just wanted to drop a quick note on MountainWest RubyConf 2009, which I went to last weekend in Salt Lake City. I really loved the conference, and hope it continues to grow. I like that they don’t pack the conference with alot of expensive frills, and just basically focus on providing content. Here’s my quick takeaways:

Wow…lots of Macs. I feel like an endangered species here with my Windows laptop.
James Edward Gray II: we should be reading more code than we write. No one would think much of a writer who didn’t read much.
Sequel looks really cool. Jeremy Evans, who is the creator, gave this presentation. I have been just using DBI and sending SQL queries “natively”, but I really like the way Sequel Rubifies database access.
Jeremy McAnally was hilarious – and I learned some things too, but need to re-look at my notes to remember what those things were.
Ben Mabey gave an interesting talk on Cucumber…I’m still a little fuzzy on how it differs from RSpec, or what domain it occupies vs. RSpec…but I’ll definitely look into it further. The RSpec book covers Cucumber as well, so I’m sure I’ll delve deeper. Besides, any framework named after vegetation can’t be all bad.
Philippe Hanrigou spoke about learning lessons from Smalltalk…in a greater sense, he spoke of learning from the masters, which I think is a cool topic. I think we can learn from everyone who comes before us, and learning about DaVinci can make us better Ruby coders. Weird, huh.

All in all, a great experience. Some of it went over my head, and some was Rails-specific. But much of the conference was really useful, and I look forward to going back next year. Maybe I’ll see you there.

Why Ruby is great as a “sticky” test framework

2009-03-09T13:50:00.000-07:00

One great advantage of developing a test framework in a scripting language like Ruby is that it allows you to glue together different types of tests in a single framework.

Here’s a perfect example.

I had to write a test checking all the log files in a directory structure for a text string (in this case, a password). In other words, I wanted to make sure the password wasn’t stored in plain text anywhere in the log files. I found that Powershell has some awesome tools for doing this type of search on a Windows machine. So, I wrote a Powershell script to search the directory, and if a match is found, it creates a “fail.txt” file:

if (test-path fail.txt)
{
del fail.txt
}

$adminSearch = dir -rec -filter *.log 'c:\windows\temp' |
select-string "PASSWORD=thisisthepassword"
if (!$adminSearch)
{
    Write-Output("pass")
}
else
{
    Write-Output("fail")
    Write-Output([string]$adminSearch.count + " instances of passwords in log files: ") |
        out-file -encoding ASCII -filepath fail.txt
    Write-Output($adminSearch) | out-file -append -noclobber -encoding ASCII -filepath fail.txt
}

This done, I then write a Ruby/RSpec script that simply checks for the existence of the fail.txt file:

require 'spec'

describe "Information Disclosure Prevention" do

it "Should not show passwords in plain text" do
    system("powershell.exe -File ./lib/passwordsearch.ps1 -command \"& {set-executionpolicy unrestricted}\"")
    result = FileTest.exists?("fail.txt")
    result.should == false
end
end

And that’s it. As you can see, it’s easy to wrap RSpec examples inside any command/script that can be run from the command line.

Getting elements – Firebug to the rescue

2009-03-04T11:18:00.000-08:00

In using Watir, probably one of the biggest challenges is accessing elements that are buried inside several layers of Ajax/HTML. You can run the show_all_objects() method, but that doesn’t always show everything on the page, and it can be difficult to parse through manually. So what’s a test developer to do?

Firebug to the rescue.

Firebug is a Firefox plugin that provides several useful tools, but one is that it has a great DOM inspector that allows you to easily copy any HTML element on a page by simply clicking the “Inspect” button and clicking on the element to be inspected. It then allows you to easily copy the element text to the clipboard.

I will often use Firebug and watir-console together when building a test method. There are times when I’m not sure of the best way to access an element, but with watir-console and Firebug running, I can experiment and fiddle with different code blocks, and find out what will work best.

Check it out here.

Here’s a screenshot:

Playing with Celerity

2009-03-02T10:18:00.000-08:00

I’ve started playing with a cool port of Watir to JRuby…it’s called Celerity.

Here’s the website.

This is how it is described on the website:

Features

Fast - No time-consuming GUI rendering or unessential downloads

Easy to use - Simple API

JavaScript support

Scalable - Java threads lets you run tests in parallel

Portable - Cross-platform thanks to the JVM

Unintrusive - No browser window interrupting your workflow (runs in background)

One issue I’ve had with Celerity (and JRuby in general) is that Watir supports SSL out-of-the-box, whereas with Celerity (or JRuby), when I try to access an SSL address, I get a Java exception that the certificate can’t be found. I’m sure there is a workaround, but haven’t worked on it yet.

Other than that, I’ve played with it on a non-SSL website, and it seems to work exactly like Watir, only much faster and with no browser showing up. Pretty cool.

This might be a "duh!" thing but...

2009-02-26T14:46:00.000-08:00

it was a new concept for me. I always wondered why some of the methods in the Watir RDoc didn't have any documentation. I asked Bret Pettichord about it, and it basically the gist is that if a method isn't documented, you can assume it is a non-public interface (not to be called directly).

I'm sure this is probably how things are normally done in the programming-language world, but it was new to me. Thanks to Bret for providing this insight.

Integrating Ruby testing in Intellij and TeamCity...made easy

2009-02-26T09:41:00.000-08:00

One of the greatest things about scripting languages is that they can work entirely through the command line and require very little in the way of infrastructure.

For example, in my environment at work, our development environment is Java, and the tools are very Jetbrain-centric. They use Intellij for their and IDE and (more important for me) use TeamCity for continuous integration. This has two implications for me:

JetBrains has a Ruby plugin for Intellij which works really well, so I can use the same dev environment as our developers, even though I'm working in Ruby. It has code highlighting and code intelligence, and generally works well. I'm also playing with RubyMine, a Jetbrains all-Ruby IDE. It is based on Intellij so it has a great editor, but also supports Rake and has a nifty GUI test runner. Right now it's in "free preview mode", I'm told when it releases it will be $99, which seems pretty reasonable.
One requirement I had when I started working for McAfee is that our component-level tests integrate into our TeamCity environment, so that test runs could be scheduled to be kicked off and so that test results would be easily viewable by anyone on the team at the location they are already going to. This was easy, because TeamCity has a "command line" option for it's build runner. That means anything that can be kicked off through the command line can be scheduled to run. Of course, this includes RSpec-enabled Ruby scripts. Since RSpec includes an option to output to HTML, and since TeamCity has a way of pointing to build "artifacts" to view from their website, I just set up my configuration so that the script outputs to an HTML file, and then pointed to that HTML file as my build artifact. In this way, I can run Ruby tests and produce an HTML report in the TeamCity environment.

Customizing watir-console

2009-02-25T11:22:00.000-08:00

Watir comes with a really cool command-line tool called watir-console. To start it, just type watir-console at a prompt (assuming you have the ruby\bin directory in your path). It will bring up an irb prompt, but this isn't just any irb prompt. It has already required the watir module, and offers some cool features for interactively using Watir, such as tab completion of Watir commands.

One thing you can do to make watir-console even more useful is to write a startup script. Here are the steps:

Create a helper script and put it in one of the Ruby library directories (such as C:\ruby\lib\ruby\1.8). Let's call it helper.rb.
Create a .irbrc file (just a text file) and store it in \ruby\bin. This is actually a Ruby file, and can have any Ruby commands you want to start up when Ruby starts. For our purposes, your script should 'require' the script you built in step 1.

That's it! Now, helper.rb (and its methods and constants) will be available in watir-console whenever you run it.

"What should I put in helper.rb", you ask? I'm a believer in examples, so here are a few examples of methods I created:

A 'login()' method to start a browser, go to the login page for my web app and log in. It returns the IE object, so it can be called from watir-console as "instance = login()". This allows you to then access "instance" as the IE instance to be manipulated.
Now you can run different methods to do different things. For example, my application does queries, so I have a 'new_query()' method that simply navigates to the page where I can create a query. It is accessed by calling 'new_query(instance)' from watir-console.

This makes Watir and watir-console indispensable in interactively testing web applications.

Peter Cooper lists 23 useful Ruby 1.91 links...check them out, I have (well, most of them)

2009-02-24T07:41:00.000-08:00

Here's the link...

My personal favorite...Dave Thomas' "Fun with Ruby 1.9 Regular Expressions"

Ruby Design Patterns - an epiphany

2009-02-23T15:29:00.000-08:00

I’ve been reading Russ Olsen’s great book, Design Patterns in Ruby. I’ve always had my developer friends tell me design patterns are a great thing, the jelly to your peanut butter, the Martin to your Lewis if you are a software developer. I gave it the typical…”huh…I’ll have to check that out...sometime.”

No longer.

As have been reading Olsen’s book, I am realizing, it has far-reaching implications that go beyond just writing a Ruby class efficiently. For instance, one of the principles is “separate the things likely to change from the things likely to stay the same.” This sounds simple…and it is. But it’s also incredibly powerful. Have you ever created a large suite of manual test steps, only to have the software change and need to re-write virtually all of them? If you apply this principle, your manual test scripts will be more durable and more flexible. And it didn’t involve writing a single line of Ruby (or any) code. It simply involved applying a true principle.

Take a look at this book…it will definitely cause you to think in a different way. And that’s always a good thing.

Brett Pettichord working on new Watir testing framework

2009-02-21T15:43:00.000-08:00

Brett Pettichord's latest blog post has some information on a new Watir-based testing framework. It looks pretty interesting. Since I use Watir extensively for my testing work, I have a particular interest in Brett's work.

I guess my initial question is...what exactly does it provide that Watir/RSpec doesn't? I might download and play with it some...will blog about my impressions.

MountainWest RubyConf 2009 - see ya there

2009-02-21T10:21:00.000-08:00

I’ll be at the Mountain West RubyConf 2009 in Salt Lake City March 13-14…if you’ll be there, look me up. I hope to make some good contacts in Rubyland and learn some stuff. Besides, my work is paying for the conference, so who can beat that?

Also, I may be presenting to the Software Association of Oregon (SOA) in the near future about Ruby, Watir and RSpec (and how to build a web testing framework using them). Stay tuned.

The idea behind this blog

2009-02-21T10:20:00.000-08:00

I’m a software qa engineer who works with Ruby, Watir and RSpec every day, and I noticed there weren’t any blogs out in the ’sphere for folks like me. So, here I am. I often have more questions than answers, but I figured maybe if I just start writing, others might find something here that’s useful too.

Welcome to my black-cloud-qa-type, Ruby world.

"The Simplest Thing that Could Possibly Work"...or not

2008-07-18T08:31:00.000-07:00

In a recent blog post, Jay Fields argues that the canonical (at least in the agile world) "Simplest Thing that Could Possibly Work" tenet might need some caveats:

Often the emphasis of the phrase is like so: the simplest thing that could possibly work. However, recently Ian Robinson was discussing the idea that the emphasis should be: the simplest thing that could possibly work. The difference is only emphasis, but it's a change worth considering.

I'd suggest that we go one further, and emphasize it as the simplest thing that could possibly work. When you think about it, what does it mean to work?

To successfully compile?
To run without crashing?
To be well-designed?
To be easily maintainable?
To please users?

All these could be used as criteria for "working", given a particular context.

In my mind, ultimately the most important question to ask in applying this maxim is, "how do I define 'work'?"

Groovy Thoughts

2008-07-16T22:36:00.000-07:00

I've been diving into Groovy lately...I just started a new job a few weeks ago at McAfee, and most of our application is written in Java. As a dynamic-language lover, it chaps me to think of spending all my time writing tests in Java. Writing tests (in this case, white-box tests aimed at business logic level functionality) has always seemed to me to be ideally suited to scripting languages.

I've looked at Jython and JRuby, and, as much as I love both those languages, it would seem to me that I want to leverage the Java knowledge as much as possible that exists here. Hence, I'm looking at Groovy. I'm reading Venkat Subramaniam's "Programming Groovy", and love the language so far. It seems to bring much of the dynamic language features (dynamic typing, optional semicolons, closures, automatically importing a bunch of Java classes) that I'm looking for. And it's a much easier sell to a Java shop than trying to get them to bite on a language (such as Java or Ruby) that's a total departure from what they have invested in.

It would seem we are exactly the type of Java users Groovy is aimed at. I'll blog more on my thoughts of Groovy (and I'll write a review of Venkat's book) in future posts.