• 16:43 Downloading yet another IE 6 VHD. Bonus: IE 8 beta VHD.
• 16:52 Fyi - IE VHDs - tinyurl.com/y64upm

Automatically copied from http://www.twitter.com/jenk3 via LoudTwitter cause it's easy :)

Faulty character set for text messaging results in misunderstanding which results in assault and death.

Because of course if an insult is on a computer / cellphone screen, it must have been deliberately sent and cannot have been a bug or typo. And of course an insult cannot be mended with an apology.

There are days when I really do not understand humans.

...is that often the metrics aren't what you really wanted. In tech support, "shorter calls" were a big metric. Shorter calls meant more customers were served and problems were solved more efficiently, right? Right?

Some calls, yes. Perhaps even the 50% or 80% most common calls. But not the less common calls. Not the troublesome calls. Those took longer, and one 1/2 hour call would screw up your stats for the day.

"Of course you should take the time to do the job right, just not more time than necessary," management would chirp. And yet the rewards went to those who did the job quickly - and not necessarily well. Since people repeat the behavior that gets rewards, this led to less service.

In testing, focusing on bug counts can be a similar problem¹. My first test lead didn't look at how many bugs were reported by each individual on his team. Instead, he read every bug reported by his team. He found it was very clear who was breaking bugs into tiny little bits so's to inflate bug counts (often a habit acquired elsewhere), who was sloppy in figuring out repros, who was noticing that two different symptoms had a common cause and reporting it as such, who was mostly finding bugs in areas other than their own (which if a conversation determined their area just didn't have many bugs to find could be a good thing) and so on.

Of course, it was expected that a test lead would read and know all the bugs for the areas covered by his team. And my first test lead did so. Among other things, this made him aware of what was being reported by those not assigned to test the area. (Often these were bugs on areas that weren't officially released for testing yet and such. But sometimes it was a clue that tests needed revising.)

Sometimes the focus on bug count goes beyond using it as a measure of tester productivity. The Defect Black Market relates just one example. This Dilbert shows a slightly less, er, subtle approach.

¹The problems with using bug counts as a measure of tester productivity are discussed in-depth in Testing Computer Software by Kaner et al.

Oklahoma sex offender registry exposes and executes SQL statements in the URL, enabling downloads of social security numbers, birthdates, addresses, et cetera. Who knows, maybe their site executed INSERT statements too.

Nobody accessing sensitive government databases should assume that users don't know SQL. And yet.

...from The Wall Street Journal:

Talk to your information-technology department before you post naked pictures of Lindsay Lohan on your business’s Web site. The latest reminder of this ironclad rule comes from New York Magazine, which posted photographs of the 21-year-old actress recreating one of Marilyn Monroe’s more scandalous shoots on its site yesterday. (Sorry fellas, the link is to an article.) The site, which was promptly overwhelmed by art-photography fans, ground to a halt. It’s a classic Internet-age problem: The marketing department has some promotion that is bound to drive traffic to a site, but never bothers to tell IT. So the site crashes. In a testament to human nature, these incidents often involve scantily-clad women, such as the famous Victoria Secret online fashion show that nearly brought down the Internet in 1999.

I've been on both sides of that equation - both times involving that little low-traffic domain called microsoft.com. ;)

I haven't run the Windows debugger (or the Windows debug kernel) since I quit being paid to do so in 1999. But when I did, I made use of crib notes created by Raymond Chen, a Windows dev who also opened a wide variety if interesting and useful bugs.

A few days ago, Raymond reviewed the book Advanced Windows Debugging, stating that "Even the section with the "Oh come on every moron knows this already" title Basic Debugger Tasks has stuff that I didn't know."

Eep.

So if you're doing Windows programming and stepping through the code in the IDE isn't enough to figure out what's going on ... if you're dealing with, say, a deadlock or heap corruption ... this might be the book for you. :)

Seattle Weekly article on the day labor side of testing, games testing in particular. Reading the article, it's not clear if the writer thought this was a "slice-of-life" article or an exposé.

As exposés go, well, maybe I'm jaded. Sure, it's not a job I would want. On the other hand, it sounds better than other jobs I've had. In the late 80s, I got $6/hr for kid wrangling at a non-profit. According to the BLS, low-level child care a median wage of $8.25/hr now, comparable with the low-level games testing discussed in the Weekly.

Duties? I would keep 10 preschoolers engaged and safe. I also tied 8 or 9 pairs of shoes at a time; cleaned kids (with wipes) and cots (with Lysol) after "napping accidents"; made lunch or snack for 100; cleaned tables, chairs, and the floor after 100 kids had lunch; held a screaming kid under the sink while washing sand out of his eyes (had bruises for a week after that one); carried upset or sick kids; read out loud a lot; organized; cleaned; turned jump ropes; et cetera.

And, of course, there's also the bit about how screwing up as a games tester means a support call, maybe a recall, or being fired. In a day care, negligence can cause a child's death.

Not that child care is the worst job in the world, either. Yes, you deal with bodily fluids (and yes, waste solids) fairly regularly, but they are pretty well contained. Working in a center meant I had other adults for comradarie and assistance. As jobs go, it was steady, had flexible scheduling, and included vacation, sick time, and health insurance.

But I can't help wondering if an article on how day care isn't all hugs around the knees and experiencing wonder through the eyes of children might be a nice companion piece to the one on how low-level games testing isn't the happiest job on earth.

"A busy CPU is a happy CPU. Leave it running [pseudorandom tests] all weekend, it'll be happy to see you Monday." - Harry Robinson from Google, talking on "The Bionic Tester"

"I am certified to practice law. This means I can be sued for malpractice. I like this." - Cem Kaner

I start posts on this and then get busy and abandon them. A few folks who have posted are here.

I found a ruby library for testing in IE & I'm working through the tutorial. So I've got IE and the irb window up, and I'm typing

ie.button(:value, 'Login').click

instead of clicking the Login button.

So yeah, it's for scripting. But it's also an oddly verbose way to rest my mouse hand....

I ran across this quote in a software testing article at Jonathan Kohl's blog.

Weinberg's "Rule of Three", which goes something like this: if you can't think of three ways a potential solution can fail, you don't completely understand the problem space.

Google found a couple other variations.

Jerry Weinberg says, “If you haven’t thought of three possibilities, you haven’t thought enough.”
- quoted at stickyminds

and

Whenever I'm aware that I'm making an interpretation, I have another choice: I can allow myself to know that more than one interpretation is possible. A good check on premature interpretation is the Rule of Three Interpretations:

If I can't think of at least three different interpretations of what I received, I haven't thought enough about what it might mean.

This rule slows down the Interpretation step and gives me, the receiver, a chance to engage my brain before using my mouth. Even after I have thought of three possible interpretations, however, I should always be aware of one more possibility: that my list still may not include your intending meaning.
-- Jerry Weinberg, Quality Software Management Vol 2, Chapter 6, quoted here

So, here are three varations on the Rule of Three...appropriate, no?

David Gilbert discussed applying hurricane forecasting methods to software. One thing that caught my eye, that he says he has "NEVER seen [in] any model of software planning", is "The Cone of Uncertainty".

Starting at the point where the storm is now, following roughly along the predicted path of the storm, is an ever expanding cone. This cone represents where, given all the information currently available, the storm MAY go if something in the model changes. [...T]his is the part of the model that forecasters talk about the most, and encourage everyone to pay the most attention to. This is the part of that model that takes into account Risk and Ambiguity.

Shipping software is ambiguous, and taking that into account is a good thing. But (to continue the metaphor), project management is all about getting to a particular destination at a particular time; tracking a hurricane is trying to see where & when it will arrive.

But perhaps there's room for both. Yes, aim for the goal. But also track the "cone", and forecast how far off track the project may currently get...

There's a lot of testing buzzwords that just make me sigh. But I like "Context-Driven". The link has good info, but the example was the "aha" moment for me:

Consider two projects:

1. One is developing the control software for an airplane. What "correct behavior" means is a highly technical and mathematical subject. FAA regulations must be followed. Anything you do -- or don't do -- would be evidence in a lawsuit 20 years from now. The development staff share an engineering culture that values caution, precision, repeatability, and double-checking everyone's work.


2. Another project is developing a word processor that is to be used over the web. "Correct behavior" is whatever woos a vast and inarticulate audience of Microsoft Word users over to your software. There are no regulatory requirements that matter (other than those governing public stock offerings). Time to market matters -- 20 months from now, it will all be over, for good or ill. The development staff decidedly do not come from an engineering culture, and attempts to talk in a way normal for the first culture will cause them to refer to you as "damage to be routed around".

Testing practices appropriate to the first project will fail in the second.
Practices appropriate to the second project would be criminally negligent in the first.

The testing is designed for the project, not the project for testing.

I ran across this at Testing Reflections:

Lesson 1: Know your enemy

• Lack of Quality is not your enemy, it’s [the] developers enemy. Your enemy is absence of evidence of it.

Lesson 2: Know your equipment and how to use it

• Armour: standards, guidelines (universal defence against any attacks)
• Shield: test [plans], test cases (when used properly may block some attacks by “the enemy” – do you remember who is an enemy?)
• Sword: testing - test execution (use to attack the enemy)

Lesson 3: Choose appropriate equipment

• Good armour reduces movement.
• Shield decreases offence.
• A single soldier may use long sword (knight) or two swords (samurai)…
• An army [can] use good armour, big shields, normal swords

A lot of this makes sense. Lesson 1: We are looking for facts. Lesson 3 is about tradeoffs...and how test plans are like battle plans - they don't survive initial contact unscathed!

Other thoughts?

Raymond talks about some locale-specific notes on number formatting, including that China & Japan group in fours. So the US number 12,304,567.89 would be

1230 4567.89

in China or Japan.

I thought this was interesting. But then I have been told I have weird values for "interesting".

I also liked this comment:

Travel broadens the parochial mind you never knew you had, and makes you a better programmer/designer/techie.

As a bonus it also exposes you to better beer.

from joelonsoftware, talking about software company management:

( Companies need infrastructure ) and if your programmers even spend one minute thinking about this that's one minute too many. ( ... ) That's why you have management.

It's for the kind of stuff that no company can avoid, but if you have your programmers worrying about it, well, management has failed, the same way as a 100 foot yacht has failed if the millionaire owner has to go down into the engine room and, um, build the engine.

( Some companies are too sales-focused, others are too code-focused )

Both of these companies can easily be wiped out by a company that's driven by programmers and organized to put programmers in the driver's seat, but which have an excellent abstraction that does all the hard work to convert code into products below the decks.

A programmer is most productive with a quiet private office, a great computer, unlimited beverages, an ambient temperature between 68 and 72 degrees (F), no glare on the screen, a chair that's so comfortable you don't feel it, an administrator that brings them their mail and orders manuals and books, a system administrator who makes the Internet as available as oxygen, a tester to find the bugs they just can't see, a graphic designer to make their screens beautiful, a team of marketing people to make the masses want their products, a team of sales people to make sure the masses can get these products, some patient tech support saints who help customers get the product working and help the programmers understand what problems are generating the tech support calls, and about a dozen other support and administrative functions which, in a typical company, add up to about 80% of the payroll. It is not a coincidence that the Roman army had a ratio of four servants for every soldier. This was not decadence. ( ... )

Management's primary responsibility to create the illusion that a software company can be run by writing code, because that's what programmers do. And while it would be great to have programmers who are also great at sales, graphic design, system administration, and cooking, it's unrealistic. Like teaching a pig to sing, it wastes your time and it annoys the pig.

I added the bolding in that last paragraph because I think it does a great job of explaining what needs to be done in the software business besides making great software - in fact, what needs to be done in most businesses. It's stuff that most people don't think about because, usually, no one individual does it all. But it's the stuff that can make-or-break useability, sales, operations, and employee satisfaction. As Joel goes on to say,

Microsoft does such a good job at creating this abstraction that Microsoft alumni have a notoriously hard time starting companies. They simply can't believe how much went on below decks and they have no idea how to reproduce it.

I would amend that to be most people who have worked at successful companies have no idea how much is done outside their own area...and have no idea how to reproduce it.

Full article: here.

A former coworker of mine, Raymond Chen, posed a "what would you do?" question about how Windows Vista should handle getting incomplete data back from a server. Click here to read his post ;)

I think this provides a window into day-to-day software quality decisions that most people don't usually see. In particular, the amount of interdependency that occurs when dealing with an open platform like Intel/Windows.

Note that Raymond does not name the buggy server software in question, other than to note it's not Windows. This is SOP when discussing things outside of NDA.

For the less-initiated, the information would travel from

the server -> Microsoft's network driver -> Windows OS file APIs & Windows Explorer

APIs (Application Program Interface) are how programs interact with each other. Compare to UI (User Interface).

As for what I'd do: ( I assume you've read the problem already )

"The price one pays for pursuing any profession or calling is an intimate knowledge of its ugly side."

I ran across this while looking up titles for a crossword. It's very true. I would say it's even truer for testers because we are paid and encouraged to focus on what *doesn't* work.

Also: "All roles are dangerous. The world tends to trap you in the role you play and it is always extremely hard to maintain a watchful, mocking distance between oneself as one appears to be and oneself as one actually is."

Raymond's blog entry today is on "ground rules for using function parameters". He's specifically trying to list the "unwritten" rules, aka the rules that are taken for granted. Comparing calling conventions to driving directions, he's trying to write the equivalent of "Don't drive on the lawn" and "Next right does not mean driveway unless you're told otherwise". Overall I found it entertaining (but then I find kooky software errors entertaining :)

http://blogs.msdn.com/oldnewthing/archive/2006/03/20/555511.aspx

We now have an internal blogging server at work. I'll probably mostly use it for musings on testing / QA / software dev stuff.

My first post: ( Read more... )

11 ways to boot testing success

&

5 questions to diagnose how testing is going

Yes, a C board game. Here's a partial pic of the board. More info is at http://www.c-jump.com ;)

This interview transcript illustrates some of the chicken-and-egg factors in creating systems that prevent errors. But this one bit on root cause analysis caught my eye...

I think the term ‘root cause analysis’ is an unfortunate one. It implies that there’s some real root, there’s some cause back there which set the whole thing going like a chain reaction. But of course, there’s never one cause. So I don’t really like the term, but I do like what it implies, which is to go back from the immediate sharp-end people, from the immediate things that went on, and ask questions: ‘What was it that provoked this person to do this particular unsafe act? And what were the decisions upstream from that, that left those people for example, short-handed, or with inadequate tools or equipment? What decisions were going on?’ say on the Board level, in the tension between protection and production. But it’s a finite process.
- Jim Reason, Emeritus Professor of Psychology at the University of Manchester.

This is something I've run into with RCAs. Either mgmt is looking for one simple thing to fix and doesn't want to deal with the overall system, or (in the shrinkwrap world) the documentation necessary may not exist. Le sigh.

As a bonus, from the same interview, on the Columbia accident: ( Read more... )

Suppose that you were running a QA group and the programmers made a lot of time-related bugs. Could you send the programmers out for retraining? Could you require them to adopt the new practices? Could you give bonuses to the ones who made the most progress? If you don’t have the authority to manage and train the programming staff, then you don’t have the power to assure quality. If you do have that power, your title is probably Vice-President of Product Development, not Manager of Quality Assurance. Testing groups and testing-plus-other-measurements groups are doing Quality Assistance not assurance.
- Cem Kaner in The Impossibility of Complete Testing

This is why I tried several times to change my former title from Quality Assurance Manager to Software Test Manager....

Jonathan Kohl also has good things to say about Test Automation. A sample:

"Test automation" is a valuable tool we can use in our tester's toolbox to help us do more effective testing. It does not and can not replace a human tester, particularly at the end-user level. It is a sharp tool though, and we can easily cut ourselves with it. Most test automation efforts fail because they don't take software development architecture into account, they don't plan for maintenance, and they tend to be understaffed, and are often staffed by non-programmers. [...] Anyone who claims they can do software test automation without programming is either very naive themselves, or they think you are naive and are trying to sell you something.

Kohl points out the connections between trivializing software testing, thinking that testing itself can be automated (not just some tasks), and poor results. Some of you will nod at Kohl's observation that

When we get it wrong in test automation, [the fallout] is more insidious; it may take a long time to realize a problem is there. By that time, it might be too late. Customers are quietly moving on to competitors, talented testers are frustrated and leaving your company to work for others.

If you're new to test automation, a good primer is Architectures of Test Automation (PDF) by Cem Kaner.

BTW: I know there are 6-7 pro testers / former testers who read my LJ semi-regularly. Also devs & other software folk. Are you comfortable discussing this stuff openly or should I create a filter?

I dislike detailed procedural test scripts, and not just because I hate writing them. (Maintaining them is worse!) They encourage stale / rote testing, discourage creativity, and generally suck. Nice to see I'm not the only one - Jonathan Kohl & Shrini agree. I especially liked Johnathan's example "dev script".

A script is useful to someone who is learning the product. However: Should you have inexperienced folks who don't know what's going on doing the testing?

I've had a few people mention the Software Quality Engineer Certification, or CSQE. It's the first software-testing related cert I've heard of, so I looked into it a bit.

It seems interesting to me - but as a "hm, lots of interesting stuff here" not a "yeah, that would help me do my job better". It's QA in the broad sense; project management and dev management and test management and testing and and and. I will note that many of the topics involved in the CSQE dovetails with my recent Amazon purchases.... ( books on testing, management & project management ) Offhand, I get most of these practice questions right. However, "According to Crosby", "Halstead's software science metric" and "According to IEEE standards" draw blanks. So do terms like "cyclomatic complexity", "requirements analysis methodologies", "minor nonconformance", and "defect escapes". Some I could deal with in context, mind you. I even got the "According to Crosby" one right by just picking the right answer and not worrying about what Crosby thinks... ;)

But. The fact that I'm tripping over terminology - and references to Six Sigma, ISO 9000 & IEEE - tells me this is not "my" [shrinkwrap software] world. Does that mean it's a bad thing? No. Just not sure

1. How much of it would be learning new things
2. How much of it would be learning new names for things I already know
3. How much of it would be useful

#2 & #3 in part depend on where my career goes down the road. I think the new terminology would become useful pretty quick if I ever wanted to move out of shrinkwrap....