After reading the chapter on usability testing, I am finding myself a little more forgiving of the many shortcomings that I see in software, particularly the rich, deep and complex like Microsoft Office.
For example, on the issue of reliable statistics, how could ANYONE extrapolate samples for an audience as universally ubiquitous as, in effect, the known world and possibility come close to obtaining the needed data to interpret test reliably? Even if they can we are too damned different to please us all! In Presentation Theory, we have a principle that we borrow from Marketing Theory which is “know your audience”. It is difficult to create a presentation that is gear properly to your audience even when it is known and relatively homogeneous.
How insane (revering to measuring the validity of a usability test) is any attempt to gain a true understanding of any workable reality with one of the main aspect of that measurement is “…some common sense”? And if the man rule is getting test users is that they “should be as representative as possible” of the intended users, or test task chosen to be “as representative as possible to the uses…“ aren’t these as obvious (and therefore meaningless) as saying, “the best investment is one that produces the maximum return with the least amount of risk”? I guess it isn’t meaningless…because it is the truth. (The men and women who helped give birth to the “Age of Enlightenment” must be rolling over in their graves! All except…perhaps Nietzsche.)
And the fact that all of this must be done with adherence to a budget. To get all of just the right mix of quality disparate elements (test subjects, experimenters, etc.) at a “reasonable cost…my mind is trying to absorb all of this, but it can’t. I cannot get a mental image of how this ever gets done to any level of success in anything more than the most simple of systems with the smallest and most homogeneous of audiences. I think that this cognitive overload may be the result of my long-time residence in the “Ivory Tower” think. I think have been teaching for too long and have been separate from the real world of business and commerce for a critically long period of time. Twenty years ago, this would not have freaked me out. In 1993. I understood and was able to picture the “ordered chaos” that is the marketplace. I cannot anymore…or at least not as well…and that fills me with more than a small amount of sadness and anxiety.
Finally, in reading about the “Thinking Aloud” method, and seeing it first described, I immediately though about the cognitive conflicts that “thinking” and “speaking” impose. I could be remembering this incorrectly, but I seem to remember reading that the part of the brain that “speaks” is the same as the part that “solves” problems. If I did remember this correctly, then would there not be a fundamental cognitive conflict it trying to perform both operations simultaneously? It would be the same problem that one might encounter if one was trying to compose more than a few sentences thought automated speech recognition (ASR). Stipulating the accuracy of the output for the moment, I could not, for example have “spoken” the 600+ words of this blog as “naturally” as I was able to type it. Even with perfect ASR recognition, it still would have taken me longer to speak this blog and it may not have turn out the same as it would have come from a different brain reality. I was happy to see, as I read on, that these issues were addressed.