Minds at Work:

Does Technology Truly Globalize Assessments Systems?

Vernon Bryce, Kenexa
 
HOST:  Welcome to Minds at Work, where our job is to help your mind become inspired by your work. I’m your host, Drew Hoffmeyer. In today’s episode, entitled, “Does Technology Truly Globalize Assessment Systems?”  Vernon Bryce, Master of Science, Managing Partner of Kenexa’s European operations, identifies challenges in assessment as technology advances. Although technology is providing a platform to globalize, Bryce identifies linguistic, economic and cultural issues that may interfere with developing a truly global assessment system and reviews the standards created by the International Test Commission (ITC) for global assessment systems.
MR. BRYCE: Today, we receive and transmit more information faster, cheaper, invisibly and internationally than ever. It’s getting cheaper, and even more inventive, day-by-day. The world’s share and access to information person-by-person, individual-by-individual is accelerating fast.
Inevitably, experts predict that this will globalize so many business opportunities and processes. This includes, for example, global assessment processes. And here’s my question—will it truly globalize assessment processes?
Let’s look at some of these opportunities and their problems. First of all—global linguistics. Despite technological advantages, interestingly by the year 2010, maybe 2012, it’s said that only three principle languages will dominate globally. Now this is really important—number one Cantonese; number two Hindi; and number three English, in that order. So why don’t we just think really basically here about technology? As always, fortunately, life is not as simple as just translating.
One of the challenges that will be facing us is which form of Cantonese or Hindi or English should be used? It isn’t just a question ofone, two or three—there are several well-established forms within each language. And secondly, who will decide which language best fits the occasion? So despite the technology, there’s obviously going to be some linguistic challenges.
Actually there are several more highly-significant, historically steeped languages where millions will challenge, and rightfully too, the dominance of these so-called principle languages. There are several millions of Arabic speakers—at last count over 300 million; French, German, Japanese and Spanish speakers, for instance. Furthermore, taking into account other wide subcultures on this planet, there are many hundreds, if not thousands, of national and local languages and their variations, dialects and even accents. So how does this help assessment experts and what do they do practically about this?
The second question to ask is global economics. Before we develop answers to these very promising technological innovations, and the linguistics, there’s an even bigger economic picture. There’s obviously a massive demand for measures of people attributes as they move around, within and between organizations, country by country, internationally—and this has led to many complex and costly assessment regimes and techniques. So the opportunity is there—the question is the scalability of it. We may find that technology is actually racing ahead of psychology.
Other global economic trends, for instance, will influence the pan-national organizations, of course. There’s an increasing dependence on scarce international labor, the so-called war for talent. Maybe we should be focusing [assessment] on the niche talent streams [say job families].
So just imagine the attractive premiums and benefits that will be placed on simple, yet valid, reliable and also just systems. We would soon see these economic forces changing, perhaps forcing ranges of fast, transferrable assessment solutions.
So we have the linguistics, we have the economics—the economics is an obvious opportunity, the linguistics may be the barrier.
Third—global assessment. So, are there any rules of the road already in place for would-be harvesters of this great global technology promise to help meet these linguistic challenges and yet respond to these oceans of economic opportunity? Well fortunately, we’re not starting cold, despite these fast advances in technology. For many years, companies have been investing in international, cross-cultural assessment. And at this moment, there are hundreds, if not thousands, of very carefully designed and planned assessment centers. The problem is, of course, these are very manual and not necessarily automated.
There are also a number of multi-rater surveys, which are pretty much automated opinion surveys. And also the standard, traditional psychometric assessment applications, taking place earnestly across the globe. Extent to which technology involved is our challenge. Every designer is fully convinced that their tools are fair, and accurate and reliable. Every designer is aware from their hard experience that there are several elephant traps along the way.
So this is where we need to blend, I think, our assessment experience with the technology advantage. Simplified assessment systems, particularly technology-driven, would obviously attract vast economies-of-scale. The stakes are really high and the opportunities vast for the masters of this new technologically based, assessment universe. Those can fully exploit the high speed, the low cost, provided we can get the cross-cultural, pan-national standards and results now. Rewards are really high for those who can get it right. So how can we do this?
Another factor that we need to take into account is cross-cultural aspects before we move into some rules of the road. So it isn’t just about linguistics, it isn’t just about economics, there’s also some cultural norms that we should take into account. For example, highly rated managers in Thailand may have very different answers to their neighbors in India. This is what’s known as the criterion question in psychology—and we have to get the criteria right.
We may find that despite a preferred answer in one environment, we actually get a lot of the non-preferred answer because the culture dominates. There are certain taboos in certain cultures, for example, regarding for instance, what you believe in or whether you should state this openly or make this a quiet assumption.
So where the very nature of the test encourages us to actually be very open about our beliefs, and yet cross-culturally, we might find certain cultures are not very open with trusting their responses to people who are not in their culture. For example, the honorable thing to do in some places is to protect a family member, even when they’ve done wrong.
In some cultures, the words ‘no problem’ means “let’s give it a go”—even though it’s not been done before; even the word ‘yes’ might mean “I understand” – e.g. in Japanese modern culture. The word ‘now’ might actually mean mañana (or tomorrow) in some cultures; whereas the term ‘now now’ (heard in Africa) is the equivalent in western cultures of “immediately”. Closer to home, we may find that the word ‘okay’ simply means I understand or I hear you. To others,okay is a yes.
Now all these cross-cultural factors affect item design. So now we’ve got culture, we’ve got linguistics, which language to choose, and we’ve also got the whole issue of fairness—let alone international credibility with international clients. Even when language issues are known, it may not be safe to declare your online tool is open for business. Cultures will have an immense difference.
So is it time to call in some cavalry to rescue us from these problems—the culture, the language, despite the economic opportunity? Well, as we travel through this domain, there are some international guidelines now agreed by many people have not heard of, the International Test Commission. From at least 1999, a number of test standards for international, cross-cultural assessments] need to be applied. So despite the technology gains, here’s some very well chosen, aligned opportunities which we should really satisfy. In fact, they are standards, and there are a number of standards.
Those who may Google the ITC (the International Test Commission) will notice, it’s a non-stock corporation incorporated in the U.S., but there are a lot of advisors from the U.S. to Africa to Scandinavia, to England, and across the Americas, across to Australasia. Now, there are a number of points that the ITC makes—whatever your process, there are around nine basic rules of the road.
First of all, utility. Whatever you do, there must be a “reasoned justification for the assessment based on job needs and job analysis.” There must be correlations between the scores, the inputs if you like, and the behavioral outcomes. There has to be this level of utility and they describe it clearly as a “full, simultaneous use above the collateral sources of information.”  So, in a nutshell, there has to be a reason justification, combining the scores on the test and the behavioral outcomes, often known as utility.
The second one they describe as soundness—representativeness of assessment content, norm groups, difficulty levels, accuracy, reliability, validity, freedom from bias, also the social acceptability, practicality of the test and responses to questions asked. So the basic soundness of the tool has to satisfy this collection of standards as well. These two, just utility and soundness, offer us a great basis on which to start globalizing through this massive technology advantage.
The third is fairness, the lack of bias, meaningfulness for all groups, despite gender, culture, education, ethnicity, age—all the demographics you can think of. There are standards, obviously, across all these demographic groups to ensure that local standards, local patterns, local conventions and habits apply within this rigorous methodology. Disability is another issue, and I guess this will be a global issue. How do we manage for disability? The very nature of collecting the evidence may be affected by certain disabled communities.
The fourth is preparation. As all test inventors, or test developers, test implementers will know, you need to apply best advice, linguistically and dialectically, on the prior application of the test. These responsibilities include the principle of consent, the quality of materials and the quality of the people that administer it.
So, those are four areas to start—there’re just another five.
Administration, as labeled in the ITC, the rapport with the test taker, the context, obviously simple things like freedom from distraction to maximize the opportunity for someone to succeed and to make that opportunity consistent. Primary language is an interesting one, and the basic advice is to use mother tongue wherever possible. This is very difficult and quite challenging, where people may have a range of cultures in their background, or in fact, the authenticity of their language may not be challenged. Authentication, primary language, dealing with candidate’s difficulties, or distress in administration is another factor. For example, it is reasonable to ask people after a test administration, “how did they find the assessment?” because context may make a difference. So the administration factors are very important to get right.
Scoring is the sixth one, standards of scoring, proof of analysis, proof of scales and their validity. And there needs to be appropriate records of calculation to ensure that fairness, accuracy and reliability and fairness is actually taking place. So there’s a whole set of scoring disciplines that need to take place. Now these six areas are not often well known to people who develop tests using technology, I guess because it looks very, very straightforward. But following these nine rules really does help.
On to the last three, interpretation. Despite great preparation, despite great design and soundness of the scales, then interpretation of the data collected needs to make sense both theoretically, conceptually, practically, scale-by-scale. People need to understand the statistics that are being gathered, the demographical aspects, and also the avoidance of socio-stereotyping or cultural stereotyping. Another factor, which is listed by ITC, is the role of prior experience if you like educated candidates. That also needs to be collected prior to the event. So there’s a whole issue here about collecting some simple biographic or demographic evidence before someone captures the information in a test administration.
The last couple of areas—communications. The legitimate use of the test. What is legitimate? What isn’t legitimate? For example, in many cultures, it’s held that in communicating the use of tests, it should be made very clear that performance evidence should be used on-the-job, at work, rather than through these tests in order to decide whether someone should stay or leave a particular job. A lot of the conventions in its natural assessment is that the test is there for a purpose, often for job-fit prior to a job being undertaken rather than saying, does this person continue to fit? And there are various on-the-job tests that can be used.
Finally, number nine is review. There should be a continuous tracking against outcome criteria so that the test continues to do what it was designed to do, continues to do what it “says on the tin.” There needs to be relevant review of populations, criteria and changes, revalidation and updating of the tests.
So whilst it may sound complicated, these nine areas are insisted upon by the ITC. And organizations will be held to accountability against these standards. As I said before, whilst technology is running ahead, giving us fantastic opportunities in assessment design, these nine underlying test principles must still apply.
In conclusion, there are outstanding global assessment opportunities technologically, economically and even psychologically. Linking good practical business process, good practical science and technology—using these standards to glue it all together is a very rewarding purpose, yet still full of surprises and trapdoors. Internationally, there’s a lot to learn from each other. And as an absolute minimum platform, I would strongly advise people and companies to consider how they will apply the ITC standards.
HOST: We hope you enjoyed hearing today’s episode, and thank you for joining us at Minds at Work, where we offer you insight into becoming better employees, parents, partners, friends and neighbors. Visit our website at Kenexapods.com to find the original transcript of this episode or just to drop us a line. That’s KENEXAPODS.com. This episode was brought to you by Kenexa, a leader in building the world’s greatest work forces and serving humanity every day.