Uncategorized | Make Your Job a Calling

Checklist for Evaluating Online Assessment Systems

September 27, 2012 · by Templeton Editor

In the old days, you had to meet with a counselor to take career assessments. The instruments came with a booklet, a score sheet, and a number 2 pencil. After taking the assessments, the counselor had to mail it off for scoring before you could see your scores and experience and interpretation of the results.

These paper-and-pencil instruments are still around, but career assessments are now ubiquitous on the web. Many of them are packaged with other assessments in career assessment systems, for which you are usually required to pay a fee to access. There are many of these in the marketplace—close to a hundred by our latest count. How can you tell the good ones from the bad ones? The following questions can help you evaluate a career assessment system you might be considering. Before making the leap, logging on, and entering your credit card number, think them through and make sure you have the info you need to make the right call.

Does the assessment system use instruments with strong support for reliability and validity? Reliability and validity evidence is the quality control criteria that psychologists and other professionals use to evaluate assessment instruments. For more information, check out our information for psychologists. A good assessment system will provide at least a summary of reliability and validity evidence for its scores, probably with a link to more detailed, highly technical information. If it does not, contact the company and ask for this information. If they are reluctant to provide it or say that it is not available, you cannot be sure that the information you’ll get from the assessments is accurate. Move on to another option.
Has the assessment system as a whole been empirically evaluated for effectiveness? Only a few assessment systems have been tested in experiments designed to investigate the effect they have on the career decision-making confidence (among other outcomes) of users. If the system you are evaluating has been tested, what were the results? If it has not been tested, how do you know it will be helpful?
Does the assessment system provide an opportunity for you to interact with a professional who is trained to interpret your scores? Most career assessment systems are self-directed, but some provide access to human interaction, even by phone or skype, or by a counselor in your area who can work with you to interpret the results provided by the system. You can benefit from navigating an assessment system on your own, but research shows that interaction with a counselor significantly improves the effectiveness of computer-based career assessment systems.
Does the assessment system provide linkages of your assessment data to job titles that are predicted to be a good fit for your profile? Many systems provide more than just scores on your attributes—they use those scores to recommend good-fitting jobs. If a system offers this, from what source does it draw its information about jobs? Some use the O*NET, the U.S. Department of Labor’s occupational information database. Others have their own proprietary database. A good assessment system should disclose whatever source they use, so you can evaluate the quality of the information.
Can the assessment system link you directly to potential employers? Some systems provide job postings. Not many link you to actual positions on the basis of your profile of scores, but this kind of “e-Harmony for jobs” function is probably the way of the future.
Is the assessment system cost-effective? Most systems are affordable, but the fees do vary. How much does the system you are evaluating cost, and how well does that cost reflect the value you receive? Will you derive proportionally greater benefit from assessment systems that are more expensive than others?
What promises does the assessment system make? Be wary of any claims that success is guaranteed, or that even hint that your results will reveal the career path you “should” pursue. Career decision-making is a complex endeavor, and although assessment systems can play an important role in your process by giving you helpful information, that information is just one piece of the puzzle. It can inform your choices, but not make them for you. Success is never guaranteed, and no system can tell you what you “should” do with your life.

Quality Control Criteria for Assessments

September 27, 2012 · by Templeton Editor

For psychologists, reliability and validity serve as quality control criteria.

Reliability

The question of reliability is a critical one to ask of all measures. Formally, reliability is “the degree to which scores are free from unsystematic error.” If scores on a test are free from this, they’ll be consistent across related measurement. So, an easy way to think about reliability is the word “consistency.”

Note the word “unsystematic” in the definition. Scores on a measure may contain error but still be reliable. For example, if a thermometer is always 5 degrees low, it will be “reliable”–meaning consistent–but it’s inaccurate. Or, consider the case in which one professor is a harsh grader and one is a lenient grader. Each of them may be reliable, but one assigns grades that are systematically too low, and another gives grades that are systematically too high.

Types of Reliability

There are four primary types of reliability that psychologists use to evaluate an assessment’s scores. They are as follows:

1. Test-retest reliability. This is a measure of stability, and it asks the question: How stable are scores on the measure or test over time? Do you get the same results (or at least very close to the same) on two separate occasions? This is computed numerically using a correlation coefficient. The thing to keep in mind with test-retest reliability is that it only makes sense if the trait being measured is supposed to be stable. Most variables that career counselors want to assess (e.g., interests, values, personality, ability) are quite stable on average, once people reach early adulthood. (Otherwise it would make little sense to measure these things and use them to inform career decisions that may have long-term implications.)

2. Alternative forms reliability. This applies only to instruments for which there are two or more separate forms that are intended to be equivalent. For example, the SAT and ACT each have multiple forms. This is a measure of equivalence, and it asks this question: Do the two different versions of this measure give the same results? Are the two versions essentially the same? To test this, a test developer would administer the two versions of your assessment to the same people, and then calculate the correlation between these two sets of scores.

3. Internal Consistency Reliability. This is a measure of consistency within the test. How consistent are scores for the items on this test? Do all the items fit together? Do they all measure the same thing? There are two main types of internal reliability: split half and coefficent alpha.

Split-half. To find the split half internal consistency reliability, you’d start by administering your measure’s items to a group of people. After everyone has taken it, you’d randomly separate the pool of items into two halves. You’d treat these halves as if they were each a separate measure of the construct. Then you’d calculate the correlation between the scores you obtain from each half of the test.
Coefficient alpha. One problem with split-half reliability is that you may get a different measure of reliability for each way you split the measure in half. You could simply separate the even-numbered items and the odd-numbered items, but you could also take the first half and the second half, and your correlation may end up being different. Coefficient alpha is a type of reliability coefficient that test developers can calculate which is the average of all possible split-half reliability coefficients. Conceptually, you can think of it as essentially the same as splitting the measure in half many different times—as many different ways as is possible—and calculating the split-half reliability coefficient for each pair of halves, then calculating the average of all those split-half reliability coefficients. In practice there is a formula that makes calculating this easy. Coefficient alpha can also be thought of as an index of the degree to which each item contributes to the total score.

4. Inter-rater reliability. Sometimes you measure things not with a paper and pencil measure that the participant completes, but by having other observers rate the participant’s behavior, such as in a behavioral assessment. Reliability is still important in this case, but it looks a little different. Now the question is whether the two (or more) raters agree with each other.

To establish evidence for inter-rater reliability, you would have two raters rate the same participants. Then you would calculate the relationship between the two raters. Once you have established evidence for inter-rater reliability, you can say confidently that the rating that occurred was consistent.

Validity

Sometimes validity is referred to as addressing the question “does the test measure what it is designed to measure?” A broader and more appropriate way to think about validity is referring to it as addressing the question “does the test accomplish its intended purpose?” or “does the test meet the claims made for it?”

Types of Validity

There are three primary types of validity that psychologists use to evaluate an assessment’s scores. (The list below has four types, but the first doesn’t really count.) They are as follows:

1. Face validity. This really isn’t validity in a scientific sense. If a measurement instrument has face validity, it just means it looks like it measures what it’s supposed to measure. If you have a measure of interests and the items look like they are tapping into a person’s interests, you have face validity. This can be good if it builds rapport with the test-taker, but it doesn’t have any real scientific value. All things being equal, it’s good if you have it, but it doesn’t mean that the measure is accomplishing its intended purpose.

2. Content validity. Content validity refers to how well an assessment covers all relevant aspects of the domain it is supposed to measure. Conceptually, you could ensure content validity if you could write items to cover absolutely every detail about your construct. For example, if a test developer wanted to measure a particular style of leadership, that developer could write every single item she could possibly think of that would be relevant to that leadership style. Then she could take a random sample of those items and include them in the scale. Unfortunately, this is not only impractical, it is impossible.

Often, content validity is assessed by expert judgment. You could assess the content validity of a measure by having an expert or experts examine the items and determine whether the items are a good representation of the entire universe of items. If the leadership style measure described above exam has low content validity, it would probably include a lot of items that aren’t relevant to that style, and it would probably leave out items that are very relevant to that style.

3. Criterion-related validity. This refers to how well an assessment correlates with performance or whatever other criterion you’re interested in. It answers the question of “do scores on the measure allow us to infer information about performance on some criterion?” There are two types of criterion validity: concurrent and predictive.

Concurrent validity. This refers to how well the test scores correlate with some criterion, when both measures are taken at the same time. For an interest inventory, for example, a good question is whether people who are, say, engineers score high on a scale designed to measure interest in engineering.
Predictive validity. This refers to how well the test scores correlate with future criteria. For example, what percentage of people will end up in a career field down the road that corresponds to high scores on scales designed to measure a person’s interest in that field?

4. Construct validity. Construct validity most directly addresses the question of “does the test measure what it’s designed to measure?” It refers to how well the test assesses the underlying construct that is theorized. To demonstrate evidence of construct validity, a test developer would show that scores on her measure would have a strong relationship with certain variables (the ones that are very similar to what is being measured) and a weak relationship with other variables (those that are conceptualized as being dissimilar to the construct being measured). There are two types: convergent and discriminant.

Convergent validity. Convergent validity is the extent to which scores on a measure are related to scores on measures of the same or similar constructs. For example, let’s say your personality test has an extraversion scale. You might expect that the more extraverted a person is, the more likely she or he is to have high levels of sociability. If there is a strong positive relationship between scores on your extraversion measure and the scores on measures of sociability, then your scale’s scores have evidence of convergent validity.
Discriminant validity. Support for discriminant validity is demonstrated by showing that an assessment does not measure something it is not intended to measure. For example, if you have an extraversion scale in your measure, you might consider also administering a measure of emotional stability. We know that extraversion and emotional stability are two different things. If you asked people to take your scale and a measure of emotional stability and you find a small correlation between scores on these two scales, you have shown that your scale measures something other than emotional stability. Note that you haven’t shown what itdoes measure, you have just shown what it does not measure.

NOTE: An instrument’s scores can be reliable but not valid. However, if an instrument’s scores are not reliable, there is no way that they can be valid.

Why does this matter? If you want to know how “good” a career assessment is, ask this question: “What is the evidence of reliability and validity?” A counselor should be able to answer this question, and an online assessment portal should provide some basic information (and links to more detailed information) to show that its scores provide good information. If no information about a particular test can be found, we’d encourage you to avoid that assessment instrument, or at the very least to view scores generated by that assessment in only the most tentative way.

Do your Career Goals Fit with Your Life Goals?

September 27, 2012 · by Templeton Editor

As we note in Chapter 4 of Make Your Job a Calling, most people agree that experiencing a positive sense of meaning—defined as “the sense made of, and significance felt regarding,the nature of one’s being and existence”[1]—is fundamental to living “the good life.” Yet how many take serious steps toward living meaningfully at work? One way to think through this question is to evaluate how well your career goals fit within the context of your life goals. Try this: for starters, think about your life as a whole. What, ultimately, is most important to you? How would you describe your life’s purpose? With answers to these questions in mind, list at least five life goals you are currently pursuing. (We recommend you write these down, either here, in a journal, or on a separate sheet.)

1.

2.

3.

4.

5.

Next, think carefully about your career for a moment—your current job situation, the kind of work you most want to do, and the steps you need to take to bridge the gap between these, if there is one. How close or far away are you from where you want to be? What role do you want your career to play within the broader context of your life? With your answers to these questions in mind, list at least five career goals you are currently pursuing:

1.

2.

3.

4.

5.

Now look closely at the goals you listed above for your life and your career. To what extent are your career goals in line with your life goals? Are you happy with your answer to this question? If not, what needs to change?

[1] M. F. Steger, Patricia Frazier, Shigehiro Oishi, and Matthew Kaler, “The Meaning in Life Questionnaire: Assessing the Presence of and Search for Meaning in Life,” Journal of Counseling Psychology 53.1 [2006]: 80–93, 81.

Career Development Striving Exercise

September 27, 2012 · by Templeton Editor

There are two parts to this exercise. The first asks you to identify five things that you are currently striving for in your career. The second asks you to rate each of those “strivings” along seven criteria. Give this a shot, then read on.

Part 1. Consider the activities you are currently engaging in to help you build a successful career. We might call these types of activities “career development strivings.” Be as honest and as objective as possible; do not simply give socially desirable strivings or strivings you think you “ought” to have. Please write down what you currently consider to be the 5 most important career development strivings in your life.

Part 2. Next, please rate each of your five career development strivings (S1-S5) based on the following questions, using the following scale:

How do I use this? This exercise is based on a study that appeared in Journal of Career Development. In that study, we asked research participants to do essentially the same thing you just did. We found that ratings on the first criterion of part 2, which assess outcome expectations, were positively associated with career decision self-confidence and intrinsic motivation. Ratings on the second criterion of part 2 measure a sense of self-efficacy or confidence, and were related to intrinsic motivation and meaning in life. Ratings on the third criterion measure a sense of calling, and were associated with intrinsic commitment, meaning in life, and religious commitment.

The other ratings were not included in that study, but have proven helpful to our career counseling clients. The fourth set of ratings target motivations for meaning-making, the fifth target the level of social support you perceive, and the sixth target the sense of vigor and persistence you feel as you pursue each striving. In all six cases, if you have a lot of high ratings (e.g., mostly 4s and 5s), you are on track toward experiencing beneficial outcomes such as those we investigated in the Journal of Career Development study. If you have a lot of 1s and 2s, you probably are struggling with those strivings. This brings us to the seventh set of ratings, where you identified the extent to which you feel you need help with each striving. If those ratings are high, seek support! In Make Your Job a Calling, we recommend actively engaging the career development process with the help of friends, family, and trusted mentors in your life, as well as a trained career counselor. We also recommend both formal and informal assessments and self-help exercises. Whatever combination of resources and supports you use, take the reigns and engage the process actively. Doing so takes effort and hard work, but trust us, it will pay off.

[1] Dik, B. J., Sargent, A. M., & Steger, M. F. (2008). Career development strivings: Assessing goals and motivation in career decision-making and planning. Journal of Career Development, 35, 23-41.

What is jobZology?

September 27, 2012 · by Templeton Editor

jobZology^TMis a career assessment system currently being developed by Career Analytics Network, Inc., a start-up company in Fort Collins, Colorado, founded in partnership with Colorado State University. jobZology^TMis designed to provide high quality and low cost (that is, free!) assessment information to job seekers, along with resources to help navigate the career decision-making and development process. jobZology^TMalso includes a decision engine that directly connects job-seekers with employers on the basis of psychological fit, thus providing a more streamlined pathway for people to live out their callings, and for employers to quickly find people who fit their open positions and organizational culture well. Bryan Dik, coauthor of Make Your Job a Calling, is a cofounder and current chief science officer for the company. For more information or to register for updates, go to www.jobzology.com.

You at Your Best

September 27, 2012 · by Templeton Editor

In Chapter 4 of Make Your Job a Calling, we note that one informal strategy for assessing your strengths is to think carefully about a recent situation when you were at your best. Give it a shot. Start by identifying the situation, a specific event within the last few weeks in which you felt you were clearly at your best. Got it? Now, replay it in your mind a few times, focusing carefully on the details of the moment. In a journal or on another sheet (because you will need space for this), write out answers to these questions:

Using a step-by-step account, how did the events of this situation unfold?
What did you do well?
What was the outcome?
Thinking back on it, what specific personal strengths did you show in this situation? List as many as you can.
Circle or highlight the top five. In which other situations have you observed these strengths?

Once you have those top five strengths, try this experiment: Make a conscious, deliberate effort to use these strengths more often, and in new ways, in your job (or in other areas of your life, if you are not currently employed) every day for the next week. Over the course of that week, as you do this, what do you notice or experience? In what ways has using these strengths changed the way you feel about your work?

Martin Seligman and colleagues have shown that people instructed to write about a “you at your best” experience, reflect on their personal strengths illustrated in the story, and then review the story once a day for a week to further reflect on their strengths were significantly happier and less depressed compared to people in a control group at the end of that week.[1] However, these effects didn’t last into the weeks and months that followed. Why not? One explanation is that merely reflecting on your strengths is not enough; you have to use them. Another finding from the same study supported that interpretation: Individuals who were given feedback on their top strengths, and who were then instructed to use one of their top strengths in a new and different way each day for a week, were significantly happier and less depressed not just at the end of that week but even six months later! Of course, these participants were instructed to use their strengths generally, not only at work, and happiness and depression are not the same thing as meaning. However, another recent study demonstrated that when people use their strengths at work, they are in fact more likely to experience their work as meaningful.[2] To summarize the obvious conclusion: Don’t stop at merely identifying your strengths, use them!

[1] Martin E. P. Seligman, Tracy A. Steen, Nansook Park, and Christopher Peterson, “Positive Psychology Progress: Empirical Validation of Interventions,” American Psychologist 60.5 (2005): 410–21.

[2] Hadassa Littman-Ovadia and Michael F. Steger, “Character Strengths and Well-Being among Volunteers and Employees: Towards an Integrative Model,” Journal of Positive Psychology 5.6 (2010): 419–30.