Tools of the Trade: Rating Measures
The ‘Tools of the Trade’ series offers information on many of the tools commonly used in assessments at Tide Pools. We wanted to provide information about what these tools are and why we find them useful in assessments. Of course, this is for informational purposes only and does not replace consultation with a licensed clinician. Our series begins with Rating Measures.
How they work
Rating measures are questionnaires assessors use to gather observations about a patient. They ask the person completing the form (the ‘rater’) to note how often they observe someone displaying a certain behavior - like calling out in class, forgetting daily materials, or becoming distracted by ambient noise. The ratings usually use words like ‘often’, ‘rarely’, and ‘always’ and each of these is assigned a score. Scores are then compared to a representative sample of people the patient’s age. This helps us to see if someone is presenting with more concerns than we would typically expect.
Broad band rating measures look at a person’s overall functioning and screen for common diagnoses and concerns.
Narrow band measures are more specific, meaning they look at particular diagnoses or specific types of functioning. These can include executive functioning, social responsiveness, sensory processing, or specific diagnoses like ADHD, depression, and anxiety.
Assessors often use broad band measures to screen for common challenges and diagnoses. Narrow band measures help to gather more specific information about a patient’s functioning in a specific area. Together, these measures offer an expedient way to obtain a snapshot of how a person, and important people in their life, see them functioning out in the world.
Advantages
Rating measures may appear like a blunt, one-size-fits-all tool, but, in the right hands, they have many nuanced advantages. They involve key stakeholders in the assessment, they are an efficient way to gather diagnostic information, they provide data to focus the scope of an assessment, and they offer data about real experiences in everyday life.
Involving Key Stakeholders
Rating measures involve several key stakeholders in the assessment. Allowing important individuals to share their perspective enriches our findings and builds trust in the final assessment. We typically send rating measures to multiple individuals. For children, this includes parents / caregivers, teachers, and oftentimes the child themselves. For adults, this may include the adult, a partner / spouse / roommate, and sometimes other family members.
Improving Efficiency
Many diagnoses require that an individual present with significant symptoms across a period of time and across different settings (like home and school). Rating measures allow us to quickly and efficiently gather information about how a patient presents in different settings.
Facilitating Dialogue
Most people see the patient differently. These differences in viewpoint show up as differences in scores. When differences arise, we work to better understand what this difference might tell us. For example, a child whose behavior is a concern at home, but not at school, may benefit from the structure of the school day. Or they may find the sensory environment at school exhausting and overwhelming, leading them to fall apart when they come home, where it is safer to do so. Assessors look at data from multiple sources to understand these differences. So these differences have value - they help us know where to look deeper and also where dialogue and collaboration might be most needed and impactful.
Refining the Assessment Plan
Rating measures can be administered online and do not require a psychologist to observe the person completing them. They can be completed while a patient is on a waitlist for assessment or in the initial stages of an assessment. Along with other sources of data (for example, interviews and reviewing important records), rating measures can support a data-informed plan for the assessment. This way, the assessor can focus the assessment on what is most useful and also identify less obvious concerns that need attention. When done well, keeps the scope of the assessment focused, this reduces the need for additional assessments down the line.
Validity
Validity refers to how closely assessment results represent what is really going on. Thoughtful use of rating measures can greatly improve the validity of an assessment- building confidence in its conclusions, and clearing the path for hopeful next steps.
In-office assessment measures have several advantages, but their greatest disadvantage is they are completed in a quiet, one-on-one environment. The open office or the typical classroom look much different. Rating measures help us to close this gap by bringing in observations of a person’s actual functioning in the world - observations from the patient and the people best positioned to notice their strengths and challenges.
For example, we may ask a child to complete rote, clerical, tasks for a few minutes at a time to assess their attention and focus. This may reveal a core challenge with attention and information processing. Or it may not. A school day is much longer than a few minutes and the demands on attention are much more complex than in a psychologist’s office. Gathering information about a child’s functioning across days or weeks at a time from their parents and teacher may be the key to better understanding a child’s actual challenges in the world.
Let’s use attention as an example. Computerized tests of attention typically ask a person to focus on a screen for several minutes and push a button or trigger when key stimuli are present. Sounds great, right? Well, what we find is that having parents and teachers complete rating measures of attention and executive functioning tends to lead to more accurate diagnosis. Computerized tests of attention measure attention in one context - but attention differences (say for folks with ADHD) are heavily influenced by context. We get better information when we ask people who see the patient, day in and day out, in their everyday context.
Disadvantages
While rating measures have a lot to offer - in our practice, we feel they are essential - they do have limitations. I list a few of them below and then close this post with some suggestions for how to make the most of this part of your assessment process.
Risky When Used Alone
Rating measures are not as reliable as a thermometer. A thermometer can consistently produce the same measurement - if it is 75 degrees outside, a thermometer will keep reading 75 degrees. Rating measures, like most psychological instruments, come with a bit of noise and inconsistency. They have a larger margin of error. Sometimes they say someone has ADHD or Autism when that is likely untrue. Sometimes they say a person does not show enough symptoms for ADHD and Autism when a closer assessment shows that they do.
This is just one of the many ways that scores on rating measures can be misleading. Because they are presented as quantitative data, often with fancy bar charts that carry diagnostic labels, they can give the impression that a significant score says all we need to know. Unfortunately, in some settings, rating measures are used that way. The companies that publish rating measures make clear they should not be used alone. This is because psychologists understand that they have draw backs. They are incredibly helpful, but they are only at their best when we integrate them with other sources of information.
Rating measures are also susceptible to misinterpretation. We provide scores from all rating measures we use to allow for transparency and to facilitate review of our report by collaborating professionals. The scores are often attached to familiar names like diagnoses (e.g. ADHD) or experiences (e.g. mania). This can lead to misunderstandings - like assuming a child is manic if their mania score is elevated, when they, in fact, are not. This is why it is so important to incorporate additional sources of information. Rating measures are helpful, but they are also highly sensitive - they quickly point to potential concerns. This is valuable - it helps us to make sure we are not missing something but can be confusing if one looks at scores in isolation.
Missing People
Rating measures are often the first step of the assessment process and patients may complete them before they really have a chance to get to know their assessor. It is understandable for patients and their families to feel concerned that the rating measure will inaccurately represent a person - that it will ‘miss’ them. Psychologists guard against this by holding rating measures lightly and only drawing conclusions that are supported by multiple sources of evidence. It is also important to know that rating measures are not a statement of who the patient actually is, but a collection of data about how people view their behavior.
There is another way that rating measures can ‘miss’ people. For a rating measure (or any assessment tool) to be effective, it has to include many different types of people and many different types of clinical presentations. Many clinical diagnoses are complex and can present in a variety ways. Gender, race, socioeconomic status, language profile, and other factors can influence the presentation of a diagnosis. If they are not represented in a rating measures norms, the measure will produce less valid data. Psychologists work to track these meaningful differences, select appropriate measures for the individual patient and, again - seek multiple sources of evidence.
They Take Time
Rating measures are time consuming and some of the most important people we ask to complete them (parents and teachers) already have many demands on their time. We work to be selective when choosing rating measures and, in complex cases, completing them is necessary. Worse yet, different measures often have overlapping questions. It is important to know that different measures focus on different domains - even if some questions are repetitive, they are a necessary part of the process. To get the data we need - some repetition is inevitable.
We are fortunate that many rating measures can now be sent by e-mail and completed online. This also means a computer will reliably score them - freeing up the psychologist to focus on interpretation. The test publishers have different websites with different quirks. Completing measures often means tracking different e-mails and links. We work to make this accessible by compiling these tasks into one list and checking in to support their completion.
Insensitive Language
Rating measures often use clinical language. For some patients, this may feel awkward, even insensitive. We see improvements in this area over time, but there is more work to do. Additionally, some rating measures require assigning binary gender (M/F) to the patient’s profile. I have worked with colleagues and test publishing companies to address this issue. The good news is, we have seen some publishers introduce more inclusive gender options. The bad news is not all have. This may, understandably, raise concern for many patients. Keep in mind that we see this concern and have developed ways to use the tool in alignment with a patient’s identity - even if the test publishers’ language is not.
Tips for Patients and Families
Take Your Time - Try to minimize stress and distractions when completing these measures. It will lead to better data. You may be able to save your progress, take a much-needed break, and come back to them later.
Complete Your Own Measure - Although it is tempting to discuss the questions with other important people, it is best not to. We want to learn how you are seeing things.
Answer Every Question - Do your best to answer each question to the best of your knowledge. If too many are blank, we cannot produce the necessary data.
Feel Free to Take Notes - It is best not to agonize about the difference between ‘Sometimes True’ and ‘Mostly True’ answer choices. No single question determines the outcome, so it’s ok to give your best guess. It is also ok to take note of a confusing question, and share your thoughts on it when you meet your assessor.
Share Privacy Concerns - Your psychologist may ask for your permission to send rating measures to parents, family members, spouses, teachers, or co-workers. If you have concerns about involving one or more people in the process, please discuss it. We may be able to offer information that makes the process more comfortable or modify the process.
