Originally published December 02, 2021 in Credentialing Insights - The online journal for I.C.E.
Interview by Robert C. Shaw, Jr., PhD, National Board for Respiratory Care (NBRC)
After graduating from the University of Nebraska with a doctorate in quantitative, qualitative and psychometric methods, Brett Foley, PhD, has worked with programs in the education, certification and licensure areas. He currently serves as the director of professional credentialing and a senior psychometrician at Alpine Testing Solutions, and is the chair of the Nebraska Board of Engineers and Architects as a public member. Formerly, Foley served as president of the Northern Rocky Mountain Educational Research Association. His research interests include standard setting, policy considerations in testing and using visual displays to inform the test development process.
Robert C. Shaw, Jr., PhD, vice president of examinations for the National Board for Respiratory Care (NBRC) and member of the I.C.E. Publications and Editorial Committee, recently spoke with Foley on how he got into psychometrics, what being a public member of a board of directors is like and more.
RS: You have worked with programs that had missions in education, licensure and certification. What have been some memorable programs from these three areas?
BF: One of the best parts of my job is getting to work with experts in a variety of different fields and playing a small role in assisting in their important work. Early in my career, I had the opportunity to work with the Hawaii Department of Education on their state-level summative assessments for kids with severe cognitive disabilities.
More recently, I’ve shifted focus to credentialing. While there are many programs I’ve enjoyed working with, I’m especially proud of the long-term work I’ve been able to do with the National Council of Architectural Registration Boards (NCARB), the International Hearing Society (IHS) and the National Board of Examiners in Optometry (NBEO). These groups respectively produce credentialing exams for architects, hearing professionals and optometrists. In addition to doing essential work in protecting the public, these organizations also have been active research partners for Alpine, advocating for innovation, the dissemination of new practices and presenting at conferences like the I.C.E. Exchange.
RS: In what year did you earn your doctorate from the University of Nebraska and what was your first psychometric gig?
BF: I finished my doctoral work in 2010. My first psychometric gig was at the University of Nebraska at Lincoln while still in graduate school. I was lucky enough to earn an assistantship (and later a full-time job) at the Buros Center for Testing. It was a fantastic experience, giving me my first opportunity do practical, hands-on work in psychometrics. Much of my time was spent working on equating, standard setting studies and content alignment studies for state departments of education.
RS: You are a public member and current chair of the Nebraska Board of Engineers and Architects while having a background as a psychometrician. How does the public member of this board become its chair?
BF: Following a meeting, I was out at dinner with a group of architect subject matter experts who happened to be board members back in their home states. They were discussing the challenges they were facing in recruiting public members to their boards. I asked what they looked for in a public member and if someone with a testing background would be useful in such a role. They were very positive about the idea and encouraged me to pursue it. After getting home, I did a quick search and found that the Nebraska public member position would be opening in the following few months. I filled out an online application and a couple of months later got a fancy certificate in the mail from the governor.
It has been an awesome experience, and I need to take this opportunity to thank Alpine in being supportive of its employees engaging in professional service. For me, it has been more professional development than service alone; in my first six months on the board, I learned more about the non-assessment elements of the licensure (e.g., legislative concerns, disciplinary actions, education, experience) than I had in many years of psychometric work.
An especially meaningful aspect of the role is that I get the chance to celebrate the success of candidates. With all the necessary work we do in the assessment community to set up the hurdles candidates must pass on their way to a credential, it’s fulfilling and humbling to have my signature on the certificate that may hang above the desk of an architect or engineer for their entire career, or to speak to first-time licensees at a ceremony at the Nebraska state capital.
The board’s mission is “To establish requirements for education, experience, examination and enforcement for the practices of engineering and architecture to safeguard life, health and property, and to promote public welfare.” In short, protection of the public. I was able to become chair because both the state statutes and board members recognize the public member as an essential component of and full participant in the board. As such, board leadership shifts between architects, engineers and public members across their five-year appointments.
RS: I attended a session during the Council on Licensure Enforcement and Regulation (CLEAR) virtual meeting in fall 2020 in which a regulator from Canada proposed that half the members of an occupational regulation board should be members of the public to assure that the needs of the public are a priority rather than the needs of occupational incumbents. I propose you are in a unique position to think through such a scenario so what do you see?
BF: I don’t have a good answer as to what the “right” number of public members on a board should be. Clearly, I recognize the value of public members in ensuring that the interests of folks outside of the profession are represented. However, all board members I have met who are working in the profession take their role of protecting the public very seriously. Additionally, they bring essential subject matter expertise. For example, it is difficult to judge the competence of a professional in the field if you lack competence yourself. In my view, the strongest boards feature a mix of public members, professionals in the field in question and representatives from closely related professions. This helps make sure there is a diversity of views represented, there is sufficient subject matter expertise to make informed judgments on technical issues and no one constituency can ignore the concerns of the others.
RS: I saw an announcement about a CLEAR webinar you gave that proposes a different way to arrive at the passing decision within a program that assesses candidates with a test. Anticipating that Brett the psychometrician will describe what he saw during the webinar, how does Brett the public board member look at the proposition?
BF: Great question! Brett the psychometrician has spent much of his career focused on the accuracy of tests for making pass/fail decisions. Brett the public board member is much more concerned about the accuracy of testing systems for making credentialing decisions.
While these ideas are similar, the differences are subtle but important. Testing systems include all the hurdles candidates must pass to earn the credential (for example, architects must pass six different exams) as well as the retake policies for each of these exams. Each of these hurdles have properties that affect associated exam-level accuracy (e.g., reliability), and each additional assessment and retake opportunity influences the likelihood of false-positive or false-negative credentialing decisions. Estimating the accuracy of credentialing systems by taking all these factors into account was the topic of the CLEAR webinar and continues to be a topic of interest for me (both as the psychometrician and the public member).