1-31-25 YES!: Bias in Assessment with Katherine Gielissen, MD, MHS

Transcript

• 00:00Oh, do we have the
• 00:01slide of
• 00:02the goals,
• 00:06Linda?
• 00:09The clinician educator milestones?
• 00:11Yeah. Yep. They'll be they're
• 00:13in here. We could do
• 00:13you wanna go to that?
• 00:15That'd be great. K.
• 00:16There you go.
• 00:29Andreas, I think you could
• 00:30you could introduce
• 00:31the session.
• 00:33Great.
• 00:34Well, welcome, everyone.
• 00:36Welcome to the this, latest
• 00:38session of YES,
• 00:42which stands for Yale Medical
• 00:44Education Series. And
• 00:46today,
• 00:49for those of you who
• 00:50have joined us in the
• 00:51last,
• 00:52I don't know, half dozen
• 00:53or so sessions,
• 00:55you will know that we
• 00:56have made a very,
• 00:59explicit,
• 01:00effort intentional effort
• 01:01to link every one of
• 01:03these talks to
• 01:05goals for clinician educators.
• 01:08The milestones that,
• 01:10ACGME
• 01:11has put forth.
• 01:12And today, the two such
• 01:14goals that, come to mind
• 01:16are recognition and mitigation of
• 01:17bias on the learning environment.
• 01:19In fact, we're gonna be
• 01:20talking all about bias in
• 01:21a second,
• 01:23and specifically how it, relates,
• 01:26to learner assessment.
• 01:28Our
• 01:29speaker,
• 01:30who, Dana will introduce,
• 01:32comes all the way through
• 01:33the magic of Zoom from
• 01:35the Republic of Georgia, and
• 01:36we're delighted that she's boomeranged
• 01:38back,
• 01:40through such magic to, her
• 01:42home here in New Haven.
• 01:44So, Dana.
• 01:46Yeah. It's my great pleasure
• 01:48to
• 01:49rewelcome back Katie Gillison,
• 01:51who's, assistant professor at Emory,
• 01:54currently, but was here for
• 01:55a number of years, including
• 01:57doing her med peds training
• 01:58here. She got,
• 01:59did a med ed
• 02:01fellowship, with Donna Windisch and
• 02:02got her master's in med
• 02:04ed here,
• 02:05and and held a number
• 02:07of leadership positions here, but
• 02:09then got swept off to
• 02:10Emory where they recruited her
• 02:11to be the inaugural
• 02:13program director for their new
• 02:14med peds program. So we
• 02:16wish her much luck on
• 02:17the upcoming
• 02:18match, but,
• 02:19heard this,
• 02:21session
• 02:22before, and you're in for
• 02:23a real treat talking about
• 02:25bias and assessment. And and,
• 02:26Katie,
• 02:28take it away. And for
• 02:29those who are just joining
• 02:31the CME codes on the
• 02:32next slides, we'll hover over
• 02:34that for a minute.
• 02:35And,
• 02:36we will be monitoring the
• 02:38chat.
• 02:40But I think, Katie, if
• 02:41you have a question
• 02:42that is pressing
• 02:44and it doesn't look like
• 02:44we're noticing the chat, please
• 02:46just unmute and go ahead
• 02:47and ask, and we'll also
• 02:48have some time for questions
• 02:50at the end.
• 02:51So we'll put that happy
• 02:53to get interrupted with any
• 02:54questions that come up.
• 02:56And the CME codes on
• 02:57this slide. So if anyone
• 02:59wants to enter that, but
• 03:00I'm sure Linda will be
• 03:01popping that in the chat
• 03:02as well for people to
• 03:03enter in.
• 03:06Yeah. So I'm really excited.
• 03:07Thank you for the generous
• 03:08introduction,
• 03:09Dana and Andres. I appreciate
• 03:11it. It's always such a
• 03:12pleasure to,
• 03:13come back to Yale even
• 03:14if it's only virtually,
• 03:16to,
• 03:17share some
• 03:18of my knowledge with, my
• 03:19former colleagues.
• 03:21And I'm very excited to
• 03:23talk about this topic in
• 03:24particular because it's something that
• 03:26I think we all try
• 03:27to think about as we're
• 03:28thinking about doing, high quality
• 03:30assessments.
• 03:31We're gonna be doing an
• 03:33activity a little bit later
• 03:34in the session today
• 03:36in which I'm gonna ask
• 03:37you to individually reflect on
• 03:39an assessment that you have
• 03:40performed.
• 03:42So I would ask that,
• 03:44if you have a little
• 03:45time or you have access
• 03:46to a computer,
• 03:47to pull up a recent
• 03:48evaluation
• 03:49that you might have completed
• 03:51on a trainee. That could
• 03:52be a medical student, a
• 03:53resident.
• 03:54If you don't do evaluations
• 03:56regularly, maybe pulling up a
• 03:58letter of recommendation that you
• 03:59might have written for a
• 04:01trainee or a colleague.
• 04:03And we're gonna be just
• 04:04using this as a a
• 04:05part of a reflection exercise
• 04:06that we're gonna do later
• 04:07on.
• 04:08Ideally, I'd love you to
• 04:09pick something that has a
• 04:10lot of words in it,
• 04:11and has some narrative in
• 04:13it.
• 04:14And I like to start
• 04:15the session with a little
• 04:17bit of a riddle, and
• 04:18I want you to think
• 04:19about, what your knee jerk
• 04:20reaction is to this riddle
• 04:22as you hear it.
• 04:24A father and son are
• 04:25in a horrible car crash
• 04:27that kills the father.
• 04:29The son is rushed to
• 04:30the hospital.
• 04:31Just as he's about to
• 04:32go under the knife, the
• 04:33surgeon says, I can't operate.
• 04:35This boy is my son.
• 04:37How could that be?
• 04:40Anyone wanna put their thoughts
• 04:41in the chat?
• 04:47Thank you, Sarah.
• 04:50Okay. Great. Every everyone's seen
• 04:52this riddle before. So,
• 04:54but they actually used this
• 04:56riddle with some undergraduate students
• 04:58a number of years ago.
• 05:00And there were all kinds
• 05:01of guesses like this person
• 05:03has two dads potentially.
• 05:05The father was actually a
• 05:06priest,
• 05:07was another example.
• 05:09So sometimes our knee jerk
• 05:11reactions are are presumptions
• 05:13about,
• 05:14gender roles, in medicine can
• 05:16actually impact the way that
• 05:17we interpret information.
• 05:19And that's what we're gonna
• 05:20be talking about today. Our,
• 05:22inclinations, our presumptions, and our
• 05:24biases, and how that can
• 05:25actually affect the information that
• 05:27we're providing on assessment,
• 05:29for our trainees that could
• 05:31can actually impact them,
• 05:33in the real world and
• 05:34long term. And we're gonna
• 05:35talk through some examples of,
• 05:37that in the literature.
• 05:39So the goal of the
• 05:40session is to understand the
• 05:42impact of cognitive biases on
• 05:44learners and systems,
• 05:46to identify types of bias
• 05:48in assessment and approaches to
• 05:49mitigate them,
• 05:51and to reflect on our
• 05:52own practice of assessing learners
• 05:54and really think deeply about
• 05:56the words that we're using
• 05:57when we're assessing learners
• 05:59and the, biases that we
• 06:01all bring to the table
• 06:02when we're performing assessments.
• 06:05So,
• 06:06what is unconscious bias?
• 06:09It is our natural people
• 06:10preferences.
• 06:12We as humans,
• 06:13having to interpret this world
• 06:15around us and all the
• 06:17complexities of that world,
• 06:19our brain kind of sifts
• 06:21through information,
• 06:22and our brain is hardwired
• 06:24to look look for patterns.
• 06:26This is something that evolutionarily
• 06:27we have learned to do
• 06:29as humans.
• 06:30And, those, pattern recognitions
• 06:33actually,
• 06:34bias us towards certain affinities
• 06:36for certain groups
• 06:37or, likeness to us or
• 06:40similar backgrounds to us.
• 06:42And, unfortunately,
• 06:43a lot of this,
• 06:45happens below our level of
• 06:46awareness.
• 06:47So even though our intentions
• 06:48are one way, our brains
• 06:50are sort of hardwired to
• 06:52sift through and look for
• 06:53patterns that our brains are
• 06:54comfortable with.
• 06:56And this can, in fact,
• 06:57impact the way that we
• 06:58perceive the world around us.
• 07:00It can impact the way
• 07:02that we perceive our learners'
• 07:03knowledge,
• 07:04their ability level, our expectations
• 07:07for them,
• 07:08their readiness for independent practice,
• 07:10a lot of things that
• 07:11actually impact our learners in
• 07:13very real and meaningful ways
• 07:15in the way that we
• 07:15interpret how they're performing in
• 07:17a workspace.
• 07:20And I think this can
• 07:21be really uncomfortable for us
• 07:22because as physicians, we work
• 07:24really hard to be unbiased.
• 07:25You know, we've been trained
• 07:27to, take care of our
• 07:28patients in the best way
• 07:30possible no matter what their
• 07:31background is. And so sometimes
• 07:33it can be hard for
• 07:34us to, internally accept that
• 07:37the way that we see
• 07:38the world can be biased.
• 07:40So I think a lot
• 07:41of us believe that we're
• 07:42fair, unbiased.
• 07:44We treat all trainees the
• 07:45same. We we try to
• 07:46have our highest aspirations.
• 07:49And it can be really
• 07:51hard too because our brain
• 07:52sort of defends itself from
• 07:53this uncomfortable truth about ourselves
• 07:55as humans that,
• 07:57you know,
• 07:58that belief that we might
• 07:59be biased is in conflict
• 08:01with our intentions,
• 08:03oftentimes in the learning space.
• 08:06And, you know, this is
• 08:08part of our rational
• 08:09ordering of the world. It's
• 08:10the way that we put
• 08:11the world together,
• 08:13as as clinicians and as
• 08:14humans. And so it can
• 08:16be really uncomfortable
• 08:17to accept that,
• 08:18all of us have our
• 08:19own biases based on our
• 08:21own backgrounds.
• 08:24And it's sort of an
• 08:25adaptive behavior. You know? Like,
• 08:27we have to process
• 08:28a bunch of information, especially
• 08:30as physicians
• 08:31who are practicing,
• 08:33clinicians.
• 08:34We're we're often sorting through
• 08:35and sifting through a lot
• 08:36of information
• 08:38simultaneously.
• 08:39And these, automatic,
• 08:42sort of,
• 08:43categorization and pattern recognitions of
• 08:44our brain help us to
• 08:46sift through that information.
• 08:48But it also
• 08:50puts us at risk for,
• 08:52these biases that we're concerned
• 08:54about. So for those of
• 08:55you who, have learned about
• 08:56system one and system two,
• 08:58clinical reasoning, you know, when
• 09:00we're making rapid decisions
• 09:02about, clinical things in the
• 09:04workplace
• 09:05or we're making rapid decisions
• 09:06on whether or not to
• 09:07trust a trainee to do
• 09:08x y z in that
• 09:10workplace,
• 09:10sometimes it can cause us
• 09:12to have a little bit
• 09:13of an intellectual shorthand,
• 09:15which can predispose us to
• 09:16these biases.
• 09:20There's a lot of known
• 09:22biases in the workplace and
• 09:23in the learnings environment that
• 09:25we have with our trainees,
• 09:26both, medical students and residents.
• 09:29So, that can be affinity,
• 09:31you know, feeling connected to
• 09:32a certain person. I
• 09:35I am, the first to
• 09:36admit I'm very, biased towards,
• 09:39medical students who are interested
• 09:40in med peds. I often
• 09:42feel an immediate affinity towards
• 09:44them. I'll I'll feel more
• 09:45biased to spend more time
• 09:47with them, etcetera, etcetera.
• 09:49Our perception biases,
• 09:51halo effects are really, predominant
• 09:53in our, learning environments where
• 09:56we tend to think the
• 09:57best of people and, don't
• 09:58necessarily look at things that
• 09:59they need to do to
• 10:00grow.
• 10:02Confirmation biases, so making an
• 10:03immediate assumption about a person
• 10:05and then looking for information
• 10:07that just confirms those assumptions.
• 10:09So these are just some
• 10:10examples of many different types
• 10:12of biases in the workspace.
• 10:18And it can impact a
• 10:19lot of of training assessment.
• 10:21So,
• 10:23our learners are very aware
• 10:24that these things happen. I
• 10:26feel like the younger generation
• 10:27of learners in particular
• 10:29are very, savvy and clued
• 10:31into the ways that bias
• 10:32impact them directly.
• 10:34So even things like the
• 10:36way that we teach in
• 10:37the manner we teach, and
• 10:39a very specific example could
• 10:40be, you know, on a
• 10:42lecture in dermatology,
• 10:44when we don't in aren't
• 10:45inclusive of different skin tones
• 10:47in those,
• 10:49images that we show of
• 10:50different types of eczema.
• 10:52That's a little bit of
• 10:53queuing, in the learning environment
• 10:55of our own specific biases
• 10:57of what something looks like,
• 10:59and it might lead to
• 11:01a lack of applicability to
• 11:02other backgrounds.
• 11:04Other things that can, impart
• 11:06biases is when our assessment
• 11:08instruments are not objective
• 11:10and really following along with,
• 11:12the types of outcomes and
• 11:13learning that we're looking for,
• 11:15and then our own implicit
• 11:16biases like I've talked about
• 11:18in the last several slides.
• 11:21So I wanted to before
• 11:22we move on to talking
• 11:23about how to combat some
• 11:25of these biases, I wanted
• 11:26to give some,
• 11:28really concrete examples of different
• 11:30sorts of biases that have
• 11:31manifest in the in assessment
• 11:33in medical education
• 11:35just to give you a
• 11:36sense of how prevalent,
• 11:37this problem can be and
• 11:39the impact that it can
• 11:40have on learners in particular.
• 11:43So this is one of
• 11:44my favorite studies that came
• 11:45out in twenty twenty three
• 11:47in academic medicine.
• 11:49And the authors of this
• 11:50study were, family medicine,
• 11:53physicians
• 11:54who were really interested in
• 11:55understanding
• 11:56the halo and horn effect.
• 11:58So,
• 11:59what they did is they
• 12:01developed two, videos showing learner
• 12:03performance.
• 12:05One was a male trainee
• 12:06and one was a female
• 12:07trainee.
• 12:08And for each of those
• 12:10videos, they subdivided them into
• 12:12two different types.
• 12:14Each of those videos were
• 12:15exactly the same. They had
• 12:16the exact same script. The
• 12:17male trainee and the female
• 12:18trainee said the exact same
• 12:20things.
• 12:21For each of those videos,
• 12:22they put a label in
• 12:23the front.
• 12:25One that said this was
• 12:26an above average learner and
• 12:28one that says that was
• 12:29a below average learner. So
• 12:31in total, there were four
• 12:32different types of videos. One
• 12:33for the, two for the
• 12:35male, two for the female,
• 12:36each above or below average.
• 12:39And they gave the,
• 12:41viewer a prompt.
• 12:42So these were faculty members
• 12:44who were, viewed these videos,
• 12:46basically saying they this is
• 12:47an above average learner, and
• 12:49we want you to assess
• 12:50their performance on this particular
• 12:52skill.
• 12:53They had seventy faculty observer.
• 12:55They were each were randomized
• 12:57to one of four videos.
• 12:59And what they found was
• 13:01just by putting the word
• 13:02above average or below average
• 13:04in front of the video,
• 13:06it
• 13:07resulted in significantly lower,
• 13:10scoring for those who were
• 13:11labeled as below
• 13:13average than those who are
• 13:14labeled as above average.
• 13:16And this was irrespective if
• 13:18it was a male or
• 13:19female trainee featured in the
• 13:20video. In fact, they were
• 13:21pretty similar across genders in
• 13:23this particular example.
• 13:25And that was pretty profound
• 13:27a pretty profound finding that
• 13:29just by providing a labeling,
• 13:31irrespective of the the types
• 13:33of skills that were demonstrated
• 13:34in the video,
• 13:36the, faculty observers were already
• 13:38biased towards,
• 13:40scoring that person in a
• 13:41particular way.
• 13:43So those words actually are
• 13:45really powerful. And I think
• 13:46we can think of a
• 13:47couple examples of our own
• 13:49workplace where we have, you
• 13:50know, inherited a learner potentially
• 13:52who maybe has a label
• 13:54of being a difficult learner
• 13:56or who are struggling
• 13:57in which we are already
• 13:58biased in into thinking about
• 14:00their performance in a particular
• 14:02way. So I thought this
• 14:03was a really profound and
• 14:05impactful study. And, when I
• 14:06read it, I was really,
• 14:08it made me really think
• 14:09about the way that I
• 14:11thought about learners who are
• 14:12coming in onto my own
• 14:13service who might have already
• 14:14had that labeling of that
• 14:17halo or horn effect.
• 14:20There's been a number of
• 14:21different studies on the effect
• 14:23of gender in assessment.
• 14:25So here's an I'll give
• 14:27you a couple examples of
• 14:28these, but here's one from
• 14:29the emergency medicine literature.
• 14:31So this was actually a
• 14:32really robust study that was
• 14:34published in, JAMA a number
• 14:36of years ago where they
• 14:38took narrative comments across five
• 14:40different programs
• 14:42and analyzed them for their
• 14:43content,
• 14:45based on gender.
• 14:46So it was two hundred
• 14:48eighty three, residents in total
• 14:50and over ten thousand comments.
• 14:51So it was a really
• 14:52robust study.
• 14:54And they found, after they
• 14:56did the sorting process that
• 14:57men were more likely to
• 14:59receive
• 15:00specific,
• 15:01actionable,
• 15:02competency based feedback
• 15:04and, were more likely to
• 15:06be rated at above expected
• 15:08performance.
• 15:09And that was irrespective
• 15:10of, faculty gender. So whether
• 15:13it was a male or
• 15:13female, faculty,
• 15:15they were more likely to
• 15:16rate, male trainees,
• 15:18this way.
• 15:20For women trainees,
• 15:23they received comments about their
• 15:25low skill levels, and they
• 15:26were often associated with comments
• 15:28about confidence,
• 15:30versus men who are rated
• 15:33at low skilled level who
• 15:34are more likely to receive
• 15:35comments, that contain actionable items.
• 15:39So when a a female
• 15:40trainee received information that said,
• 15:42like, you're not performing at
• 15:43the level expected that of
• 15:45you, it was often because
• 15:46they weren't confident enough, whereas
• 15:49males would receive things like
• 15:51work on your procedural skills
• 15:52in this way.
• 15:54And interestingly, women faculty raters
• 15:57were more likely to rate
• 15:59rate residents at,
• 16:01performing below level, and they
• 16:03were more likely to, rate
• 16:04female residents in particular at
• 16:06performing below level.
• 16:08So it's a really interesting
• 16:09study.
• 16:10Here's a graph pictorially demonstrating
• 16:12some of the trends that
• 16:13they found in the study,
• 16:15regarding other things like trustworthiness
• 16:17and bedside manner, indicating that
• 16:19men were more likely to
• 16:21receive positive comments,
• 16:23on these particular,
• 16:25themes.
• 16:26So really interesting study. There's
• 16:28a little bit of a
• 16:29heterogeneity in the literature about
• 16:31gender bias,
• 16:33in terms of assessment.
• 16:34But this was a really
• 16:36robust study that showed, very
• 16:37clear gender bias, in certain
• 16:39directions.
• 16:42Here's another study for from
• 16:44JGEM, the Journal of General
• 16:45Internal Medicine, in which they
• 16:47took
• 16:48medical student evaluations,
• 16:50and they subdivided them based
• 16:52on gender and also whether
• 16:54or not they received an
• 16:55honors or a pass grade.
• 16:57So this was a multi
• 16:58institutional study, where they looked
• 17:00at a lot of different
• 17:02evaluations
• 17:03and used a technique called
• 17:04natural language processing,
• 17:07which basically allows you to
• 17:08subdivide information,
• 17:10narratively,
• 17:11really quickly using, a little
• 17:13bit of
• 17:14artificial intelligence.
• 17:16And they found that
• 17:18words that differed by gender,
• 17:20usually, especially for female gender,
• 17:23represented personal attributes. So women,
• 17:26were more likely to receive
• 17:28comments like they were lovely
• 17:30or empathetic
• 17:32or fabulous,
• 17:33whereas men got comments more
• 17:35like, relevant,
• 17:37humble, modest, deeper,
• 17:40scientific. So there were some
• 17:42pretty clear gendered differences in
• 17:44the language used to describe,
• 17:47performance in the clinical work
• 17:51space. Similarly, in the same
• 17:53study, they looked at,
• 17:55URM status and honors versus
• 17:57pass grades.
• 17:58They did note in that
• 17:59study that URM students were
• 18:01much less likely to receive
• 18:02honors grades, which we're gonna
• 18:04actually get into into the
• 18:05next study that we'll review
• 18:07together as a group.
• 18:09But they found that personal
• 18:11attribute words were more common
• 18:13in URM students. So things
• 18:14like pleasant,
• 18:17soft,
• 18:18cultural,
• 18:19were associated with, URM students
• 18:22versus,
• 18:23stellar,
• 18:25smart, and impressive or more,
• 18:27associated with those who are
• 18:28non URM.
• 18:30So you can see there
• 18:31are some really strong clustering
• 18:33that occurred based on,
• 18:36status in in the domains
• 18:38of URM versus non URM
• 18:39here in this study.
• 18:40So another really interesting study
• 18:43that made me think, you
• 18:44know, really carefully about the
• 18:45words that I'm using to
• 18:47describe performance in the clinical
• 18:48workspace
• 18:50and why I was assigning
• 18:51those particular
• 18:52adjectives, adverbs to this particular
• 18:55trainee,
• 18:56and made me think really
• 18:57carefully about,
• 18:59why I was using certain
• 19:00language to describe performance.
• 19:04And the last study I
• 19:05wanted to review today, I
• 19:06think, was,
• 19:08kind of, like, set off
• 19:09a little bit of a
• 19:10bomb in the academic world
• 19:12and really made a lot
• 19:13of, leaders in in schools
• 19:15of medicine think deeply about
• 19:17their,
• 19:18systems of assessment
• 19:19and how those systems affect
• 19:21individual learners and their ability
• 19:23to
• 19:24attain certain career achievements.
• 19:26So, this was a really,
• 19:28important study that came out
• 19:30in twenty eighteen,
• 19:33published by, some educators at
• 19:35UCSF.
• 19:36And,
• 19:38where they studied basically how
• 19:39little differences in assessed clinical
• 19:42performance
• 19:43can amplify down the line.
• 19:45So what they essentially did
• 19:47is they looked at four
• 19:48cohorts of their own medical
• 19:50students at a single institution.
• 19:53And the reason why they
• 19:54were interested in looking at
• 19:55their own students is because
• 19:56they were they noticed over
• 19:58the last several years that
• 20:00despite,
• 20:01their efforts at recruiting
• 20:03more URM students,
• 20:05into their school of medicine,
• 20:07they noticed that,
• 20:08despite being successful in recruiting
• 20:10those students in increasing rates,
• 20:12they didn't notice that those
• 20:14students were, entering into competitive
• 20:17residencies at the same rates,
• 20:19as they were recruiting those
• 20:20students in. So they were
• 20:21just trying to understand the
• 20:23upstream
• 20:24reasons why that might be
• 20:25the case.
• 20:26And they noticed when they
• 20:27did that that, the students
• 20:29were less likely to receive
• 20:31designations
• 20:32like AOA,
• 20:34indicating that they had, a
• 20:35high academic achievement within their
• 20:37medical school.
• 20:39And they found, downstream when
• 20:41they looked at why the
• 20:43these students were get less
• 20:44likely to receive AOA designations,
• 20:46they found that there was,
• 20:49differences in grading in the
• 20:50clerkship year in particular
• 20:52that favored,
• 20:53non URM students at their
• 20:54school of medicine.
• 20:56And when they looked a
• 20:58little bit more closely
• 21:00at where this difference in
• 21:01grading was occur and how
• 21:02big that magnitude of difference
• 21:04in grading was, they found
• 21:06that the size and the
• 21:07magnitude of differences were small.
• 21:10But,
• 21:11as a result,
• 21:12received half of its many
• 21:14honors grades and were three
• 21:15times less likely to be
• 21:17selected for honors society. So
• 21:18when I say the size
• 21:19and magnitude of the differences
• 21:21were small, we're talking about,
• 21:23like,
• 21:23scales of point three on
• 21:25a five point scale differences
• 21:27that determined that somebody received
• 21:29honors or not. So it
• 21:30was a really tiny amount.
• 21:31And when we think about
• 21:32our assessment,
• 21:34instruments,
• 21:35it probably was educationally
• 21:37insignificant
• 21:38in terms of their performance,
• 21:39but it actually resulted in
• 21:41real consequences
• 21:42for them.
• 21:43So these authors, term this
• 21:45the amplification
• 21:46cascade. So they saw these
• 21:48tiny differences in assessed performance,
• 21:50which cascaded down the line
• 21:52into larger differences
• 21:54in,
• 21:55grades,
• 21:56selections for awards, and what
• 21:58they also thought was a
• 21:59result of, these students being,
• 22:02less likely to apply into,
• 22:04more competitive specialties
• 22:06because they didn't have that
• 22:07AOA designation that helped them
• 22:09stand apart.
• 22:12And I think it was
• 22:13a really brave thing for
• 22:14the school to publish this
• 22:15out in the literature,
• 22:17to really show that they
• 22:18they found that they had
• 22:19systemic biases within their assessment
• 22:22network.
• 22:23And, as part of this
• 22:24publication, they also shared their
• 22:26process by which they really
• 22:28examine themselves as a school
• 22:29of medicine and their systems
• 22:31to make sure that they
• 22:32were really being thoughtful across
• 22:34the board, not just these
• 22:35individual assessments, but across the
• 22:37board different ways in which
• 22:39they could potentially be inserting
• 22:41biases into their assessment
• 22:43and how they could, from
• 22:44a systems,
• 22:46perspective,
• 22:47change their process of grading
• 22:49to ensure that students were
• 22:51all receiving
• 22:52fair, equitable assessment
• 22:54that allowed them to,
• 22:56you know, reach for whatever
• 22:58subspecialty
• 22:58that they wanted to do
• 23:00for their career aspirations. So
• 23:02I thought this was a
• 23:02really incredible study. I really
• 23:04recommend that you you read
• 23:05it if you're interested in
• 23:06learning a bit more, but
• 23:08it it definitely sent a
• 23:09lot of waves. And I
• 23:10think,
• 23:11this was at the very
• 23:12beginning of a lot of
• 23:13schools choosing to move to
• 23:15a pass fail grading, which,
• 23:16of course, was accelerated,
• 23:18during the pandemic in particular.
• 23:21But it really makes a
• 23:22lot of schools,
• 23:24have to be honest with,
• 23:25why they were designating certain
• 23:27people as honors.
• 23:30So I wanted to just
• 23:31pose a question to the
• 23:32group, and people can feel
• 23:34free, if you're comfortable, to
• 23:35unmute yourself or pop something
• 23:37in the chat.
• 23:39Have you seen bias manifest
• 23:41in the assessment of learners
• 23:42where you work or teach,
• 23:44and how have you seen
• 23:45that manifest?
• 23:47If anyone's willing
• 23:48to share.
• 24:04I love that no one
• 24:05has seen any bias ever
• 24:06manifested in their learning environment.
• 24:08I
• 24:09I have a I have
• 24:10an example. I'm Russell. I'm
• 24:11a Russell. NICU fellow. So
• 24:13I'm transgender. So I'm a
• 24:15trans male, and I,
• 24:17transitioned in medical school from
• 24:19female to male. And I
• 24:21saw all of this in
• 24:22real time how all of
• 24:23a sudden I was getting
• 24:24better marks
• 24:26as a man even though
• 24:27I'm the same person. So
• 24:28I thought it was it's
• 24:29cool to not cool. It's
• 24:31not cool, but it's, like,
• 24:32really interesting to see the
• 24:33data because I know that
• 24:34this happened and has had
• 24:35a has
• 24:37Wow. Thank you for sharing
• 24:38that, Russell. That's, like,
• 24:40such a profound experience of,
• 24:43our gendered biases within assessment.
• 24:45And the fact that you
• 24:46experience that as you transition
• 24:48is, I think,
• 24:50exemplifies
• 24:51exactly the point that I'm
• 24:52trying to make that we
• 24:53just all have our own
• 24:54inherent biases towards certain things.
• 25:00I'll go ahead and share
• 25:01as well. My name is
• 25:02Miriam O'Neil. I'm one of
• 25:04the geriatric medicine fellows,
• 25:06and I I experienced this
• 25:09with a
• 25:10program director who is female,
• 25:14who had a very obvious
• 25:16preference for all the,
• 25:18male residents in the program.
• 25:21And it was almost this
• 25:23this just
• 25:24viewpoint
• 25:25of men being better than
• 25:27women kind of at a
• 25:28baseline,
• 25:29even from a woman.
• 25:31Wow. I'm so sorry that
• 25:33you experienced that. And,
• 25:35I can imagine that
• 25:37especially since I think we
• 25:38all,
• 25:40as I as I tried
• 25:41to,
• 25:43exemplify earlier on, like, this
• 25:45idea that we have these
• 25:46biases is,
• 25:48like, not a comfortable thought
• 25:49and, pointing that out to
• 25:51others could is also extremely
• 25:53uncomfortable and especially with the
• 25:55power dynamic with a program
• 25:56director.
• 25:57It's even more uncomfortable,
• 25:59and,
• 26:01I'm sorry that you experienced
• 26:02that,
• 26:03gender bias within your training.
• 26:05And I'm sure it made
• 26:06it,
• 26:08like, feel like an unsafe
• 26:09space for you to express
• 26:11that discomfort as well.
• 26:17Can I ask a good
• 26:18question, actually? Yeah. Sure.
• 26:20So I I'm I'm one
• 26:22of the surgeons, and,
• 26:24I've seen bias throughout my
• 26:25training.
• 26:26But, I just wanted to
• 26:28ask, like,
• 26:29especially in this climate that
• 26:31we are right now when
• 26:32we're making so much progress
• 26:34with, DEI and we're sort
• 26:35of taking a couple steps
• 26:36backwards,
• 26:38you know, a lot of
• 26:40the viewpoint
• 26:42of all the bias is
• 26:43like you can't really prove
• 26:44it. Right? And it's just
• 26:45more someone's opinion, but we
• 26:47all know it exists
• 26:48profoundly.
• 26:50How do you convince folks
• 26:53that it's there in the
• 26:55workplace,
• 26:55in the educational
• 26:57space, and how do you
• 26:59prove
• 27:00it to them? Because, you
• 27:01know, there are certain facts
• 27:02you can you have data
• 27:03for outcomes, your clinical outcomes.
• 27:05You have data for how
• 27:06many
• 27:07papers you publish this and
• 27:08that. But in in the
• 27:09system, it's much more
• 27:12vague to some degree.
• 27:14Yeah. I think this is
• 27:15such a hard question to
• 27:16answer, Shilpa. Like, when we
• 27:18get into our group activity,
• 27:19there are no, like, validated
• 27:21tools
• 27:23to, like, measure bias,
• 27:25identify bias, you know, in
• 27:27in a predictable
• 27:29and,
• 27:30reproducible way.
• 27:33And so what for me
• 27:34as an assessment person, I've
• 27:36I've, worked in assessment for
• 27:37my entire career.
• 27:39Really trying to be as
• 27:41objective,
• 27:43and really determining
• 27:44behavioral benchmarks for performance
• 27:47is one of the best
• 27:48ways for us to combat
• 27:49bias
• 27:50and to,
• 27:51for us as educational leaders
• 27:53to own up to the
• 27:54benchmarks that we've set forth.
• 27:56So if the if the
• 27:57goal for achievement is this
• 27:59and we're not actually
• 28:01going back to that benchmark,
• 28:03then we're not doing our,
• 28:05learners a service, and we're
• 28:06we're putting ourself at risk
• 28:07for bias. I would say
• 28:09two,
• 28:11proactive and ongoing monitoring of
• 28:13assessment information.
• 28:15You know, are we seeing
• 28:16a
• 28:17trend towards certain subpopulations
• 28:20within our learning environment,
• 28:22getting higher or lower marks
• 28:23than others? And why is
• 28:25that trend occurring? Is this
• 28:27actually and returning to those
• 28:28benchmarks again of the what
• 28:30performance we're actually looking for,
• 28:32what outcomes in learning we're
• 28:33actually looking for, and really
• 28:35deciding whether or not this
• 28:36learner is actually showing us
• 28:37those behaviors. And if they
• 28:39are, then, addressing that that
• 28:42bias within the assessment system.
• 28:44I don't know if I
• 28:45answered your question in a
• 28:46satisfactory way, but that's that's
• 28:47some of the things that
• 28:49come to mind for
• 28:53me.
• 28:54So I wanted to ask
• 28:55the flip question, which is,
• 28:57you know, we obviously
• 28:59are now as educators trying
• 29:00to do this for the
• 29:01students, but how about students
• 29:02rating their professors?
• 29:03I'm sure that there's probably
• 29:05some data on the flip
• 29:06side where female professors probably
• 29:08get ranked a little harder
• 29:09or how they're teaching or
• 29:10whatever.
• 29:11Just curious on that as
• 29:12well.
• 29:13Yeah. And, I think that
• 29:15there's also the Claudia,
• 29:17I I actually don't know
• 29:18of any papers, but I'm
• 29:19sure I could find some
• 29:20pretty easily on a assessment
• 29:22of,
• 29:23attendings.
• 29:25I think there's also, like,
• 29:27the,
• 29:27inherent fear and bias that
• 29:29occurs
• 29:30of rating someone who's,
• 29:33above you in rank or
• 29:34in status.
• 29:36We did, like, a little
• 29:37bit of a pilot a
• 29:38number of years ago at
• 29:39Yale where we asked, surgery,
• 29:41residents,
• 29:43a little bit about how
• 29:44they felt assessing their own
• 29:46supervisors. And there was a
• 29:47lot of discomfort with that
• 29:49because they were they were
• 29:50scared of harming,
• 29:51their supervisors in their academic
• 29:53trajectory.
• 29:55So there's a lot of,
• 29:56complexity
• 29:57to assessment because it's not
• 29:59just,
• 30:00unfortunately, in the current environment
• 30:02that we have,
• 30:03it it really should be
• 30:04as objective as possible, but
• 30:05it in some ways, it's
• 30:06also a little bit of
• 30:08a social,
• 30:09event
• 30:10in which you are, assessing
• 30:11someone else's performance in this
• 30:13complex
• 30:14social environment that is our
• 30:15learning environment. So I'm I'm
• 30:17not sure if I answered
• 30:18your question in a satisfying
• 30:20way, but, I think it's
• 30:21really complicated. Yeah. I also
• 30:23I'm also curious about the
• 30:24response. Like,
• 30:25I don't know what it
• 30:26is, but for example, I,
• 30:27you know, I just got
• 30:28reviews and they were wonderful.
• 30:29But if there's one negative
• 30:30comment in there, I will
• 30:31fixate on the negative as
• 30:32opposed to, you know, ninety
• 30:33nine percent were like, great.
• 30:35I I would be driven
• 30:36to madness by it. Whereas
• 30:37most guys are like, yeah,
• 30:38whatever. I heard one guy
• 30:39say he said, oh, I
• 30:40don't even read my reviews.
• 30:41I'm like, it's such a
• 30:42different approach, and I don't
• 30:43know if it's just a
• 30:44gender or just personality
• 30:46or you know, I'm curious
• 30:47about that as well.
• 30:48Yeah. And it, like, makes
• 30:49you wonder too how many
• 30:50other students or residents felt
• 30:52this way working with me,
• 30:53but they weren't brave enough
• 30:54to say it. You know?
• 30:56I look at it as
• 30:57a victory. If a student's
• 30:58comfortable enough to, like, give
• 30:59me a review that's,
• 31:01constructive, it at least means
• 31:03that I've created a comfortable
• 31:04learning environment potentially.
• 31:06But it's I think that
• 31:07can be hard.
• 31:09And And it would be
• 31:09an interesting study to, like,
• 31:11determine how,
• 31:12you know, different genders or
• 31:14different identities,
• 31:15sort of perceive constructive
• 31:17feedback and, all those kinds
• 31:19of things. So it's a
• 31:20very complicated question. No. I
• 31:22actually now I try to
• 31:23because it's very tough to
• 31:23get feedback, so I try
• 31:24to elicit it after any
• 31:26session and conscious of what
• 31:27I can do just to
• 31:28improve and just be open
• 31:29to it as a positive
• 31:30thing, which is a different
• 31:31way of approaching it. Like,
• 31:33good or bad, I'd like
• 31:34to know, and that way
• 31:35can only get better. So,
• 31:37I agree. You have to
• 31:38just approach it in a
• 31:38positive way.
• 31:40Totally. Yeah.
• 31:41Alright. So let's talk about
• 31:43moving forward a little bit,
• 31:45and different strategies we can
• 31:46use. And I think, through
• 31:47our discussion recently, we we
• 31:49went over a couple different,
• 31:51strategies. But,
• 31:53you know, I think awareness
• 31:54is really important.
• 31:56There's no a hundred percent
• 31:58panacea for us to remove
• 31:59biases completely from, the way
• 32:01that we perceive the world
• 32:03around us. But understanding where
• 32:05bias comes from,
• 32:06having intentionality,
• 32:08and taking pauses can be
• 32:09really helpful, and we'll go
• 32:10through some some different strategies
• 32:12we can use to try
• 32:13to reduce bias. So,
• 32:15in the past, you know,
• 32:16like, I think it was
• 32:18regarded as, like, this
• 32:20conscious intentional aberrant. And certainly,
• 32:22there are still, unfortunately,
• 32:24examples of people being consciously,
• 32:28racist or biased or, any
• 32:30other,
• 32:30ist kind of,
• 32:32label there. But for a
• 32:33lot of us, it's like
• 32:34an unintentional,
• 32:36type of,
• 32:38tendency that we all have.
• 32:39So these days, we kind
• 32:41of see bias as a
• 32:42normative, unconscious,
• 32:44and largely unintentional,
• 32:46act that, all of us
• 32:48are predisposed to in our
• 32:49in our various ways based
• 32:51on what we bring to
• 32:51the table and our own
• 32:52life experiences.
• 32:55And,
• 32:56like some of this, it
• 32:58can become over time a
• 32:59little bit hardwired into our
• 33:01cognitive functioning,
• 33:03in which we have these
• 33:04intentions. We have but we
• 33:06have this internal wiring from
• 33:07our own lived experiences
• 33:09that includes our emotions,
• 33:11you know, our past behaviors,
• 33:13our expectations,
• 33:14how we frame things,
• 33:16and result in our actions
• 33:18and decisions down the line.
• 33:19But the good news is
• 33:20if if we really work
• 33:22intentionally, some of that wiring
• 33:24can be undone
• 33:25by being really intentional about
• 33:27exposing ourselves to different experiences,
• 33:29different people, and being open
• 33:31to feedback on our assessment
• 33:33information,
• 33:34over time.
• 33:36Here's an example of just
• 33:37how how hard our brain
• 33:39can be wired in certain
• 33:40ways. So,
• 33:42how many of you think
• 33:43that
• 33:44square
• 33:45a is darker than square
• 33:46b?
• 33:52I'm gonna say that we
• 33:53all probably see square a
• 33:54as darker than square b.
• 33:56Victor
• 33:57raised raised their hand. So
• 33:58yeah. Agree. But when we
• 34:00actually line it up with
• 34:02a same color,
• 34:04bar on either side, square
• 34:06a is exactly the same
• 34:08color as square b. And
• 34:09even in this picture and
• 34:11toggling back and forth, I
• 34:12still can't see that square
• 34:14a is the same color
• 34:16as square b, but that's
• 34:17my brain. It's just the
• 34:19hardwiring of my brain to
• 34:20interpret differences
• 34:22that I see in the
• 34:22world around me,
• 34:24even though they're the exact
• 34:25same color. So it can
• 34:26be really hard to for
• 34:27us to our brains to
• 34:28overcome some of this hardwiring
• 34:30that we've developed over the
• 34:32course of our lives.
• 34:34And we are, as physicians,
• 34:36we're really prone to bias
• 34:37because of our work environment
• 34:39and because of our learning
• 34:40environments.
• 34:41The things that predispose us
• 34:43to that type one reasoning,
• 34:44that quick thinking
• 34:46that makes us more prone
• 34:47to bias are stress, multitasking,
• 34:50time problems,
• 34:51needing to wrap stuff up.
• 34:53I I, as a physician,
• 34:54I'm constantly trying to wrap
• 34:55stuff up so I can
• 34:56move on to the next
• 34:57thing.
• 34:57Fatigue, sometimes even fear of
• 35:00repercussions
• 35:01of our assessments can change
• 35:03the way that we think
• 35:04about things. So we're really
• 35:05predisposed to this.
• 35:08And, you know, bias and
• 35:10cognitive error can't be fully
• 35:11trained out, like I said
• 35:12before,
• 35:13but we can reshape
• 35:15our implicit attitudes and curb
• 35:17their effects on our assessment.
• 35:19Being objective,
• 35:20being reflective,
• 35:22getting external feedback, that can
• 35:23all help in various ways.
• 35:26So,
• 35:27some things that can be
• 35:28helpful,
• 35:29and that have been shown
• 35:30in the literature to reduce,
• 35:33variability in assessment and bias
• 35:35are, being open to faculty
• 35:36development. So the fact that
• 35:37you're all here,
• 35:39is a really good sign
• 35:40that you're really open to
• 35:41changing and,
• 35:43working on,
• 35:44getting feedback to reduce biases
• 35:46in your assessment.
• 35:48Trying your best to recognize
• 35:50inferences that you make about
• 35:52learners. So,
• 35:53I will, give you an
• 35:54example in just a second
• 35:55of an assessment that I
• 35:56did on a resident recently
• 35:58that has a little bit
• 35:59of an inference in it,
• 36:00and trying to reduce those
• 36:02inferences. Like, for example,
• 36:05you know, determining someone's performance
• 36:07based on their level of
• 36:08training as an inference,
• 36:10or their intentionality is, in
• 36:12a particular,
• 36:13learning environment is an inference.
• 36:15Trying our best to use
• 36:17behaviorally based language.
• 36:19So avoiding,
• 36:20language that's focused on personality,
• 36:23like they were nice or
• 36:24they were compassionate,
• 36:25but rather saying things like
• 36:27they spent an hour at
• 36:28the bedside with mister Johnson,
• 36:30I think can be really,
• 36:31really helpful in identifying what
• 36:34the behaviors are that we're
• 36:35seeing that makes us think
• 36:36that this person's performing well
• 36:38in the workplace.
• 36:40Using assessment instruments and guides
• 36:43can be really helpful, and
• 36:44I encourage for those of
• 36:45you who do assessments on
• 36:47students and residents, really taking
• 36:48the time to read that
• 36:50assessment instrument
• 36:51because I know the Yale
• 36:52School of Medicine and the
• 36:54residency programs have worked really
• 36:55hard
• 36:56to try to ground assessments
• 36:58in actual observable behaviors,
• 37:01to make sure that we're
• 37:02assessing,
• 37:03trainees in the most objective
• 37:04way possible.
• 37:06So, trying our best to
• 37:07be directly observe our learners,
• 37:10really being objective in what
• 37:12we saw,
• 37:13and not presumptive in what
• 37:14we thought the intention was
• 37:16in that interaction,
• 37:18using criterion rather than normative
• 37:20reference scales,
• 37:22using competency based tools. Those
• 37:24are all really, really helpful.
• 37:25Even behavioral checklists can be
• 37:27really helpful,
• 37:28in reducing biases.
• 37:30And then the other thing
• 37:31that can be extremely helpful,
• 37:33and this is this comes
• 37:34from a lot of the
• 37:34reliability
• 37:35data and assessment,
• 37:37out in the world and
• 37:38competency based assessment is it's
• 37:40actually really helpful if learners
• 37:42get lots and lots of
• 37:43different observations by different faculty.
• 37:46So for those of you
• 37:46who are in leadership roles
• 37:48in education,
• 37:49the more assessments you get
• 37:51of learners, the more viewpoints
• 37:52of different assessors you get,
• 37:54the less bias, you will
• 37:56have and the clearer picture
• 37:57you would have of that
• 37:58learner in particular.
• 38:00So, that can be a
• 38:01really helpful way to sort
• 38:03of
• 38:03even out the assessments,
• 38:05and even out the biases
• 38:06that come to the table
• 38:07naturally as part of assessment.
• 38:11So individually,
• 38:12we can recognize that we
• 38:14all have biases and just
• 38:15own up to that and
• 38:16be honest with ourselves.
• 38:18I found it helpful to
• 38:20get feedback.
• 38:22So that can be in
• 38:23the form of having someone
• 38:25else review your assessment,
• 38:27or using an a tool
• 38:29or instrument,
• 38:30to help, reduce assessment. And
• 38:32I'll I'll share with you,
• 38:33that, actually,
• 38:35artificial intelligence can be a
• 38:36helpful way to do this.
• 38:37I'll share an example in
• 38:39just a second of me
• 38:40using artificial intelligence to,
• 38:42evaluate one of my recent
• 38:44assessments.
• 38:46Be practice what's called constructive
• 38:48uncertainty.
• 38:49So really thinking about observing
• 38:51yourself in action
• 38:53and being more thoughtful in
• 38:55considering your perspectives and understanding
• 38:57that your perspective
• 38:59is come from your own
• 39:00background and other people might
• 39:02see the situation a little
• 39:03differently. I'll share with you
• 39:04in the next slide a
• 39:05technique called pause, that can
• 39:07be helpful in making that
• 39:08that,
• 39:09constructive uncertainty thing.
• 39:13You know, be comfortable with
• 39:14the awkwardness and discomfort
• 39:16of being honest with yourself,
• 39:18when you're thinking about your
• 39:20own biases
• 39:21and, being intentional about engaging
• 39:23with those who are different.
• 39:25Listen to their perspectives and
• 39:27experiences
• 39:28through appreciative inquiry.
• 39:30We just went through our
• 39:32first rank meeting for, my
• 39:33med peds program, and I
• 39:35was so appreciative
• 39:37that I had a really
• 39:38diverse panel of individuals helping
• 39:40me construct that rank list
• 39:42because it helps me to
• 39:43see things about individual,
• 39:45applicants that I did not
• 39:46see without their help. And
• 39:48it was really, really helpful
• 39:50to be able to use
• 39:51them as, a helpful framing,
• 39:53reference guide of, different perspectives
• 39:56on what these individual applicants
• 39:58brought to the table.
• 40:00Here's that technique I mentioned.
• 40:02It's called pause.
• 40:03Just a way for you
• 40:05us all to just pause
• 40:06for a moment and think
• 40:07about what we're doing in
• 40:09the moment of assessment.
• 40:11And, you know, I I
• 40:12have been there with you
• 40:14guys. I've filled out a
• 40:15bunch of residency
• 40:16assessment forms. I filled out
• 40:18a bunch of med school
• 40:19assessment forms. Taking the time
• 40:21to assess learners is really
• 40:22hard,
• 40:23takes a lot of brain
• 40:24space, it takes time.
• 40:26And,
• 40:27but these are really impactful
• 40:29things that go on to
• 40:30their their dean's letter, help
• 40:32to determine whether or not
• 40:33they're able to progress through
• 40:34residency.
• 40:35And so taking the time
• 40:36to pause can be really
• 40:37helpful. So pay attention to
• 40:38what you're assessing
• 40:40and acknowledge your own reactions,
• 40:42judgments, emotions,
• 40:44in that assessment.
• 40:46Understand that maybe there's some,
• 40:49other viewpoints,
• 40:51or approaches in that moment,
• 40:53and try to be as
• 40:54objective as possible in framing
• 40:56your assessment,
• 40:57and then execute an assessment
• 40:59with minimal bias.
• 41:02Was there a question in
• 41:03the chat? I just wanna
• 41:04make sure I'm not missing
• 41:05anything. Okay. Great.
• 41:08So I wanted to share
• 41:09with you just in, like
• 41:10like, an honest way, a
• 41:12recent assessment I did of
• 41:13a learner,
• 41:15and how I used, ChatGPT
• 41:17to, just assess whether or
• 41:19not I was using any
• 41:21biases in my own assessment.
• 41:22So this is a de
• 41:23identified,
• 41:25assessment of an intern that
• 41:27I worked with several weeks
• 41:28ago.
• 41:30And, here's just an example
• 41:32of some of the language
• 41:32that I used in my
• 41:33assessment. So overall, x is
• 41:35operating well above their train
• 41:37level of training. They demonstrated
• 41:39excellent communication,
• 41:40emotional intelligence, and clinical reasoning
• 41:42skills.
• 41:43Despite the business of busyness
• 41:45of service and, several patients
• 41:47who are acutely ill, x
• 41:49maintained a calm and professional
• 41:50demeanor that enhanced trust. They
• 41:52follow through on tasks, and
• 41:54I could always trust them
• 41:55to ask for help if
• 41:56they were not sure.
• 41:58And in the, sort of,
• 42:00reinforcing
• 42:01feedback, I I said,
• 42:03you know, despite some frustrations
• 42:04regarding patient care delivery,
• 42:06x used the energy of
• 42:07their frustration effectively by communicating
• 42:10with our nursing staff and
• 42:11filling an RL report.
• 42:13So I actually,
• 42:15de identified this and, put
• 42:17it on chat GPT
• 42:18just to, and I asked
• 42:19chat g p t to
• 42:21analyze my assessment for any
• 42:23biases,
• 42:24gender biases,
• 42:26you know, any any types
• 42:27of biases.
• 42:28And it was actually a
• 42:29really helpful output because it
• 42:30helped me to it provided,
• 42:33like, a bulleted feedback on,
• 42:35my assessment and actually helped
• 42:37me to to reflect on
• 42:38a couple different things, not
• 42:40just for the statement, but
• 42:41for my assessment as a
• 42:42whole. So I'm just gonna
• 42:43give you an example of
• 42:44what CHAT GPT spat out
• 42:46for me, just a couple
• 42:47of the bullets that it
• 42:48provided that I thought were
• 42:49good thinking points. It it
• 42:51probably wouldn't change too much
• 42:52what I put in the
• 42:53content of my assessment, but
• 42:55it helped me to think
• 42:55about it.
• 42:57So,
• 42:58ChatGPT
• 42:59basically said, here are some
• 43:01potential areas of bias in
• 43:02your assessment that you should
• 43:03think about. So one was
• 43:05the lack of comparative
• 43:06context, which I actually agreed
• 43:08with, that I thought I
• 43:10probably would if I were
• 43:11to go back and change
• 43:12my assessment, I would do
• 43:13this. So,
• 43:14while the review states x
• 43:16is operating well above their
• 43:17level of training, it doesn't
• 43:18provide a benchmark or comparison
• 43:20with peers.
• 43:21This omission may intent unintentionally
• 43:23reflect implicit biases if similar
• 43:25language is not consistently applied
• 43:27to others of different backgrounds.
• 43:29And I think that's really
• 43:30important. You know? I wasn't
• 43:32really using criterion.
• 43:33I was using norm reference
• 43:35language when I made that
• 43:36statement.
• 43:37And,
• 43:38I really should have, hearkened
• 43:40back to, like, what were
• 43:41the behaviors of this particular
• 43:43intern that made me feel
• 43:45like they were a super
• 43:46high performer?
• 43:47And I I should have,
• 43:49added a little bit more
• 43:50phrasing to help,
• 43:52the the person reading my
• 43:53review, the program director, understand
• 43:55a little bit more,
• 43:57why this purse person was
• 43:58performing at a level that
• 44:00I said was above the
• 44:01level of an intern. So
• 44:02I thought this was helpful
• 44:03feedback.
• 44:05It also pointed out that
• 44:06potentially,
• 44:07and, you know, depending on
• 44:08the, self identified gender of
• 44:10this individual,
• 44:11I could have been using
• 44:13gendered expectations. So,
• 44:15calm and professional demeanor,
• 44:17is, noted as enhancing trust,
• 44:20and this could be gendered
• 44:21as male trainees are often
• 44:23assumed to be calm and
• 44:24authoritative.
• 44:25And, actually, this trainee
• 44:26was a male. So, it
• 44:28was helpful for me to
• 44:29kinda think about that, whether
• 44:31or not that was
• 44:32helpful,
• 44:34information to provide.
• 44:35Ultimately, I don't know if
• 44:37I would change this language
• 44:38a ton, but it was
• 44:39helpful for me to kind
• 44:40of reflect on whether or
• 44:41not I could have maybe
• 44:43given a more specific example
• 44:44of,
• 44:45this individual at the bedside
• 44:47talking with a patient and
• 44:48what the outcome of that
• 44:49conversation was.
• 44:51And then similarly, frustration framing.
• 44:54So the RL statement that
• 44:56I put in there where
• 44:57they used,
• 44:59a frustrating situation to file
• 45:01an RL,
• 45:02Chiachi b t pointed out
• 45:03that, you know, if in,
• 45:05certain situations,
• 45:07this framing of assertiveness could
• 45:09be a positive, and that's
• 45:10that's often mostly associated with
• 45:12males
• 45:13versus it might not be
• 45:14equally celebrated in women or
• 45:16individuals and to be thoughtful
• 45:17about the language,
• 45:19or surrounding that. So I
• 45:20actually thought this was a
• 45:21really helpful exercise for me.
• 45:23It didn't really change too
• 45:24much of what I would
• 45:25write in the content of
• 45:26my assessment, but it was
• 45:27a really helpful
• 45:28reflection exercise.
• 45:30So I would love let's
• 45:32see how we're doing. We
• 45:32have fifteen minutes. So I'd
• 45:34love to take maybe about
• 45:36seven minutes,
• 45:38individually for you to pull
• 45:40up a recent evaluation
• 45:42you completed on a trainee.
• 45:44Ideally, pick something that has
• 45:46a lot of, words,
• 45:49or narrative to it. If
• 45:50you don't have something like
• 45:51that, maybe pulling up a
• 45:52recent letter of recommendation that
• 45:54you might have written.
• 45:56And then use the worksheet
• 45:57that hopefully, Linda has posted
• 45:59in the chat or Sarah,
• 46:00one of the two, has
• 46:02posted posted in the chat
• 46:03to evaluate your written assessment.
• 46:06We'll take a couple minutes.
• 46:08We'll I'll I'll try five
• 46:09minutes. So we'll wrap up
• 46:10at twelve fifty.
• 46:12And then I might ask
• 46:13some volunteers
• 46:14to share your reflections.
• 46:16So we'll take until twelve
• 46:18fifty
• 46:19for each person to maybe
• 46:20pull up an evaluation. In
• 46:22the meantime, feel free to
• 46:23ask questions.
• 46:25And I will post repost
• 46:27the handout because there have
• 46:28been some chat activities since
• 46:30that one. But if you
• 46:31got so if you got
• 46:32it before, this is the
• 46:33same one.
• 49:32Okay. In the interest of
• 49:34time, I'm gonna gather everybody
• 49:36back together. I know that
• 49:37five minutes is not nearly
• 49:38enough time to reflect deeply
• 49:40on such
• 49:42a important topic, but I'm
• 49:44just kind of curious,
• 49:46of the group who had
• 49:47a chance to review some
• 49:49of your recent assessments or
• 49:50a recent assessment.
• 49:52Did any folks have any,
• 49:54takeaways or things that, potentially
• 49:57they felt,
• 49:59was helpful in this reflection?
• 50:19Wondering also if anybody tried
• 50:21chat GPT.
• 50:27To answer,
• 50:29William Rando's question, I'm actually
• 50:30not familiar with this, literature
• 50:32on stereotype threat and the
• 50:33use of smart feedback. So
• 50:35if you wanna share what
• 50:36you know, I would very
• 50:37much welcome any thoughts that
• 50:39you have.
• 50:40Oh, thank you, Katie.
• 50:43I I will it it's
• 50:44it's a,
• 50:46it deals with many of
• 50:47the things that that that
• 50:48you're talking about here.
• 50:52And but people might be
• 50:53interested. It it does provide
• 50:55there's been research done and
• 50:57and
• 50:58on the way
• 51:00the different ways that we
• 51:02give feedback
• 51:04to, underrepresented
• 51:06minorities
• 51:07and
• 51:08and and also research on
• 51:09the effect of that feedback.
• 51:10And I'll just make one
• 51:11point because it's interesting,
• 51:13which is research shows that
• 51:16any
• 51:17that that a person who
• 51:19is an,
• 51:21an unrepresented minority in a
• 51:22group, whether it's the only
• 51:24woman in a largely male
• 51:26group or,
• 51:28tend to take negative feedback
• 51:31as evidence that they don't
• 51:33belong.
• 51:34Mhmm. Whereas members of the
• 51:36of the majority population take
• 51:38it as, oh, I I
• 51:39need to get better at
• 51:40that. And so
• 51:42the authors have developed something
• 51:44called smart feedback,
• 51:46which is a way
• 51:47designed to mitigate
• 51:49that process.
• 51:51And and,
• 51:52maybe we could talk about
• 51:54it sometime.
• 51:55Yeah. I assume that it's
• 51:56like the smart framework of
• 51:57specific, measurable,
• 51:59actionable, timely.
• 52:01Yeah. Yeah. I think it
• 52:02like, that all hearkens back
• 52:04to being as objective as
• 52:05possible. And I I think
• 52:07one of the other things
• 52:08that's really important about these
• 52:10like, the benchmark,
• 52:12thing that I mentioned before
• 52:13that Chopa asked about was,
• 52:15I think it's also important
• 52:16that we make sure that
• 52:17those benchmark
• 52:18benchmarks are really clear to
• 52:20our learners,
• 52:22so that they know what
• 52:23they're trying to achieve,
• 52:25and that we we go
• 52:26back to those,
• 52:27as leaders, as educational leaders,
• 52:30that this is the goal,
• 52:31and, that's why you're getting
• 52:33this feedback, either something that's
• 52:35constructive or, something that's positive
• 52:37for that individual.
• 52:39And, you know, if they
• 52:40are achieving above that to,
• 52:42like, point that out,
• 52:44I think those objective benchmarks
• 52:46are really important. They're very
• 52:48hard to identify
• 52:50because of the work that
• 52:51we do is really complex,
• 52:52but they're very important to
• 52:54set out,
• 52:55from the get go and
• 52:57use.
• 52:59Any other reflections from this
• 53:00activity?
• 53:03I I have another question.
• 53:04Sorry.
• 53:05It was really interesting and
• 53:07fascinating to put chat GPT
• 53:09and see what your biases
• 53:10are.
• 53:11Has anybody
• 53:12I think that's great because
• 53:13I'm gonna do that now
• 53:14for some of my evals.
• 53:16Has anybody
• 53:17done that as a study?
• 53:19And then is there a
• 53:20way to use that to
• 53:21train faculty
• 53:23to
• 53:24be more specific in their
• 53:25feedback?
• 53:27Yeah. I,
• 53:29I don't think anyone has
• 53:30specifically
• 53:31published anything on using CHAT
• 53:32GPT,
• 53:33as a study for,
• 53:35evaluation of assessment.
• 53:38There have been other studies
• 53:39where people get, like,
• 53:41objective evaluation on their assessment,
• 53:43I believe,
• 53:44and to, like, improve their
• 53:45performance.
• 53:46But I don't think, like,
• 53:48using AI has been a
• 53:49thing. One of the things
• 53:50that's really challenging because we're
• 53:52we're trying to use AI
• 53:53in our assessment,
• 53:54for our CCC and our
• 53:56internal medicine program here. And
• 53:58one of the tensions that
• 53:59we have as a group
• 54:00is,
• 54:01using private information in what
• 54:03can, like, be sifted through
• 54:05in a public domain
• 54:06and whether or not there's,
• 54:07like, more,
• 54:09safe,
• 54:09AI technologies that could be
• 54:11used that are more like
• 54:12HIPAA compliant,
• 54:13which is why I encourage
• 54:14folks to, like, de identify,
• 54:16their assessments before putting it
• 54:18into the
• 54:19ether that is ChatGPT.
• 54:22But I think it would
• 54:22be a really interesting study
• 54:24to look at.
• 54:25Definitely, there's been some natural
• 54:26language processing studies to look
• 54:28at biases and assessment and
• 54:30narrative assessments.
• 54:31Natural language processing is a
• 54:33little, I feel like a
• 54:34little less robust than some
• 54:36of the AI technologies that
• 54:37are coming out, so it'll
• 54:38be interesting to see what
• 54:40other information we get in
• 54:41in upcoming years.
• 54:47Other reflect any reflections?
• 54:51I I thought this was
• 54:51a really helpful exercise, and
• 54:53I was struck with the
• 54:55review of my own evaluation
• 54:57for how often I was
• 54:58sort of sharing my opinion.
• 55:01And so kind of reframing
• 55:02this is, you know, this
• 55:03is not my opinion of
• 55:04how this trainee is doing.
• 55:06This is my evidence
• 55:08for
• 55:09what what level they're meeting
• 55:11or not meeting. And I
• 55:12noticed that, at least in
• 55:14the med heavy valve that
• 55:15we use, if you want
• 55:16to give someone a five,
• 55:17you have to justify it.
• 55:19And my evidence
• 55:21was in those justifications, but
• 55:22not in my summary
• 55:24comments. My summary comments included
• 55:25a lot of the language
• 55:26that,
• 55:27you've warned about. And,
• 55:30when I'm reading it back,
• 55:31I notice it's a lot
• 55:32of sort of, like, my
• 55:33impression
• 55:34rather than,
• 55:36objective data like you were
• 55:37talking about. So I I
• 55:37think this was super helpful.
• 55:38Thank you for having us
• 55:39do this. Yeah. Dana, were
• 55:41you gonna say something?
• 55:43Well, I was just gonna
• 55:44say I appreciate that it
• 55:45everybody took the that you
• 55:46had us go through this
• 55:47and people are taking the
• 55:48time. But I think, potentially,
• 55:50one of the very powerful
• 55:52things that you emphasized
• 55:54is that maybe people don't
• 55:55take the time because they
• 55:56don't think there's a big
• 55:57impact.
• 55:57Like, pilots do a checklist
• 55:59because they're gonna crash,
• 56:01but they don't think the
• 56:02vet educational crashing. And it's
• 56:04it's worth taking the time
• 56:05to do the check you
• 56:06know, the mental the pause.
• 56:08And, I really appreciate that
• 56:10because I think people undervalue,
• 56:13the the consequences of their
• 56:15assessments.
• 56:16Yeah. I I think that's
• 56:17such an important point to
• 56:18make, Dana. Like, the words
• 56:20that we use to describe
• 56:21our learners can have long
• 56:23lasting impacts on them, you
• 56:24know, whether or not they
• 56:26apply for that really highly
• 56:27competitive residency
• 56:29or whether or not they're
• 56:30thought about for chief positions
• 56:32and,
• 56:32all all kinds of things
• 56:34like that. So,
• 56:35but I think more importantly
• 56:37at the, like, crux of
• 56:38it, making sure that we're
• 56:39objective objective in our language,
• 56:41we're,
• 56:42helps our learners more. You
• 56:44know? Like, whether or not
• 56:46I'm likable does not help
• 56:47me very much. But if
• 56:48I'm showing some behaviors that,
• 56:51make patients feel uncomfortable,
• 56:53with my clinical care, that's
• 56:55something I could actually
• 56:56change,
• 56:57and,
• 56:59alter and focus on, as
• 57:01part of that feedback cycle.
• 57:03So,
• 57:04helping our learners to achieve
• 57:06the outcomes that we're hoping
• 57:07of them being competent,
• 57:09thoughtful, compassionate
• 57:10clinicians,
• 57:11and what behaviors we need
• 57:13to see to show that
• 57:13they can do that is
• 57:15really what the the goal
• 57:16is here.
• 57:19Awesome.
• 57:19I'm sorry. Kate, can I
• 57:21share real quickly?
• 57:22Of course.
• 57:24So I put a, letter
• 57:26of recommendation I did for
• 57:28a student.
• 57:29I'm not currently in a
• 57:30situation where I'm teaching. And
• 57:32so I thought it was
• 57:32sort of interesting. I kind
• 57:33of got back,
• 57:35a couple
• 57:36things, potential
• 57:37problem areas.
• 57:39Mhmm. One in gender bias
• 57:40because I highlighted the student's,
• 57:43evaluation of women's health and
• 57:45her work as a certified
• 57:46doula.
• 57:47And although those are positive
• 57:49aspects of her profile is
• 57:50was essential to ensure they
• 57:52don't overshadow her other achievements
• 57:54or reduce her gender based
• 57:56roles.
• 57:57I also got dinged on
• 57:59potential cultural bias.
• 58:01I mentioned the student had
• 58:03co founded a writing workshop
• 58:04for women of color.
• 58:06And although this is a
• 58:07positive reflection of her commitment
• 58:09to diversity
• 58:10and inclusion,
• 58:11it's crucial to ensure that
• 58:12it's not the only context
• 58:13in which your leadership is
• 58:14recognized
• 58:15as it might inadvertently create
• 58:17a perception of pigeonholing her
• 58:18into specific cultural identity based
• 58:21roles.
• 58:22I got three positive observations
• 58:24because I highlighted that she,
• 58:27had graduated Phi Beta Kappa
• 58:28magna cum laude,
• 58:31and,
• 58:32that both were considered objective
• 58:34achievements,
• 58:36and that I had pointed
• 58:37out her leadership
• 58:39and community service,
• 58:41which presented her in a
• 58:42positive light. And, also, I
• 58:44had pointed out her professional
• 58:46experience,
• 58:48and continued dedication to women's
• 58:50health. So sort of a
• 58:51mixed bag, but kinda highlighted
• 58:53for me as a woman
• 58:54of color,
• 58:55also
• 58:56being potentially biased
• 58:58when I'm writing these letters.
• 58:59So very helpful.
• 59:01Yeah. I and, I think
• 59:03it's, thanks for sharing that,
• 59:04Anne.
• 59:05I think it's also important
• 59:06to realize, like, ChatGPT and
• 59:08other AIs are not perfect
• 59:09at all. And sometimes you'll
• 59:11read some of the feedback
• 59:12you get from it, and
• 59:13you'll be like, I don't
• 59:14agree with that. You know,
• 59:15like, maybe that person being
• 59:16a doula for five years
• 59:18or whatever their experience was
• 59:19is really important for their
• 59:20application for whatever position that
• 59:22they're going for. So you
• 59:24can always choose to disagree
• 59:25with the feedback you're receiving.
• 59:27But I have found it
• 59:28helpful to just have, like,
• 59:29someone looking over my shoulder
• 59:31in a way,
• 59:33thinking about things that maybe
• 59:34from a different perspective than
• 59:36I might have thought about
• 59:37that for that particular individual.
• 59:39I similarly put in a
• 59:40letter of recommendation I wrote,
• 59:41for one of my junior
• 59:43faculty in the program,
• 59:44and it made me really
• 59:45think about, like, what did
• 59:46I mean by her lived
• 59:48experiences,
• 59:49and, like,
• 59:51being a little bit more
• 59:52objective about what what specifically
• 59:55I meant in that letter
• 59:55of recommendation.
• 59:57So thank you for sharing
• 59:58that, Anne, and thank for
• 59:59thank you for everyone who
• 60:00participated.
• 01:00:02So in summary, we're all
• 01:00:03prone to bias when we
• 01:00:04assess.
• 01:00:05It can have lasting impact
• 01:00:07on our learners,
• 01:00:08and implicit bias can be
• 01:00:10recognized through thoughtful reflection
• 01:00:12and I think in particular
• 01:00:14external feedback. So I welcome
• 01:00:16all of us to,
• 01:00:18be vulnerable and ask for
• 01:00:19feedback from our colleagues or
• 01:00:21if you feel more comfortable
• 01:00:22using,
• 01:00:23AI technology or other tools
• 01:00:25to
• 01:00:26get that objective feedback on
• 01:00:27the assessments that you provide.
• 01:00:30Here is a link that
• 01:00:31I was told I I
• 01:00:32need to provide for feedback
• 01:00:34for me. I I very
• 01:00:35much welcome feedback. I love
• 01:00:36getting feedback.
• 01:00:38So please be objective in
• 01:00:39your feedback.
• 01:00:40I appreciate it.
• 01:00:42And if there are any
• 01:00:43remaining questions, I'm happy to
• 01:00:44take them.
• 01:00:48We do really appreciate your
• 01:00:49feedback. Even if you have
• 01:00:51to run off for clinic
• 01:00:52or something,
• 01:00:53please grab the QR code.
• 01:00:55And, Katie, maybe while people
• 01:00:57are doing that, you can
• 01:00:58just show the last slide
• 01:00:59so we can just remind
• 01:01:00people that we have our
• 01:01:02next session coming up with
• 01:01:03Andreas Martin, who will talk
• 01:01:05about starting
• 01:01:06your, scholarly writing.
• 01:01:08And we we have a
• 01:01:09med ed discussion group,
• 01:01:11with the new
• 01:01:13executive director of,
• 01:01:15IntHealth,
• 01:01:17about advancing health professions education
• 01:01:19worldwide. So click on that
• 01:01:21QR code to register. And,
• 01:01:23Katie,
• 01:01:24we miss you. You can
• 01:01:26you maybe you could have
• 01:01:27med here and pedes there
• 01:01:28or something.
• 01:01:30We wish we wish you
• 01:01:31luck in your inaugural match
• 01:01:33and all benefited from your
• 01:01:35from your presence and your
• 01:01:36talk.
• 01:01:37Thank you to everyone for
• 01:01:38inviting me.
• 01:01:40Thanks for your great questions.
• 01:01:42Have a great day.
• 01:01:44Alright.
• 01:01:46Katie, I have to pop
• 01:01:47off to a meeting, but
• 01:01:48I know you were busy
• 01:01:49as well after. So we'll