Well, good morning, everybody. Thank you so much for coming. We'll probably have a few people filtering in, but I thought we should get started. This is our reviewers workshop. We always try to do something a little different every year. We have such great reviewers. And this is one way that, in our meager way, we can thank you. We try to cover different topics every year, not just the same elementary stuff, to make you even greater reviewers than you already are. So we have two fantastic speakers today. And the first is Jim Carey. Jim is our associate editor for epidemiology and statistics, has been doing that for quite a while at AJSM, and is soon to be professor of orthopedic surgery at the University of Pennsylvania. So Jim, please take it away. All right, well, thank you. Thank you for the invitation to present. This is my eighth reviewer's workshop, which loosely parallels about eight of these critical reader editorials. Today's is on understanding statistical graphics in sports medicine. All right, my conflicts of interests are disclosed here. So the objectives for today are to recognize some standard depictions of data, flow diagrams, box and whisker plots, Kaplan-Meier survival curves, using medical literature, including orthopedic sports medicine literature, to discuss the assessment of the content of the visualized data in these standard depictions, and then to briefly explore some visual presentations of variation of data and uncertainty. So I want to make an argument that text alone is sometimes insufficient and sometimes suboptimal. Here's an example. It says, in case you ever need help telling what kind of drum and bass someone is talking about, I made this handy little chart, a little two by two grid. And I'd argue there's some parallels to surgical techniques. Like a lateral extraticular tinnitus varies more than a drum, I think. Here's an example where text alone would be suboptimal. Here's the highest grossing musician by state. And I think you can kind of quickly find your own state and find the data. And I think we'd agree this would be pretty boring if it was a series of paragraphs or a table. And Pennsylvania has Taylor Swift, so we're doing pretty well. But when is text alone all right? I think in fiction it is. I think filling gaps in text with creativity, we find this fun and rewarding. Did anyone here ever play Zork, this text-based adventure game? No one played these games? Where the player explores the ruins of the great underground empire. Sold a lot of copies in the early 80s. In 2007, it was included in the game canon by the Library of Congress as one of the 10 most important video games in history. And the game looked like this on a monochrome screen, west of house. You're standing in an open field west of a White House with a boarded front door. There's a small mailbox here. And as the player of the game, you had to kind of envision the whole environment. And then open mailbox, you type in. And in the mailbox, you find a leaflet. And it's very fun because you're filling the gaps with your own creativity. Similarly, with books, reading books, you compare that to watching a movie. Books allow the reader to be part of the story and spurs imagination. In one of the studies looking at 800 books adapted for film, it was clear that people preferred books to movies about 75% of the time. And most of the rest of the time, it was a tie. There's only about 4% of the time people thought the movie was better or much better. In contrast, in science, we don't fill the gaps with creativity. Well, it's a kind of form of creativity, but it's really confirmation bias, which is a term that refers to the interpretation of evidence in ways that are partial to existing beliefs and expectations. And so when there's gaps or matters that are not clearly depicted, the reader automatically fills these gaps with what the reader already believes. So if I imagine I do this. If I saw a report on a study that showed that there were more failures in an autographed ACL reconstruction with autograph group compared to an allographic group, but age was not clearly depicted. And I would fill this gap with my belief that the allograph group was older and less active. In that way, the findings would be consistent with my beliefs. Again, the subsequent part of this presentation loosely parallels the critical reader editorial visualizing data in case someone wants to dive in a little bit deeper. But it starts, I mean, summer vacation. So imagine that you select a new place to go. Your neighbor recommended it. You and your family have loaded the vehicle. Your spouse is driving. You're the navigator. And your neighbor sends you a text message. It says, good luck. We'll watch your house while you're gone and pick up your Amazon packages. And then a text message of about 50 directions. Head northeast on 33rd Street. Turn right on Chestnut Street near the burger place. Turn right onto Schuylkill Avenue. Take the ramp onto I-76 East. I mean, I don't think this would be sufficient for me now. Even though the directions are detailed and accurate and it would get you there, you would have to look at every street. Like, is this Chestnut Street? And then you get to the next one. Is this Chestnut Street? And you wouldn't be able to have attention to other things. So you may wish to complement these directions with a map. That would allow you to better understand the distances and relative positions of locations. It would allow the navigator to anticipate when to turn on Chestnut Street. When's the turn approaching? And then when you can see it in some of the maps now, we even drop your dot on the map. You can follow as you go and how you're approaching things. And now the navigator does not have to be looking at every upcoming intersection. They can focus on road hazards and plan the proper music for the road trip. That's very important, of course. Similarly, figures and publications guide the reader in interpretation of the study findings that are described in the text. The data graphics visually display measured qualities. And the three standard depictions we're going to go over today are flow diagrams, box and whisker plots, and Kaplan-Meier curves. I would argue that familiarity is favored over novelty when we're looking at these figures. The data should differ from study to study, but not the format. Just like when I go from Philadelphia to Denver, the map differs, but not the format. North is always up. I can see the layers of the buildings, the streets. I can superimpose traffic. Novel orthopedic sports medicine belongs in our family of journals. Novel depictions of data may be more appropriate for a dedicated journal of statistics, epidemiology, and methods. So let's start with the flow diagram, a visual display of the ordered steps of a process. A flow diagram can be helpful for most study types. In fact, templates of flow diagrams are currently provided in guidelines for authors of randomized controlled trials, systematic reviews, diagnostic studies. For randomized trials, a flow diagram can depict the number of participants who were assessed for eligibility, randomized, allocated to each intervention, loss of follow-up, and analyzed. So here's an example from the sports medicine literature, an article on matrix-applied characterized autologous culture chondrocytes versus microfracture with Daniel Sarris as the first author, and he gave me permission to review this. In this flow diagram, the reader can easily follow the flow of patients and can readily assess how many patients are in each treatment arm, how many were assessed by second look arthroscopy and MRI evaluation, the number of patients in each treatment arm, that withdrew from the studies also clearly presented along with the reasons for withdrawal. You can see all that pretty clearly in this flow diagram. The next topic we'll cover are box and whisker plots. It's a graphing tool that characterizes a sample of data using the 25th percentile, the lower quartile, the 50th percentile, the median, and 75th percentile, the upper quartile. You can see an example of a box and whisker plot on the right. The core element is the box, and its length is the inner quartile range, spanning from the lower quartile to the upper quartile, which covers the central 50 percent of the data. The median is represented by a line inside the box, and again, the median is the middle value. If there's an even number of values, then the median is calculated as the average between the two middle values. The whiskers are the lines that extend from the edges of the box, and the whisker has three common variations in length. One of the most common whisker lengths is extending to the most extreme data point that is no more than 1.5 times the inner quartile range from the edges of the box. Another common variation length is when the whisker extends to the 10th and 90th percentiles, and sometimes the whisker can just extend all the way to the minimum and maximum data values. Outliers beyond the whiskers are individually plotted. You can see on the right there's one outlier there at the bottom, and the box and whisker plots do not require any assumptions on the shape of the distribution, so this is a real strength of these plots. They can depict skewed data quite well. So here's an example, which also in the cartilage repair literature. And in these box and whisker plots, the clear boxes represent data from the microfracture group, the shaded boxes represent data from the characterized chondrocyte group, and the upper and lower edges of each box, as always, indicate the 25th and 75th percentiles. Again, the solid horizontal line in each box indicates the median, and the ends of the whiskers indicate the 10th and 90th percentiles. You can see how now you can kind of compare the two groups better than you would if you just had the mean or if you just had a range of values. You can really get a sense for where the bulk of the outcomes lie. And I should also note, beyond the whiskers, you can see the outliers are shown as individual data points, and there are a lot of outliers in these groups, a lot of variation in the data. And the third type we're going to review are these Kaplan-Meier curves. They're based on an innovative statistical method of estimating survival rates when there are incomplete survival observations, including the data, which is most of our data. The original article of note by the statisticians, Kaplan and Meier, is the most frequently cited statistics paper. For the plot, the vertical axis reflects the percent surviving, and the horizontal axis reflects time. And rather than just using predetermined intervals, such as one year, two year, five year, the Kaplan-Meier method identifies the exact point of time that death occurs or the event of interest occurs. At that point, the cumulative survival can be calculated using a number alive at each time, or the number without the event, and the number who died at each time, or the number that had the event of interest. And these curves were initially popularized in medical literature for survival of patients with cancer, as shown on the right. But this mathematical method can be identically applied to any potential event. So, one of the examples in orthopedic sports medicine comes from the pediatric ACL literature with Dr. Ganley as the senior author. And in this, he presents four Kaplan-Meier survival curves. We'll zoom in on the top two a little bit. In these Kaplan-Meier survival curves, graft rupture-free survival is what's being depicted. A, the top figure, all epiphyseal ACL reconstruction, transphysial ACL less than 16 years old, and transphysial ACL reconstruction more than 16 years old are represented alongside each other. And you can see, you can compare the curves. And then what B through D did, or the bottom part of this figure is B, each subgroup is represented separately with point estimates and 95 percent confidence interval. And I think this is where the real value comes in, seeing the uncertainty, and seeing how the uncertainty increases over time because there are less survivors informing the outcome as time goes on. And you also note that each curve is plotted in a stepwise fashion. This is classic for the Kaplan-Meier survival curves, rather than a smooth curve because the survival remains unchanged until the next graft rupture occurs, and that's what the steps are. So for visualizing data, I have three summary points. The first is to ensure the study is using standard visualizations. Again, when I look at my electronic map, I expect to see the same layers each time. The data is expected to differ, but not the format. The depictions of data in medical literature should be straightforward and conventional in general. Familiarity is favored over novelty. And I presented the flow diagrams, box and whisker plots, and Kaplan-Meier curves as typical visualizations in our medical literature. For the second summary point, I recommend that we assess the content of the visualized data. We know that depicting the important information is very important to do. Graphical displays should show data and reveal the data at several levels of detail, just like on our map, so we can zoom in, we can zoom out. Be skeptical of an article that does not reveal key data transparently and completely. And the third summary point, evaluate the details of the data, especially with respect to uncertainty. Sir William Osler stated, medicine is a science of uncertainty and an art of probability, and there's uncertainty in all of our research. How is this uncertainty presented to the reader? How is it depicted? An effective display of data must remind us that the data being displayed contains uncertainty and must characterize the size of the uncertainty. And we just saw how box and whisper plots can depict the spread of data, the variation, the variance. Kaplan-Meier curves with their shaded regions can depict 95% confidence intervals nicely. Thank you. Yeah, so I think it's up to the individual studying the authors the best way to present that particular data. I think that the top one is the one I see the most in our orthopedic sports medicine literature, where the whisker goes no more than 1.5 times the interquartile range from the edge of the box. But again, the example I showed from Dr. Saris used the 10th and 90th percentiles, which things also find there is still there's more outliers maybe that way to be depicted, but I still feel like you get a good sense of the data either way, and a lot better than with just a mean in a range or just a median value in a range, you know. I don't think so. Right, I agree with that. Yeah, I think it's very important because by the end of an orthopedic sports medicine study on cartilage repair or ACL reconstruction, by the time we get to 25, 30 years, maybe that plot is informed by five patients that they're able to follow up with. So that's why I like, I appreciate what the authors of this study did. They had them all superimposed on each other so you can kind of follow the best point estimate over time. But I like having the uncertainty depicted with a 95% confidence interval, and you can really see how that kind of blossoms over time. This is just at four years at the end, but you can imagine at 20 years how few patients may be actually able to be studied that would be considered surviving at that time point when the next graph failed. And that 95% confidence interval gets wider and wider over time. So I would encourage authors to include the 95% confidence interval plots like this study did whenever possible, which should be most of the time. Yeah, I didn't go over it, yeah, the question was what about sensor data, which would be basically any withdrawals that occurred before that point are subtracted, so that there would be an ACL failure in this group, but in between now and then, maybe a patient moved to another area or a patient died or whatever. That actually becomes part of the calculation when you look at the number alive at each time point when it's calculated. Yeah, I think that's a great point. I think there's a lot more room to put more information on these plots. Like, if you look at plot B, I mean, you could have many little time points, like how many patients were left at different time points. Think about the rich information we're able to interpret on a Google map, and we can really process a lot more information than are typically on these graphs. So I would encourage you to include as much as we can. I think it would make sense to have a cutoff. I'm not aware of anything other than a rule of thumb. If there's less than five survivors, you know, the studies should be, I mean, it still may be useful to follow patients over time. It could be a unique data set, but usually that's when it concludes. All right, if there's no further questions, I'll step down. Thank you. One lesson I learned early in my career was surround yourself with people who know more than you do about all the things you don't know much about. So Jim's one of them, and so is our next speaker. Our next speaker is Dr. Derek Davis. He's professor of radiology at the University of Maryland, and he is the only radiologist on our editorial board at the moment. I know he's got a great presentation for us, so Derek, take it away. OK, hello. I think it's a great honor to be invited to speak at this meeting. I remember when I was a much younger radiologist, I remember coming to one of these talks. It was one year was local to Baltimore and it was right around the corner. So I was able to come and watch. And I was sort of inspired by what I saw. And I find sports medicine fascinating. And ever since, I've really wanted to stay in touch. So in terms of this talk, the title is Anatomic Imaging in Sports Medicine Manuscripts, Pearls and Pitfalls for How to Approach a Review. I know that, you know, AGSM is a world renowned journal and there's all sorts of high end orthopedic papers submitted. But some of them, you know, have a different point of view. Some of them are sort of centered on imaging, which, you know, imaging is the focus. And for those papers, I want to give those manuscripts. I want to give a few comments. OK, I have nothing to disclose. I want to talk about the learning objectives of this activity. After this educational activity, participants will be able to list commonly encountered anatomic imaging modalities in sports medicine manuscripts, discuss when anatomic imaging adds value or distracts from the stated purpose of a manuscript, and specify examples of imaging do's and don'ts in manuscripts for reviewers to recognize with respect to study design, methodology and interpretation results. OK, so there are many different ways to image a patient. We live in a wonderful age where there's functional imaging, there's metabolic imaging, and then there's the old fashioned anatomic imaging. I think probably most people in this room would be most comfortable with the anatomic imaging and I have to admit, so would I. So we're going to focus on the common imaging modalities that you're probably most likely come into contact in your manuscript reviews. So common modalities include x-ray, MRI, CT, and ultrasound. And so those are the big four that you're probably already very familiar with. So in terms of the review, you know, you all get the emails asking you to either accept or decline the review. And when you accept it, you go to the website and pull up the manuscript. OK, so you pull it up and you look at it, trying to figure out what it is. And so if it's an imaging paper, it may be requiring a different point of view from what you're normally used to looking at. Sometimes the outcomes are related directly to the imaging. Sometimes the imaging is the star of the show for the manuscript and the authors are pushing a certain point of view or style. So the first thing I would do is just, you know, pick up that manuscript and read the title and read the purpose. You know, it's usually right there on the front of the abstract. And you should ask yourself, you know, is imaging significant in the title and the purpose? That's the number one question because it may require a special point of view. If yes, then what you should do is you should just look quickly through the manuscript to see what's there. The number one question I need to know is, do the figures actually contain images? So I just showed you four different basic images as examples. If a manuscript is about imaging and there's not a single image, then that's a potentially red flag. OK, so that's not uncommon. That happens. Sometimes the authors are focused on the data, not necessarily the modality. But if the modality is how they got the data, then that may be a red flag. You may need to have examples for the readers to see or to see how the data was acquired. OK, now some of these are not really imaging papers. They're just, you know, imaging is a tool. They used it, they collected data, and it's to support the overall study design and outcomes. It's not necessarily that important other than a tool. So those probably require less scrutiny. They're not that important. So the next thing I would do after looking at the title and looking for the purpose is you really need to, like, figure out what the study design is. That's critical. Not all papers are created equal. The study design, you know, may be special. So one stands out for me in particular. You need to ask yourself, you know, is the manuscript a descriptive epidemiological study? OK, that's really important. This is a special type of study design. In general, you know, these are mostly population-based, not necessarily about the individual. So how the data is acquired and how the data is interpreted may be slightly different. So what are some good things? Well, the good things are the data probably already exists. So somebody collected the data. It's in a database. Hopefully it's a well-known database that can be verified. And if there's imaging data in it, it may be something that you really don't need to look into that hard, because the authors may not have personally collected or analyzed the images. They're just using what's in the database. So that's understandable. They're using that data that's already acquired to sort of support their hypotheses, study their hypotheses, and come up with some conclusions because the data is already there. You know, there are some downsides to these type of studies with respect to imaging. They usually probably have less information about the individual participant. Maybe they don't know how the data was actually collected. They just have the data. What type of MRI scanner was used? What was the interval of the scan? You know, what was the protocol? They may not know. All they have is the end result, which is fine for what they're trying to do. But I think probably the thing that you should criticize the manuscript if they're making bold statements about the imaging. If the purpose was to study imaging and their conclusions are bold statements, then you really should be skeptical of that because these studies, the data exists already. The authors probably didn't collect it. They're just using it to support their ideas. And you probably should pay close attention and make sure that the conclusions are not overreaching what the data can actually provide. That's very important. Now, there are other types of common study designs that you're more likely to come into contact with a lot. And these will definitely require close scrutiny if it is an imaging paper. So these include clinical trials, cohort studies, case control studies, cross-sectional studies, case series, and then laboratory studies. These are probably the bread and butter of what you're going to be seeing. Now, what you need to do is sometimes, you know, the abstract self proclaims what the study design is. But you may be aware that sometimes the self-described study design is probably not what you would think the study is. So just be careful that you agree with what the paper self-described the study design because sometimes the study design is incorrect. That's one thing you want to pay close attention to. So the next thing you want to do, you've decided that it is an imaging paper. So you should just look really quickly at the introduction section. And there's a few questions you want to ask yourself. Does the manuscript really establish an appropriate context for the imaging that they're talking about? Is the use of the imaging novel? Those are important questions. And a main question that you should consider is, have you ever heard of this? Is this something out of the blue? It seems strange. Does the manuscript actually support what they're saying? You know, it should be, if they're going to use this imaging or this approach, do they give you something in the introduction section to make you feel comfortable that this has been done before and it's not brand new and they just didn't make it up? So, you know, there's some questions we all sort of like think about when we're reading the introduction section when we're trying to figure out what they're trying to present. So if it's an imaging paper, you should think to yourself, you know, is the imaging analysis actually clinically relevant? I think we've all been there. We're reading something. They're doing really important things, but we're trying to figure out how that actually fits into our lives. And we're seeing our patients. How does this help us? Does it even make sense? Does the manuscript actually address a knowledge gap? That's an important question if they're using imaging. Okay, now there are some red flags that maybe might stand out very, very up front to you. You know, when I'm reading the end of the introduction, looking at the purpose and the title, and I'm thinking to myself, you know, is the manuscript's main point already common in clinical practice? You know, so many times I've been reading through whatever manuscript, and they have some really big ideas, and they seem great, and they're probably going to prove it. But I think to myself, well, the thing that they think they're introducing, I do that every day. Like sometimes with the imaging, they profess to doing something with MRI or CT or ultrasound, and I think to myself, well, I already do that every day. So they just may be unaware that radiologists already do certain things. It's very common. So that's a key question. Okay, this is great. You proved a lot. You have positive, you know, statistically significant P values. Everything's great, but are we already doing this, or does this even make sense? So that's something you should think about. So now just some words specifically about the methods. Methods are really important for all manuscripts. They're very important for the imaging, but there's a lot of what ifs, and so I'm going to try to boil things down to the simplest facts. There are a lot of things I could have said, but I'm going to try to keep it as basic as possible. So when we talk about the imaging protocol and the technique, there's a few questions that I ask myself when I'm going through the manuscript. You know, are the details of the protocol and technique well explained? And the reason why I'm asking that is, you know, are these, whatever they're talking about, is it easily reproducible? Could I go do that myself? Could my colleagues at my institution do it? Because if something seems very difficult to reproduce, the question is, you know, how valid is it? And so that's always a thought that I have is, okay, what is your protocol? Can pretty much any investigator with similar knowledge do the same thing? That's an important question. Now, there are some potential red flags as you go through any imaging manuscript. Sometimes they're either limited or no information given about, like, the basics, you know. So you're going to do CT or MRI, but you don't say what planes you're looking at. For MRI specifically, you say you do MRI, but you don't really say what sequences were done. And these are simple things, but they're very important because just because you did MRI doesn't mean that others would be able to reproduce it. And in terms of, you know, standardization, you know, when you explain what your protocol is, when you say what your sequences are, then people can first potentially reproduce it and then maybe look for red flags that maybe what you're saying isn't necessarily correct. And so those are really important things to think about. Other red flags. And these are very common, and they happen. And it's not that these are bad. It's just that they should be explained well. One is, okay, you did a study with multiple participants with multiple MRIs or CTs or ultrasounds or x-rays. Okay. So if imaging was done at multiple facilities or institutions, if you don't adjudicate that somehow in your limitation section, explain why you did it anyway and sort of mitigate why it doesn't matter. If there's no explanation, then that's a potential red flag. Another potential red flag is that you did MRI, you did ultrasound, but maybe all study participants didn't get the exact same type of scan. So this is really important in imaging because it's sort of like baking a cake. You know, you bake the cake. Everybody knows that you go to different restaurants, the cake's slightly different. They use different ingredients. They cook at different temperatures. It tastes different. Maybe you have a preference for this restaurant and not that restaurant. So in imaging, it's critical to know where the imaging was done. Was it done all in the same place on the same scanner? Did every patient or participant receive the same protocol? And if not, then at minimum, the manuscript needs to explain in the limitation section that this was known and have some appropriate explanation. That's sort of like a minimum standard. So another thing that I think about when I'm reading a manuscript, and sometimes it's not necessarily straightforward. At some point, you have to think to yourself, is this study design plausible? Does it make sense? I mean, they used imaging. everything looks great. It looks awesome, but does it make sense? So as an example, imagine a study where they sort of did a surgery and then three years later, did an MRI to study the post-operative outcomes. So that seems great. They can totally do that. You can do MRI three years after the surgery, but the problem is in the study design, you have no idea what existed before the surgery. So if there's no preoperative MRI or immediate post-operative MRI, or even some type of surgical note for comparison, how can you really sort of judge what was done three years later? So that's like the plausibility. And sometimes it, in manuscripts, it looks very impressive except that, okay, you did the MRI, but where's the relative fact there? Was the cartilage already missing pre-op or not? Did it rub off or not? So there's a lot of different things that you could think about. Other things. Now, these are very important when you talk about study participants. One of the things that I look for most in a manuscript is just basics. Did they describe how old the patients are? Did they appropriately account for gender in the manuscript? Because when you're dealing with imaging, if you don't know the age of the participants, how can you judge the imaging? Because people change as they age. So if you're imaging a 20-year-old versus an 80-year-old, what I'd expect to find is completely different. And so what's normal for one age group is not normal for the other. And if you don't even have the basics of sort of like who the participants are in terms of age, that makes a big difference for imaging. In terms of male versus female, that sometimes is important because there are some basic differences between men and women. And not that you can't look at people together, but you should consider, would it be appropriate for them to have also stratified by gender to look at both groups separately? Because sometimes when you do that, important things pop out. Okay. So some other basics specific to imaging. Okay, so if the manuscript is an imaging paper, the principal question I ask myself are are the image analyses done with blind and independent raters? Okay, because a lot of journals and participants and their data used to be done by consensus. But if you reading by consensus, meaning that there is no independent assessment that everybody sort of comes together and comes up with an agreement about what they see, this consensus analysis is sort of falling out of favor. So if it's truly an imaging paper, then at minimum, you need at least two independent raters to judge whatever images by themselves independently. Okay, another question I ask myself, did the research team members analyze images to create data for this statistical analysis? So what that means is if this is truly an imaging paper, then it's most appropriate for the research team to have created the data. Because whatever data they've collect is gonna be analyzed. So I consider a research team member either an author of the manuscript or a person who's at least receiving an acknowledgement by the manuscript. Okay. So a lot of manuscripts, what they do is they say they had the MRIs or the X-rays or CTs, and then they reviewed the clinical imaging reports, which is one way that you could do it, but you should probably very skeptical because usually in clinical medicine, you don't know who's interpreting the images. Now in general, at least in the United States, it's a radiologist, but you don't really know who that radiologist is. Some radiologists or general radiologists with maybe not the best expertise. So just because a report says something, how do you know that's true? So in general, it's okay to have the clinical reports, but the question is, did the research team reanalyze the images and come up with their own conclusions? That's probably the most appropriate standard. Using the clinical reports, that's maybe difficult. Because at least somebody needs to verify that the clinical reports were true. That would make the data much more trustworthy. So other things about the methods. Does the manuscript clearly list how images were obtained, viewed and analyzed? And so some basic key questions. It all comes back to reproducibility. So if you're doing an imaging study, that's great. Is the data reproducible? Can somebody else not connected to the research team at a different institution or a different hospital basically do the same experiment and come up with similar things? Is the methodology supported by the literature? These are important things. So some potential red flags, which are minor, but happen a lot. So sometimes a manuscript will list that imaging analysis was done, but they don't say who's doing it. So there's many different stakeholders who could potentially be the image analyzer. So this is a big list. It goes from medical student all the way to top expert. Medical students have a valid role in doing image analysis, but it depends on what they're doing. So the thing is, there's some things that are very simple and there's some things that are very advanced. So I've done rotator cuffs research in the past and some things you could teach a medical student easily to do. So quantitative imaging, you can do manual segmentation. You can train a high school student to do that. It's very simple. It's just sort of you're outlining a muscle and pressing a button. There's other things that you do where it's sort of semi-quantitative or qualitative, where it's not necessarily that a med student could have the same knowledge as an orthopedic surgeon who's been working for 10 years, that some things need a real expert to look at it. So that's really important. And so when I look at an image analysis, my first question is who's doing the analysis? If training would really affect what they find, so you'd wanna have skilled raters doing the image analysis. So let's say you're looking at Goutelier grade, okay? If you're a working orthopedic surgeon who repairs rotator cuff, you're doing that all the time. So you're probably gonna trust orthopedic surgeons who repair the rotator cuff to sort of figure out what the fat infiltration is in the rotator cuff and assign a Goutelier grade. I don't know if you want a medical student who just came working with you to be the person doing that because they could do it, but how good are they at doing that? And that would be questionable. So you wouldn't want a situation where rater one is a fellowship trained orthopedic surgeon and rater two is a medical student because there's sort of a mismatch in knowledge for that specific task. Okay. Another problem with reproducibility is that sometimes a method is subject to measurement error. I remember reviewing a paper and they were measuring little tiny things. And I remember looking at the data and realizing that they were measuring things that were like a few millimeters, like two millimeters, and that their measurements went all the way to the thousandths place. And it was so small that I had to wrap my head around and I said, wait a minute. And I remember like we do these measurements all the time in clinical medicine and the difference between two and three millimeters on a clinical scan is nothing. If I measure the same thing 800 times, I'd probably get different measurements at least 400 times if they're really small. So you have to ask yourself, is this plausible? You're measuring things that are two millimeters and you're making judgments off of something that's 100 magnitudes less than a millimeter. Maybe that's not the best experiment to be running. So you wanna think, is this plausible? Okay. Now, when it comes to statistical analysis, you wanna ask some basic questions. Does the manuscript have any information on agreement between raters? That's very important. So even a basic measurement, you should be able to compare the raters to each other. If the manuscript doesn't have that, maybe they should put that back in. Sometimes the people who are doing this, aren't the most sophisticated with the statistical language. So they get at that, they tried to do that. But if you can't really find in the literature what they're talking about, then it's a problem. So as a real example, there was a manuscript, they were talking about how they were trying to compare the raters. But they were trying very hard, but they just didn't have the proper way to do it. And so in my review, I just mentioned that maybe they should ask the statistician to try to help them to try to bridge that gap. Because they were definitely trying, it was just that they weren't fully aware of what that meant. They tried, but their language wasn't probably gonna be acceptable to publish in a high rank journal, because you want to be able to communicate like a real agreement that most people would understand as like a real way to do it. And so that sometimes a manuscript is very close, but they need a little help from the statistician. So some brief words about the results. So some potential red flags with the data. If you're looking at images, and you're looking and the figures have images, but they don't match the method section, that's a red flag. Sometimes they'll say they did MRI, but then they're showing you CT or vice versa, or they'll show you one thing that it just doesn't match. So you're saying, well, the data that you're, the results don't match what you said you're gonna do in the method section. Sometimes, it's very small things where they don't list the imaging parameters. They'll list, they'll say what they're gonna do. And then when you look at it, they failed to basically say what they actually did. That's very common where they say something in the methods, but when you actually look at what they did, it doesn't match. So that's a problem. Other potential red flags that they don't actually, they're providing data that was never described in the method section. So I look at the table, I look at the data, I look at the graphs and I say, well, when did, when did they do this? And I look in the methods and it's not talked about. So as a real life example, let's say it was about the rotator cuff and they described the methods, and in the result section, they're talking about Goutelier grade, but nowhere in the methods did they actually say how they did the Goutelier grade or that they were gonna do it. And so those are just some basic things. So just to finish up, I'm just gonna talk a little bit about the figures. This is a figure from a recent journal article that the figures should be straightforward. This figure should be the ally of the paper. That's why when I first get an imaging paper, I look to see if there are any images. So the figure legends should explain the image so that anybody could basically interpret it. So you wanna write it to a level that it doesn't take a top radiologist to understand the image. So does the figure legend actually match the figure? That's just a basic question. Other potential red flags. You look at the figure and it just doesn't seem to fit, like the description doesn't fit the image. Sometimes the image shown doesn't match what they said in the methods, and sometimes the figure legend is just wrong. And so you just wanna make sure that if you have an image that they at least come close to describing what they're talking about, that sometimes the manuscript is challenged. Other things. Well, sometimes when you review a manuscript and you're well aware of whatever the abnormalities they're talking about, and you look at the image and you say, okay, I know they're trying to talk about this particular abnormality, but why did they pick this image? Like it's just doesn't seem to fit, doesn't seem to do justice what they're trying to say. So that's a red flag. And the figure legend should be written in a way that the non-expert can know. Okay, now lastly, these are the big red flags, that it's an imaging paper and they seem to do well, but then when they actually show you an image, it's just very poor quality. And that's one of those things where you basically, you don't have to be an imaging expert, you know it when you see it, that either you have no idea what's happening in the image or it looks like somebody took the image and took it through a car wash and then sent it in to be published. And the worst crime of all is for it to send an image in with written patient information on the screen. So that's actually pretty bad, that pretty much violates a lot of ethical code. So you should never have an image with the patient information. Okay, so to conclude in this educational activity, we discussed commonly encountered anatomic imaging modalities. We discussed anatomic imaging in sports medicine manuscripts. And we described some imaging do's and don'ts in manuscripts for our viewers to recognize. Thank you. So, to basically briefly summarize, I think what you're saying is that to have the people of different abilities, a medical student versus a top orthopedic surgeon, it's not necessarily a killer for the paper, but it should be somehow addressed in the limitation section. And sometimes a limitation is just sort of a statement. So I always think that in order for imaging paper to be fully whole, you have to sort of disclose enough that a third party could reproduce the same experiment. So I understand what you're saying, like for the rotator cuff, you don't know like how did they figure out where to take the image. And so that needs to be fully disclosed how that was done. Sometimes it really does take an expert who's been working for years to be able to do a particular task that you can't really, you could compare a medical student, but medical students really aren't trained to read images, they're sort of beginning their journey. And so I think a paper is judged how it handles the details, explaining the imaging protocol, saying what methods were used, how were the images obtained. I think these are key. And some manuscripts sort of gloss that over. And that's bad. But in terms of different people with different abilities doing the rating, it really matters what the task is. Like there's some simple tasks that even a high school student could do if they're just outlining a muscle. But if the paper sort of reports the reliability of the task, at least enter observer, they've done, they've sort of disclosed what they did. You can scientifically say, well, this is the cap, or this is the intercorrelation coefficient. You can say, well, this is the degree of agreement. And I think that's probably full disclosure, whether the agreement was bad or good, at least they've done that. And I think if there are some papers, they give you everything, they say, well, we had the top surgeon compared to the medical student, the agreement wasn't that good. And they'll say, well, there's level of difference in levels of experience. If all they had was the medical student and the top orthopedic surgeon, then the question is, what data are you using for your analysis? If they're taking several raters and they're sort of making an average, and it seems about right, then it's okay. So I would say it's all in the details in terms of whether you accept it. I think full disclosure with how they did it, and then the agreement between raters, that's a great first start. There's always something that can be approved, and that's where the reviewers can really come in and say, okay, you've done great, but we really need this extra information from you before we can consider you for publication. Well, I think, you know, Tesla strength is just one metric in the performance of a scan. So I think 1.5 and 3.0 are equivalent in terms of publication of clinical experiments. Now, sometimes you need higher Tesla strength to do experimental protocols, like for the cartilage, like 70 or 11. But, you know, what I would say is, you know, I had the analogy of a cake. Are you gonna bake the cake at 325 or 425? The end result is the taste. Does it taste good or not? And you're gonna have different ingredients. So I think the Tesla strength is, it depends. Now, if you're imaging metal, you want the lower field strength. So if you're doing hip arthroplasty or some other, like a unicompartmental knee, then you need the lower field strength. The higher field strength may be a disaster for the scan. Yeah, I mean, it's specific, so like for the quantitative, I think the ICC may be preferred. This mic is live, if you want to tell people, this mic is live, if you want to ask questions. So, well, the question was, is the ICC or CAPA better for analyzing the agreement? So, in general, the ICC is very good for quantitative data. But if you're dealing with, let's say, semi-quantitative, like an ordinal scale or some other scale, then it sort of falls short, and then you have to rely on things like CAPA. So, it all boils down to the data about what test you want to use. Yeah, so the question was if you have two reviewers who are giving conflicting answers, how do you adjudicate that? I think probably now the best standard is you have a third person come in and adjudicate that. I guess you could do by consensus, but then again, people in the imaging world have moved away from consensus entirely. It's not, you know, I think probably both are acceptable, either a third or a consensus, as long as they disclose how many times it's happened. So if they reviewed like 200 cases and it happened five times, what's the big deal? But if it happened 70 times, then that may be a problem. Okay, so the question's about ultrasound. So, ultrasound is a black box. I think it's difficult because with ultrasound, there's certain things that are straightforward. You know, it depends what the experiment's for. So, let's say the Achilles tendon. The Achilles tendon is right under the skin. It's superficial. It's pretty simple. I think probably most people could learn how to scan that within two days. If you're doing something specifically with the hip, it's very deep, you know, snapping hip syndrome. It really takes a lot of skill for the best sonographers to even get that done. And so, I've made the mistake of trying to do an ultrasound on people with very high BMI. And sometimes I'm left questioning why I did that. So, you know, ultrasound, I would say, if they're in a manuscript, if they're sort of giving you full disclosure about who did the scan and their level of expertise, they need to sort of emphasize the weaknesses that it's operator-dependent and patient-dependent. And so, it's not, I think the ultrasound is valid. I just think probably a lot of things, you know, it's sort of a alternative test for a lot that it's hard to exactly reproduce the same conditions for every single person because it's the operator may do different things, the participant may be the wrong size or wrong candidate. And so, I think you, it's sort of buyer beware. I mean, you know, if it can't be reproduced by most, then you should probably say, well, how useful is this? But it's in the hands of experts, it can be excellent and it depends on what it is. That's probably the best political answer I can give. Thank you for your time. Thank you. It's very heartening to me to see so many people coming for these reviewers' workshops because I know you always want to learn more and be better reviewers and that's why we're so grateful and lucky to have you. I want to thank both Jim and Derek for lending their expertise to us. Jim always has entertaining insights and we've learned so much from him over the years and Derek is a newer member of our team but he reminds me, you know I've been in this profession for a little while, of the old-fashioned radiologists that I knew when I was starting out even in medical school who really had a lot of experience in clinical medicine. Derek really understands clinical medicine. Sometimes some of the other editors who don't know he's a radiologist send him purely clinical papers and the evaluations he writes are truly amazing. So anyway, thanks for all that you do as reviewers. For those of you who are on the editorial board, we've got an editorial board meeting coming up at 1.30. We always are grateful for your feedback on the evaluation of these workshops so please do evaluate the workshops and please do suggest topics that might be helpful for next year. Thank you very much.

reviewers' workshop

American Journal of Sports Medicine

AJSM

Jim Carey

Derek Davis

statistical graphics

anatomic imaging

manuscript reviews

sports medicine literature

data visualization