Tuesday, July 25, 2017

Exclusion

Here's question 7 of the CSWA Workplace Climate Survey:

In your current position, how often have you been VERBALLY harassed because of the following characteristics?

{Choices include: Often, Sometimes, Rarely, Never}
Race or Ethnicity
Physical Disability Status
Mental Disability Status
Sexual Orientation
Gender Identity(Cisgender or Transgender)
Gender (Female, Male, or Non-binary)
Religion or Lack Thereof

Question 9 has the same form, with "PHYSICALLY" replacing "VERBALLY". Now, here's hypothesis 1:

Female respondents will report more verbal and physical harassment than men.

This sentence appears in the first paragraph of the results section:

Thirty-nine percent of respondents report experiencing verbal harassment at their current position, and 9% report experiencing physical harassment.

And this one appears in their support for hypothesis 1:

Women were also significantly more likely than men to report that they experienced both verbal and physical harassment because of their gender.

I have underlined that phrase because it draws attention to the glaring absence in the questionnaire of a "characteristic" that is likely to have been focus of harassment directed at white male astronomers. Indeed, though the paper doesn't tell us this (we know it only from Christina Richey's preliminary presentations at DPS in 2015 and AAS in 2016), race and gender account for the great majority of characteristics that people felt they had been harassed for. In the case of verbal harassment, they account for 65% of the reports. In the case of physical harassment they account for more than 80%.

For obvious reasons white male astronomers are not likely to report being victims of harassment because of their race or gender. If they are also straight, cisgendered, protestant, able-bodied and do not suffer from mental illness, they would seem to have no way to report their experiences on the survey. And yet, surely, they might experience harassment. Most commonly, they will experience verbal (and at times physical) harassment by professional rivals, for whom they are competing for publication, promotion and research funding. This basis for harassment has been completely excluded from the Workplace Climate Survey. The questionnaire did not even provide a generic "other" characteristic in which to report harassment.

Now, if the hypotheses tested had confined themselves to race- and gender-based forms of harassment, this wouldn't be such a big problem. Except that the conclusion that women and people of color experience more gender- and race-based harassment than white men is a bit underwhelming. But Clancy et al. claim to have found support for hypothesis 1, and they are promoting the result widely as suggesting that women, and women of color, experience more harassment than men full stop. As it turns out, this conclusion emerges from a measurement instrument that excluded the most typical experiences of harassment among white men.

I'm going to take some time to think about the consequences of this. But my initial reaction is that it completely undermines the validity of the survey, given the hypotheses it claims to be testing. Also, it raises the interesting question of whether women and minorities experience harassment based on professional rivalry (they would also not be able to report it). And that, finally, raises a question that concerns me greatly: is it possible that women and minorities are getting the axis of their harassment wrong? Is it possible that they experience harassment that is really grounded in ordinary competition as grounded in racism and sexism? If so, surveys like this are distracting them from the fight they should be fighting; and this, ultimately, will be to their disadvantage.

Comments are welcome, as always.

Monday, July 24, 2017

Hypothesis 0

According to Clancy et al., the CSWA Workplace Climate Survey was designed to test four hypotheses:

1. Female respondents will report more verbal and physical harassment than men.
2. Respondents of color will report more verbal and physical harassment than white respondents.
3. Trainees will report more verbal and physical harassment than those scientists of a higher rank.
4. Women of color experience double jeopardy where they are especially at risk for verbal and physical harassment compared to white women or men of color.

While they do not make the null hypothesis explicit, it is clear that they are testing these against their simple negation. That is, the null to be rejected is that men and women, whites and non-whites, trainees and seniors, will experience equal amounts of verbal and physical harassment.

This null, I want to argue, is not very compelling. To see why, consider what these hypotheses would look like if they were not about the sample (i.e., the "respondents") but about the population (i.e., "astronomers"). Here we need to add some information to capture the wording of the survey that the respondents were responding to, fixing both the situation and the basis of the harassment (I have marked the additions with square brackets):

1. Female astronomers experience more verbal and physical harassment [from other astronomers based on their gender] than male astronomers.
2. Astronomers of color experience more verbal and physical harassment [from other astronomers based on their gender or race] than white astronomers.
3. Astronomers in training experience more verbal and physical harassment [from other astronomers based on their gender or race] than astronomers of a higher rank.
4. Female astronomers of color experience double jeopardy where they are especially at risk for verbal and physical harassment [from other astronomers based on their gender or race] compared to white female or male astronomers.

This, it should be noted, is roughly how the results have been presented to the public by press releases and news coverage. But the importance of turning the hypotheses into statements about astronomers lies in pointing to what we can call the Astronomy Effect on the likelihood of experiencing verbal and physical harassment. Does becoming an astronomer expose you to a particular risk of such harassment? Consider four hypotheses about the general population:

1. Women experience more gender-based verbal and physical harassment than men.
2. People of color experience more race-based verbal and physical harassment than white people.
3. Trainees experience more race- and gender-based verbal and physical harassment than people of a higher rank.
4. Women of color experience double jeopardy where they are especially at risk for gender- or race-based verbal and physical harassment compared to white women and men.

It would be surprising if these statements weren't true in the general population, right? So the implicit null of the CSWA survey was that joining the astronomy community would completely eradicate race- and gender-based differences. Why would we begin there? Why would we not begin with the reasonable hypothesis that becoming an astronomer would reduce your exposure to the risk of harassment, and rising in the ranks of astronomy would reduce it still further? We might then be distraught, as Clancy and Richey have said they were, to learn that it actually has no effect on your chances of being harassed, or, indeed, actually increases it.

Such an effect (which I think is unlikely, but possible) could be explained by the continuing, if shrinking, gender and race disparities in astronomy. If becoming an astronomer increases her exposure to the company of men, she might find her exposure to harassment increase as well. Likewise, by deciding to join the community of astronomers, people of color will generally be exposing themselves to the company of white people. If their baseline experience has been among people of color, they might well experience a change similar to moving from a black to a white community.

My point isn't to argue for or defend any level of harassment. My point is just that a null hypothesis does not have to have a zero value. It just has to suggest a zero effect on the dependent variable of the hypothesized force, which, here, as far as I can tell, is the particular "hostility" that the astronomical community allegedly directs toward women and minorities. Even if astronomers are not particularly hostile to women and minorities, I want to say, we would expect them to experience more gender- and race-based harassment than white men in the same field. It's just that we'd expect the overall level of that harassment to go down when they are at work.

Finally, I would assume that the baseline level of race-based and gender-based harassment that white men face (before going into astronomy) is quite low (almost by definition, you'll note) and might drop to virtually zero in the protected space of the observatory. In a small sample, it's hard to predict what this will do for the statistical significance of the disparity between genders and races. Indeed, I suspect (but I will deal with this in latter post) that the low p-values (often < 0.001) stem from the fact that we are comparing groups that (again, essentially by definition) don't face the relevant form of harassment with those who actually might.* All this seems completely obvious to me and I'm at a loss to understand how an esteemed scientific journal like JGR:Planets came to publish the result in the form it did. Again, I'm happy to hear from people who think I'm wrong. Quantitative analysis isn't my strong suit either.

________
*Think, for example, of what would happen if we compared unqualified harassment of men and women in the general population. I think we would find that men get pushed around, as it were, as often as women. They just don't experience this as having anything to do with being men. This survey did not give white men much room to report these experiences.

Was the CSWA survey confined to recent experiences?

As I mentioned in my last post, a commenter on my first post on its publication pointed out that it does not seem true that the CSWA Workplace Climate Survey asked respondents to confine themselves to the years 2011-2015 when answering the questions. Clancy et al. are, it should be noted, adamant on this score:

At this time, we want to remind the reader that the findings of this study cannot be attributed to events from long ago: respondents were asked to only report experiences they had had in their current career position over the last 5 years. The events and experiences reported in this survey happened between 2011 and 2015. (p. 11)

But here is what respondents were told in the cover letter to the survey:

The survey is designed to request information during the respondent's current position and previous position (if the respondent has changed positions within the last five years). [...]

Directions: Please respond to Section 1 regarding your personal experiences in your current position. If you have changed career positions in the past five years, then please also complete Section 2 regarding your personal experiences in your previous position.

This does not seem to me to confine responses to the years 2011-2015. (While it's ultimately a moot point, do notice that "the last 5 years" would cover 2010-2014, not 2011-2015, since the survey was administered in early 2015.) If a respondent had been working somewhere for, say, 15 years they would reasonably interpret this as covering the whole time there. If they have changed position within five years, they are being asked to answer also for the entire time of their previous position. In other words, they are being asked to think back at least five years, not at most.

This strikes me as a serious issue, especially given Clancy et al.'s "reminder" to the reader. If the frequency of experiences is really spread over, say, 20 years and the analysis assumes they are concentrated within 5 years, this will strongly distort the result. If anyone knows what I'm getting wrong here, do let me know in the comments.

Friday, July 21, 2017

The CSWA Survey in Plain Language

It's going to take a bit of work to properly critique the the CSWA Workplace Climate Survey as published in the Journal of Geophysical Research. In this post, I want to take a sentence-by-sentence look at the plain language summary. In later posts, I will elaborate on each point by way of a critique of the substance of the paper. I have already pointed out that the press releases spin the survey in ways that the paper itself belies. As it turns out, this spin is also present in the summary. Indeed, it is also present in the abstract, but slightly more subtly. Like I say, I will go through it one sentence at a time.

Women generally, and women of color specifically, have reported hostile workplace experiences in astronomy and related fields for some time.

This is, of course, true. As Kate Clancy has noted elsewhere, it's true of every field of human endeavor, and it is true of all races and genders. Everyone has experienced hostility at work. Work is done by humans in human environments and hostility is a human capacity. Indeed, humans are "capable" of hostility in both directions: they can dish it out and they can take. In short, the first sentence is a truism. The second sentence gestures at something less trivial.

However, little is known of the extent to which individuals in these disciplines experience inappropriate remarks, harassment, and assault.

It's true that this specific question hasn't been studied directly in astronomy. But there is some evidence to suggest that astronomy and related fields are not especially hostile places, specifically to women. (Women of color are, as is often noted, very underrepresented in astronomy and do seem to get lost in such studies.) One study found that women don't think about leaving the discipline more frequently than men; another found that, while they are 1.64 times more likely to have negative experiences than men, the average level of hostility was on the order of occasionally hearing a sexist joke. But it must be granted that the extent to which individuals have particular experiences is not well understood. The next sentence implies that this study will do something about this gap in our knowledge.

We conducted an internet-based survey of the workplace experiences of 474 astronomers and planetary scientists between 2011 and 2015.*

What they here imply is misleading since the paper explicitly states that "these data cannot provide a direct assessment of prevalence". That is, their "plain language summary", presumably intended for the non-expert (or journalist) gets the reader to think that they have done a study to gain the knowledge we lack, even though the authors are well aware that the study was precisely not designed to gain that knowledge, i.e., knowledge of the "the extent" (prevalence) of hostility in astronomy. That is of course also why they don't present general findings of prevalence, only comparisons of groups within the sample:

In this sample, in nearly every significant finding, women of color experienced the highest rates of negative workplace experiences, including harassment and assault.

This may seem like a quibble, but it is worth noting: the survey asked about "verbal harassment" and "physical harassment", not harassment and assault. They don't actually know the extent to which people in their sample were reporting assaults, except on a definition of assault in which any unwanted touching constitutes an assault (I'll discuss this in another post). Note that we are not told whether they experienced these things at a generally high or low rate, mainly because the study sets no threshold to make such judgments. The next sentence does report some alarming levels of hostility, however.

Further, women of color reported feeling unsafe in the workplace as a result of their gender or sex 40% of the time, and as a result of their race 28% of the time.

This sentence is simply a misinterpretation of the relevant result. It distorts and exaggerates their actual finding, as stated in the abstract: "40% of women of color reported feeling unsafe in the workplace as a result of their gender or sex, and 28% of women of color reported feeling unsafe as a result of their race." That is, it is not true that respondents felt unsafe 40% of the time; rather, 40% of respondents felt unsafe some of the time—or, more accurately, had felt unsafe at some time in the past. Indeed, they were specifically asked whether they had "ever felt unsafe" in their current position (see also footnote*). Answering "yes" here says nothing about how long or how often they felt unsafe. If 40% of respondents had ever felt unsafe, surely the population doesn't feel unsafe 40% of the time.

Finally, 18% of women of color, and 12% of white women, skipped professional events because they did not feel safe attending, identifying a significant loss of career opportunities due to a hostile climate.

As far as I can tell, this is a completely accurate summary of the result. I have said before that this is an important one, since it shows that there is a difference between feeling unsafe and doing something about it. It has been established in other studies that women feel more unsafe than men (even when both sexes feel very safe) and it stands to reason that this would translate into more absenteeism among women. It needs to be stressed that the survey found that only 9% of respondents reported "physical harassment", i.e., arguably an actual violation of personal safety. This suggests that women generally feel less safe than they are. This isn't a particularly surprising result, especially in a climate where women are told (by scientists and politicians) that harassment is "rampant". This study, of course, is one of the things that might be making women feel unsafe. Indeed, the authors say women are unsafe explicitly:

Our results suggest that certain community members may be at additional risk of hostile workplace experiences due to their gender, race, or both.

My standing objection to this way of putting it is that it does not account for the fact that "certain community members" would be in other environments if they were not doing astronomy. The authors don't give us any way to decide what the comparative ("additional") risk of hostile work experiences would be if they went into banking, politics or even another scientific discipline. As I said at the outset, there is a risk of hostility in any human environment. If a woman of color took away from this study that she best stay out of astronomy and choose another line of work then that would be a reasonable, if tragic, conclusion to draw from the "plain language" of this summary for the public. But, since the study itself eschews any claims about prevalence, it is not actually a reasonable conclusion to draw from the survey itself. I think that is a serious problem in the communication of this result to the public. It is not only astronomers that should take issue with this; the whole ear of the public is rankly abused.

Like I say, I will be looking at the paper more closely to support these various points of criticism and raise a few more in future posts. As is my custom, I will also be asking the authors for comment. To my knowledge, there has so far not been any serious criticism of the study in the press or the science blogs. It would be to Clancy's credit if she engaged with at least one critic as part of the discussion she so insists it is important to have. But I am not holding my breath.

____________
*I will cover this in a separate post, but as a commenter on my last post pointed out, it does not seem true that the survey asked respondents to confine themselves to the years 2011-2015. As I read the questionnaire, the respondents might well have thought they were being asked "Have you ever experienced...?" I believe that the authors thought they had limited the responses in this way. But I don't think the respondents would generally understand it as limited to five years.

Monday, July 10, 2017

CSWA Study Published

The CSWA Workplace Climate Survey has finally been published. I've been following it since early 2016 and, since its authors wouldn't answer any of my questions, I've been impatiently waiting for the report. Well, here it is:

Clancy, K. B. H., K. M. N. Lee, E. M. Rodgers, and C. Richey (2017), Double jeopardy in astronomy and planetary science: Women of color face greater risks of gendered and racial harassment, J. Geophys. Res. Planets, 122, doi:10.1002/2017JE005256.

The PR push appears to be well-organized. But I notice that the subheadings of the UIL press release and the EOS interview both get the results wrong. UIL says the survey found "widespread bias"; EOS says it "reveal[s] the prevalence" of harassment in astronomy. The paper, however, says that "these data cannot provide a direct assessment of prevalence”, noting that "prevalence studies are exceedingly uncommon in research of this nature," which is true. (To their credit, the AGU and AAS get this right in their joint press release.)

In lieu of determining prevalence, the authors say* they tested four hypotheses, which I can't distinguish from the null or prior I would construct in such a study:

1. Female respondents will report more verbal and physical harassment than men.

2. Respondents of color will report more verbal and physical harassment than white respondents.

3. Trainees will report more verbal and physical harassment than those scientists of a higher rank.

4. Women of color experience double jeopardy where they are especially at risk for verbal and physical harassment compared to white women or men of color.

This isn't something that stands in need of empirical evidence. What we want to know is how astronomy compares with other fields of human endeavor. That is, we want to know whether astronomy provides a more or (as I suspect) less hostile environment for women of color than other fields. Indeed, we'd probably just be testing whether astronomy is generally less hostile for humans than other contexts. It's not going to ensure your safety 100% but it's probably a pretty good choice if it's hostility you're trying to avoid. Especially, indeed, if you're a woman of color.

Finally, it looks like a great deal is going to be made of the finding that "88% of respondents reported hearing negative language from peers". But this number does not distinguish between reports of hearing this sort of language "rarely", "sometimes" or "often". That is, the great majority of respondents reported that it is heard rarely or never. I'm going to look more closely at this in the days to come. (It actually seems a bit more complicated to disaggregate this particular result than the preliminary ones.) I just wanted to get my initial reaction out there now to encourage people to be critical in their reception of this survey. After all, the greatest respect you can pay to a scientific result is to critique it.

_________
*I'm suspicious about whether these hypotheses had been stated explicitly before the survey was designed. They were not part of Christina Richey's 2015 and 2016 presentations of the data. If I'm right about this, there are some pretty serious "degrees of freedom" in their framing. Since the authors do emphasize their p-values, there's a risk that these hypotheses are a result of p-hacking their data.