Misleading Statistics -- "Women Rising in Management" -- sort of

We first conceived of this feature of DataLine out of a growing frustration at the scarcity of good, hard data about the glass ceiling, and, to a lesser extent, dismay at the wide promulgation of some less than rigorous studies with potentially misleading conclusions.

One of the rules we have established for ourselves is to ask, when percentages are stated, or a change over time is described in terms of a percentage, a percentage of what? This is especially important when looking at small samples, which unfortunately we are still looking at when we talk about women in senior management. For example, a 300% increase sounds very big, but a 300% increase in the number of women CEO's of major companies is in fact not a very big jump -- it means there would be three instead of one. Another guideline is to press for the underlying data when an organization is reporting about itself. This does not mean assuming that an organization will misrepresent data, but rather being alert to the tendency to cast the information in a flattering light.

That is why, when one of the largest retained search firms in the world began actively promoting the results of a "study" it had done under the headline "Women Rising in Management", we were curious. Both a current and former partner in the firm told us they didn't really believe the conclusions, and assumed that the samples were very small, or the time periods were chosen to produce the most favorable results. Here is the version of the "study" that the firm published in major metropolitan newspapers.

"placements of women in senior-level jobs more than tripled between 1981 and 1991. Only 5 percent of [the firm's] senior-level placements were women in 1981, but the number climbed to 16 percent by last year ... In 1981, no women were placed in positions higher than vice president, but by last year, 21 percent of the women [placed by the firm] were placed in positions of senior vice president and above. About 11.5 percent were placed in the position of chief executive officer."

There is actually some good news embedded in this information, but it's not as broad or far-reaching as most readers might assume. The total number of annual placements for the firm, worldwide, is "about 1,000", according to the company spokesperson. So, in 1991, about 160 women (and 840 men) were chosen by the firm's clients. When we tried to determine how "senior-level" was defined, we were given several answers, and finally, with great exasperation, the spokesperson blurted, "Well, all of our placements are senior." When we asked then if their assertion that placements of senior-level women had risen to 16% really meant that placements of all women had risen to 16% over the ten-year period, the answer was yes, (we think). We still don't know why they didn't just report the changes as changes in all of the placements; we didn't get an answer to that question.

"About 11.5 percent were placed in the position of chief executive officer." When we tried to unravel this statement, we got sent to the opposite coast, to the "partner who really is familiar with this". That partner sent us back to the headquarters office to the spokesperson, who, it turned it, was the person who had actually compiled the data. At this point, the vagueness of the numbers started to make sense, although we are still not sure whether the spokesperson was in fact a statistician, her the numbers had been compiled and interpreted by a representative of the pr department.

"This was not a scientific study," the spokesperson now cautioned us multiple times. When we asked what exactly was meant by "position of chief executive officer", we were told that this included heads of hospitals and health care organizations, as well as heads of colleges or non-profits; in fact, the "majority" of placements of women "in the position of CEO" were in these areas. We couldn't get anything more specific than "majority", except for the additional statement that the CEO-level placements of women -- 18 or 19 women worldwide, to be more precise -- were "overwhelmingly in the health care industry or not-for-profit sector, and quite a few were in education."

We have no way of knowing how the overall pool changed during the ten-year span of this "study". Still, if no women were placed in positions higher than vice president in 1981, 18 or 19 women heads of hospitals, hmo's, etc., by 1991 is some kind of improvement. And then again, this is one of the top four retained search firms in the world. This data was produced, the spokesperson exclaimed, because they had seen such a dramatic improvement in their placements of women. Maybe from their perspective, these achievements really are tremendous. The results have been so encouraging, according to the Vice President of Public Affairs, that the firm has undertaken a larger study with a major university. He promised to keep us posted.