Social Science Screwup

[OlongJohnson]

I remember sitting in a Sociology class as an undergrad.

Me, too. While pursuing my engineering degree, I was able to satisfy that requirement with a course for which the guest speakers included Phil Hill and James Schefter.

[RevolverRob]

And then there's this: https://en.wikipedia.org/wiki/Replication_crisis

It's true that there is an issue with reproducibility in science, to some degree. However, the doomsday scenarios are a bit overblown. The number one reason why experiments aren't reproducible is due to lack of adequate description in the final published form of the work or due to lack of proper preparation and maintenance of datasets.

Example: I just received a review back on a manuscript, that isn't really that long (<20 manuscript pages, which is about 9 formatted pages). One of the reviewers wrote,

I have an issue with the relative emphasis you give to different parts of the text. On a percentage basis of the word count your sections have the following relative lengths: Introduction (13.56%), Material & Methods (45.56%), Results (13.14%), Discussion (25.35%), Conclusions (2.29%). Based on these counts your manuscript would appear to be a methods paper. I understand that is probably not what you intended, You should reconsider the organization and emphasis structure of your manuscript.

I authored the response and said,

We respect the opinion of the reviewer, but do not share it. Our perspective is that materials and methods is not only often the longest portion of scientific publications, it is among the most critical for allowing assessment and reproduction of our work. The reviewer certainly understands that considerable work is undertaken to conduct even “simple” projects. We have made efforts to reduce and/or remove redundant sections to simplify the text and reduce its bulk, while retaining the ability for future researchers to both access what was done and reproduce it.

In other words - I had a reviewer tell me to shorten the paper, because it was obvious to them what we had done. I mostly refused, both on principle and necessity, to cut significant chunks of a manuscript to shorten its overall length. Unfortunately, my response isn't necessarily typical, I came up in an academic tradition that does not favor brevity over clarity. Many are on the other end of the spectrum.

The reproducibility crisis is also a reflection of the places people publish their materials. In the highest impact, most visible, journals space is at a premium and methods sections are often reduced to mere snippets of what they should be. I submitted a paper that ultimately was not published in a "Big 3" journal last year, I had a methods section that had to be trimmed to <500 words. I had a "supplemental methods" section, contained in a separate document that is only published online, not in print, that had a methods section that was nearly 5,000 words long. An order of magnitude had to be cut from what would have been the published version. If someone read only the journal published version and attempted to reproduce my results that are not likely to get far.

___

Dataset management is another issue. Not even 10 years ago, there did not really exist good universal databases and formats for sharing large amounts of scientific data. Today with GitHub, Open Science Framework, etc. It's now possible to host large and complex datasets that are maintained into perpetuity in association with published works. 15 years ago, this was basically exclusively the domain of museums and we did a poor job of maintaining digital data. Now we are so much better at this, it is no longer an excuse to not take care of and maintain data.

None-the-less it happens. I once contacted an author for copies of a dataset for my master's thesis, the author responded with, "Sorry dude, it's been 10 years since that paper was published and I've moved five times. I lost those data long ago." That instantly makes the results of that particular project not reproducible. I remember my Master's Thesis advisor's reaction when I told him that, "That's bullshit. Do not EVER do that. Publish everything all together, whenever possible." And I've endeavored to do precisely that since then.

[cheby]

This was published in a peer reviewed magazine a couple years ago:
https://www.skeptic.com/downloads/co...17.1330439.pdf

More on this can be found here:
https://www.skeptic.com/reading_room...ender-studies/

" That’s how we began. We used this preposterous sentence to open a “paper” consisting of 3,000 words of utter nonsense posing as academic scholarship. Then a peer-reviewed academic journal in the social sciences accepted and published it"

[Glenn E. Meyer]

The Chronicle of Higher Education documents this sort of hilarity all the time. As long as tenure is based on number of papers and some fields have defined acceptable paradigms, you get this kind of thing.

[OlongJohnson]

As Feynman called it, "Cargo Cult Science".

Interesting background on that phrase. Worth googling.

[willie]

You see, it's like this. A man is built different than a woman.

[Robinson]

The whole concept of so-called "gender studies" is annoying.

[blues]

The whole concept of so-called "gender studies" is annoying.

You think that's annoying, what about waiting for a bathroom in Charlotte?

[willie]

Maybe it's like the restrooms in a place I visited in Austin many years ago. I could not figure out which was the men's room, and upon asking the not too bright security guard, he told me that they were bisexual restrooms.

[5pins]

Maybe it's like the restrooms in a place I visited in Austin many years ago. I could not figure out which was the men's room, and upon asking the not too bright security guard, he told me that they were bisexual restrooms.

So what bathroom did heterosexuals use?