For you science doer types, any thoughts on peer review?

[landpoke]

I found this quote when looking for a definition of "peer reviewed." Any comments as to the accuracy of this assessment?

"The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."

Richard Horton, Editor, The Lancet

I'm a bit perturbed at the moment by the use of "peer reviewed" as a hallmark of infallibility. Especially when the data, to this non-science-doer-guy's eyes, very clearly indicates that the conclusions reached by a researcher are incorrect.

As an aside, several science-doer types share my conclusions about the data and paper in question. So it's not just the ranting of an unschooled kook.

[myboynoah]

Originally Posted by landpoke

I found this quote when looking for a definition of "peer reviewed." Any comments as to the accuracy of this assessment?

"The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."

Richard Horton, Editor, The Lancet

I'm a bit perturbed at the moment by the use of "peer reviewed" as a hallmark of infallibility. Especially when the data, to this non-science-doer-guy's eyes, very clearly indicates that the conclusions reached by a researcher are incorrect.

As an aside, several science-doer types share my conclusions about the data and paper in question. So it's not just the ranting of an unschooled kook.

Science abides.

[UtahDan]

I have faith that science, collectively, gets things right over time though they may err in the short term. Kind of like how I feel about the brethren.

[creekster]

Are you asking if we are aware that peer review can be a biased and imperfect process? Or are you asking if we think it is better or worse than no peer review at all?

[landpoke]

Originally Posted by creekster

Are you asking if we are aware that peer review can be a biased and imperfect process? Or are you asking if we think it is better or worse than no peer review at all?

I suppose the first. I've grown tired of the press, through the workings of certain groups, holding out peer review as an indication of infallibility.

I am not suggesting that peer review needs to be done away with, rather just recognized for what it is.

[Indy Coug]

Originally Posted by creekster

Are you asking if we are aware that peer review can be a biased and imperfect process? Or are you asking if we think it is better or worse than no peer review at all?

I think he's asking if it's reliable enough on average to put any stock in it. IMO, peer review isn't particularly rigorous if the findings being reviewed are reinforcing what is already accepted as conventional wisdom. Thus, it sometimes can be a vicious circle of self-fulfilling prophecy.

[creekster]

Originally Posted by Indy Coug

I think he's asking if it's reliable enough on average to put any stock in it. IMO, peer review isn't particularly rigorous if the findings being reviewed are reinforcing what is already accepted as conventional wisdom. Thus, it sometimes can be a vicious circle of self-fulfilling prophecy.

Yea, I was trying to be insightful with rhetoricxal questions but I can see why you were fooled. Peer review isn't perfect, but it's about as good as we got, I think.

[landpoke]

Originally Posted by landpoke

I am not suggesting that peer review needs to be done away with, rather just recognized for what it is.

And that personal bias can and does enter into the process.

[creekster]

Originally Posted by landpoke

And that personal bias can and does enter into the process.

SUre it does. If a person is involved, its personal and there is bias.

[landpoke]

You understand that and I understand that, but I think the public perception is that peer review is a sort of holy process above corruption whereby we find the absolute truth (I might be hyperbolizing a bit here. And yes it's a word, look it up.)

Every time the topic and study which have so irritated me are mentioned in the press the term "peer reviewed" is tacked on as some sort of blessing from the Pope. I understand that "peer reviewed" is at least a standard of sorts, but it doesn't guarantee that the conclusions of a particular study are correct. Only that the researcher hasn't simply pulled stuff out of his ass.