Reforming Statistics in Psychology

David BeckerBy David Becker

Are you fed up with those unsightly wrinkles? Have you tried everything to get rid of them, but nothing seems to work? Well, throw away your anti-aging creams and forget about those harmful Botox injections! Did you know that you can reverse the aging process just by listening to music? Scientists have found that listening to songs about growing old will decrease your age by one-and-a-half years. Amazing!

Not only is this finding amazing, it’s also complete lie. The study itself was real, but the researchers (Simmons, Nelson, & Simonsohn, 2016) weren’t actually interested in testing the effects of music on age. Their true intent was to shine a light on researchers’ overreliance on and misuse of significance testing by generating a statistically significant result that has no bearing on reality whatsoever.

Many people believe that a statistically significant result affirms that a study’s findings are real, but that’s not what statistical significance measures. Statistical significance measures the probability of error in a study’s data. It’s indicated by the p value, which determines the likelihood of rejecting the null hypothesis. The null hypothesis essentially opposes the hypothesis that the researchers are testing. In other words, if the researchers hypothesize that a new therapeutic approach will have a meaningful effect or will be superior to an established approach, the null hypothesis refers to the possibility that there will be no meaningful effect or no differences between the two approaches. A lower p value (.05 is the most commonly used cutoff) suggests a higher likelihood that the null hypothesis is false. However, a high probability of rejecting the null hypothesis doesn’t make the researcher’s hypothesis true; it simply reflects greater confidence in the data’s validity (researchers sometimes refer to a confidence level above 95%, which equates to a p value below .05). Even so, it’s very easy to play with the data until you achieve statistical significance.

Simmons and colleagues illustrated in their study how easy it is to get a false positive by manipulating what they refer to as “researcher degrees of freedom.” These are the decisions that researchers make about how to collect, analyze, and report data—decisions that include when to stop collecting data, which variables to analyze, and which subsets of data to report. Many researchers make these decisions throughout the research process. For instance, researchers may decide to stop collecting data as soon as they have achieved significant result. Or they might change their hypothesis based on the data. Practices like this are often referred to as “p hacking,” the idea being that researchers will massage the data until they come up with a low-enough p value. Simmons et al. estimate that abusing researcher degrees of freedom can lead to a false positive rate as high as 61% in a given study, which they admit might be a conservative figure.

Human Mind series. Backdrop of brain, human outlines and fractal elements on the subject of technology, science, education and human mindIt’s important to note that most researchers who p hack aren’t purposefully trying to be deceptive. Many of them don’t fully grasp the potential consequences of what seem like minor decisions. In a recent example, psychologist Dana Carney rejected a psychological phenomenon she helped popularize—power poses—and essentially admitted that she and her fellow researchers unknowingly abused researcher degrees of freedom to achieve a statistically significant result that had no real-world significance (Peters, 2016).

Even when p hacking is unintentional, social psychologist Harris Cooper (2016) argues that researchers have an ethical responsibility to the scientific community and to the greater public good that obliges them to be aware of the decisions they make, and their impact. He advises researchers to make decisions about data collection as early as possible—primarily in the planning phase before the study has even begun—and to stick with them throughout the study. If the researchers change course, they need to fully report their new decisions so that anyone scrutinizing the study’s findings and methodology will understand the full context. As an example, if researchers decide to create new data sets based on the original data—a practice that Cooper advises should be rarely used—the changes need to be explicitly catalogued with clear explanations. And under no circumstances should researchers edit or omit any of the original data, regardless of which data sets they choose to analyze.

In addition to carefully documenting their decisions, researchers need to better understand the true purpose of p values and how to properly use them. Clinical psychologist Rex Kline (2013) notes in his book Beyond Significance Testing: Statistics Reform in the Behavioral Sciences, now in its second edition, that cognitive errors surrounding significance testing are so common that they can be considered a form of “trained incapacity” (p. 10). Even statistics instructors don’t even fully understand p values, according to Kline, which feeds the “ongoing cycle of misinformation” (p. 10). The problem is so pervasive that last year the American Statistical Association (ASA) released a policy statement laying out six principles to help the scientific community better understand and apply p values. This marked the first time in its 178-year history that the ASA decided to take an official position on statistical practices.

The ASA also suggested other approaches that researchers might use in addition to or instead of significance testing, including one of the most popular alternatives proposed by advocates of statistics reform: Bayesian analysis. The premise behind Bayesian statistics is fairly simple: “Begin with an estimate of the probability that any claim, belief, [or] hypothesis is true, then look at any new data and update the probability given the new data” (Novella, 2016, para. 2).

Kline (2013, Chapter 10) supports the Bayesian approach because it reflects the fundamental tenets of science and critical thinking. Namely, extraordinary claims that seem implausible given what we know about the universe (e.g., listening to music will make you younger) need to be supported by extraordinary evidence. Kline further argues that Bayesian analysis allows us to compare the competing hypotheses of researchers who have differing interpretations about our existing body of knowledge and are studying the same subject from alternate perspectives. Being able to examine new and divergent findings against our current understanding of the world encourages scientists to reevaluate the likelihood of existing hypotheses, which is fundamental to science’s self-critical and self-correcting nature.

Significant

Courtesy of xkcd

Simmons, Nelson, and Simonsohn (2016), however, are skeptical of Bayesian statistics as an alternative to significance testing. They contend that it gives researchers even more opportunities to manipulate data, in addition to those provided by significance testing. Simonsohn (2015) also argues that the default Bayesian test in psychology is biased against small effects.

If there’s one clear takeaway from this controversy, it’s that there isn’t one perfect alternative to significance testing. In fact, as the ASA points out in their policy statement, significance testing can be useful, so long as it’s properly applied. Therefore, completely avoiding p values doesn’t seem like an ideal, catch-all solution. Rather, scientists must experiment with a variety of solutions to see how best to test the validity of their findings.

In the meantime, it pays to be skeptical. The popular media tends to simplify, overhype, and misinterpret the findings of a single study—much like I did at the beginning of this post—without accounting for the complexities of scientific research. It can be difficult for those of us who are not scientific experts to figure out what to believe and what not to believe, especially when our mental filters are already overwhelmed by the constant deluge of information that floods over us every day.

But look on the bright side: At least you can be pretty confident that you won’t become younger and younger until you vanish from existence just by listening to The Beatles’ “When I’m Sixty-Four” on repeat.

References

Cooper, H. (2016). Ethical choices in research: Managing data, writing reports, and publishing results in the social sciences. https://doi.org/10.1037/14859-000

Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). https://doi.org/10.1037/14136-000

Novella, S. (2016, January 8). What is Bayes theorem? [Blog post]. Retrieved from http://theness.com/neurologicablog/index.php/what-is-bayes-theorem/

Peters, M. (2016, October 1). ‘Power poses’ co-author: ‘I do not believe the effects are real.’ Retrieved from http://www.npr.org/2016/10/01/496093672/power-poses-co-author-i-do-not-believe-the-effects-are-real

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2016). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. In A. E. Kazdin (Ed.), Methodological issues and strategies in clinical research (4th ed., pp. 547–555). https://doi.org/10.1037/14805-033

Simonsohn, U. (2015, April 9). The default Bayesian test is prejudiced against small effects [Blog post]. Retrieved from http://datacolada.org/35

March for Science

apa-march4science-banner

 “APA is proud to be an official partner of the March for Science, set to take place on Saturday April 22, 2017, in Washington D.C. We encourage all psychologists, psychology students and their allies to join this broad, nonpartisan effort to support scientific research and the use of scientific evidence for the public good.” See more on APA’s stance and plans for the Science March and beyond here.

 

With our headquarters in downtown DC, many of us here at the APA and APA Books are excited to be able to participate in this historic event. We asked around for thoughts on the march, and why people felt compelled to attend.

The opinions expressed below are those of the individuals and should not be taken to represent the official views or policies of the American Psychological Association.

“I march because I believe science is key to promoting unity. Being much more than just a set of facts, science is a way of thinking that encourages us to look beyond our ideologies and preconceived notions about ourselves and the world around us. It is through science that we can understand our tendencies toward tribalism, an “us vs. them” mentality that can limit our worldviews. My hope is that science can also help us see beyond the boundaries of our various tribes—whether they are defined by politics, religion, race, nationality, or gender—and unite us as members of the one tribe that encapsulates us all: the human tribe.”—David Becker, Development Editor, APA Books

“[I march] to celebrate why science matters and support scientists in their message that evidence-based facts are vital to inform policy and the general public. I hope the march can encourage our leaders and all Americans to value the importance of scientific information as it affects all of us, regardless of political party.”—Marla Koenigsknecht, Marketing and Publicity Specialist, APA Books

“I plan to participate in the March for Science on April 22. I fully support and defend the dissemination of alternative opinions or interpretations. But as Senator Moynihan told us, we are not entitled to our own facts. Ignoring a problem and demonizing those that raise legitimate concerns are losing propositions. We cannot make America great without education, research, funding, and a respect for truth.”—Chris Kelaher, Acquisitions Editor, APA Books

“I am marching because facts matter.”—Beth Hatch, Development Editor, APA Books

“I’m marching because climate change is one of the most pressing challenges of our time. I believe the only way to meet that challenge is by funding research, listening to scientists, and forming evidence based policies.”—Sarah Fell, Editor, Magination Press

“I grew up with science; I spent days off school and take-your-child-to-work days filling pipette tip trays in my mom’s lab. “Bringing her work home with her” sometimes meant tubes of fruit flies in the dining room. But I don’t march because of my personal connection to science; I march because science is important no matter who you are, and because science should inform policy, not the other way around.”—Katie ten Hagen, Editor, Magination Press

“The idea that listening to scientific research will somehow harm us is a disconcerting one, and not just research that applies to mental health and psychology. I grew up in a very rural area and I’ve seen first-hand the damage that can be done by a disregard for the environment. Shoving science under the rug doesn’t make the facts untrue and I’m concerned that too many people in positions of power are trying to because they see the facts as inconvenient. And so I want to join in to show that there are people who care about these things and that our voices deserve to be a part of the conversation. I’m hoping that it will open up a new dialogue about science that is separate from our political persuasions. I want people to think about the ways that science affects their everyday lives and realize that, as a country, our decision-making should be grounded in facts. We should be funding research and then listening to what the research tells us.”—Jessica Jeffers, Assistant Marketing Manager, APA Books

science-march-title-image-

 

From APA Members:
“The science of psychology has been fractured for more than a quarter century.  I march to realign psychology with what should be its common mission, and to elevate it to its rightful place among all sciences.”—Wallace E. Dixon, Jr., Ph.D.

“I am going to the DC March for Science because science is under attack, so Scientists must act. Fake news should be replaced by science news. You can’t make America Great without science.”—Kathleen Y. Haaland, Ph.D., ABPP-CN, Professor

“I am marching because science is fundamentally a search for truth, and truth has been threatened.  Science not only helps keep America great, it makes America—and the world—become better.”—Dr. Paula P. Schnurr

Nancy Schlossberg: On Aging Gracefully

This is the latest in a series of interviews with APA Books authors. Here Andrew Gifford, Development Editor at APA Books, interviews Nancy Schlossberg, a well-known authority on aging and life after retirement.  Nancy will be speaking at Politics and Prose bookstore in Washington, DC (5015 Connecticut Ave NW), this Sunday, April 23, at 1:00pm to kick off the release of her new book, Too Young to Be Old: Love, Learn, Work, and Play as You Age. See more about the event here!

Note: The opinions expressed in this interview are those of the authors and should not be taken to represent the official views or policies of the American Psychological Association.

--Photo by Rod Millington

–Photo by Rod Millington

Nancy K. Schlossberg is an expert in the areas of adult transitions, retirement, career development, adults as learners, and intergenerational relationships. Past President of the National Career Development Association, Co-President of a consulting group TransitionWorks, she is a Professor Emerita, Department of Counseling and Personnel Services, College of Education at the University of Maryland.

Dr. Schlossberg has delivered more than 100 keynote addresses, and has been quoted in the cover story in USA Today, the New York Times, Wall Street Journal, Sarasota Herald Tribune, Reader’s Digest, Family Circle, Better Homes and Gardens, U.S. News and World, Consumer Reports.  She has appeared on PBS In the Prime, Derek McGinty’s national talk radio show, CBS This Morning, CBS evening news and is featured in a 90-minute PBS Pledge Special June, 2007, “Retire Smart, Retire Happy.”

AG: In many ways, your latest book feels like part of an unofficial trilogy, starting with the smash hit Retire Smart, Retire Happy which provided a primer on how to adjust to retirement. It was followed by Revitalizing Retirement, which discussed how retirees could reshape their identity and play a vital role in their community.  Too Young to Be Old takes the lessons from the first two books and really emphasizes the idea that retirement is not an ending but a beginning. In it, you discuss relationships, finding your place, embracing adventure, and aging well. Could you tell us a little bit about your own experiences as an author, a psychologist, and a retiree who, herself, is aging very well throughout the process of writing these three books? Do you also see something of a “trilogy” here?

NS: I had not thought of the three books as a trilogy but now that I think about it, each book was an outgrowth of the other. So maybe it is a trilogy. It started with Retire Smart, Retire Happy. I had thought retirement would be a piece of cake. After all, I was an “expert” on transitions and had retired voluntarily. However, retirement for me posed unexpected challenges so I decided to learn how others fared. The result was Retire Smart, Retire Happy. That book became the centerpiece of a PBS special by the same name.

I had many opportunities to continue interviewing and learning about retirement. I realized there was another book which described the paths people follow and the need to strengthen their psychological portfolios. The result was Revitalizing Retirement. This book elaborated on what I had learned in Retire Smart, Retire Happy.

I then became involved in a number of aging projects including writing a transition column for a local magazine. Over time, I realized there was one more—the last—book to focus on aging. This new book broadened my concerns to cover more than retirement. And thus Too Young to be Old was born.

AG: Much of the inspiration for your writing on retirement and aging comes not just from your own experiences, but from the people you’ve worked with in your daily life. Especially after the first book came out, you’ve been engaged by fans and concerned retirees who have come to you with questions about what is often a difficult life transition. What are some of the encounters that have had the most impact on your philosophy and your writing?

NS: Perhaps the most important factor was my own transitions. I found the decade of my eighties filled with transitions—I retired, I became a caregiver, then a widow. I had several surgeries and orthopedic issues. I recovered, began dating and actually went on line, met a retired lawyer, and we now live together. We then moved to a retirement community.

All these transitions make the image of someone in a rocking chair fade.

In addition, many who read my columns reached out saying how they were helped. That made me realize I wanted to keep writing and sharing mine and others experiences.

AG: When Retire Smart, Retire Happy first came out, it coincided with your own decision to retire after nearly three decades teaching counseling psychology at the University of Maryland.  When did you start thinking about ways to retire and age well? Had this been on your mind even in your youth? Or was it your own life transition that spoke to you?

When I was in my late sixties, I went to a retirement party for a much older woman. She was still productive and dynamic. By accident, I left the party walking with two deans. One said, “She should have retired years ago. She is too old to teach and advise.” Right then and there, I knew I would leave before anyone said that about me. And thus began the process of disentangling first from teaching, then advising. My husband and I decided to move to Sarasota where we used to vacation. Since retiring, I have written 4 books and become active in the community. This year will be the first time I have not had a book contract since 1984. So now I will really be retired. I am a bit anxious about it. It is time to reread my own retirement books!

AG: What advice do you give retirees and the soon-to-be-retired about handling this difficult transition?

NS: If someone is struggling to figure out a new path, think about regrets. Is there anything the person wishes he or she had done? If so, is there any way to turn the regret into a plan? That can get someone thinking about a new dream, a new plan.

too young to be oldAG: Too Young to be Old is the first of your titles to really delve into the issue of ageism. America, certainly, is an aging nation. The number of Americans age 55 and older will increase dramatically between now and 2030 – from 60 million today (21 percent of the total US population) to 107.6 million (31 percent of the population) – as the baby boomers reach retirement age.   You’ve written these three books over the course of a decade. What changes have you seen in that time? Is ageism on the rise or the decline? How can individuals embrace aging, and combat ageism?

NS: Ageism is all around us. Even those who are demographically in the old or old-old group exhibit age bias. As the president of AARP wrote, we need to “disrupt aging.” The first step is to be honest about our own ageism, then confront others when they make derogatory comments like, “I live in an old person’s home,” “I just had a senior moment,” “Look at that old lady,” etc.

AG: Do you have some advice on how the children and grandchildren of retirees can help their elders age well and embrace life and happiness after the retirement transition?

NS: Family is very important to most older individuals. So it is important to stay connected. Many of the people I interviewed for Too Young to be Old resented their adult children “bossing” them. Therefore, give the benefit of the doubt to older individuals, give them as much freedom as possible, show respect and help them maintain their dignity.

AG: What is the best thing about aging?

NS: I love the freedom of being 87. I say my age with pride. I never expected to live this long and continue publishing. I like my white hair but must admit the wrinkles surprise and dismay me when I look in the mirror. It is important to remember George Vaillant’s advice–stay young at heart by learning something new, trying something different, and embracing the time you have to spend with family.

April Releases from APA Books!

too young to be oldNew from APA LifeTools®!

Too Young to Be Old 

Love, Learn, Work, and Play as You Age

Nancy K. Schlossberg, EdD

As the “Baby Boomer” generation reaches retirement age, an unprecedented number of Americans will soon be 55 or older. More so than ever before, the question on our minds is: How do I age well? In this accessible and upbeat guide, Schlossberg builds on the concepts she pioneered in her popular books Retire Smart, Retire Happy and Revitalizing Retirement with an engaging take on positive aging. Looking at the basic issues of aging–health, finances, relationships, and how to live more creatively–readers will be able to think about and develop a deliberate plan to age happily.

 

trauma psych

APA Handbook of Trauma Psychology

Volume 1. Foundations in Knowledge

Volume 2. Trauma Practice

Editor-in-Chief Steven N. Gold

The APA Handbook of Trauma Psychology is an essential resource to specialists in trauma who need comprehensive information, to practitioners who seek to familiarize themselves with the range of approaches for trauma assessment and treatment, or for students as a graduate level or advanced undergraduate level textbook.

 

 

frailty suffering vice

Frailty, Suffering, and Vice

Flourishing in the Face of Human Limitations

Blaine J. Fowers, Frank C. Richardson, and Brent D. Slife

This work addresses the human condition in its entirety and discusses the pathways to flourishing in light of the everyday limitations that we all must face. How do we realize our best selves and flourish in the face of our frailty, vice, and suffering? The authors address what they call the “breathless optimism” of positive psychology in this unique and approachable volume filled with original research and case studies. This book explains how human dependency, limits, and suffering are not just negatives to be overcome. Rather they are part of our journey towards healing and development.

 

couples on the brink

Helping Couples on the Brink of Divorce

Discernment Counseling for Troubled Relationships

William J. Doherty and Steven M. Harris

Therapists and counselors can find themselves at an impasse when working with “mixed-agenda” couples—where one partner is considering divorce, while the other wants to preserve the marriage and start therapy. Such couples are a common and difficult challenge in clinical practice.

To help confirm each partner’s agenda before taking decisive steps toward either reconciliation or divorce, this book presents a five-session protocol for helping couples understand what has happened to their relationship and each person’s contributions to the problems. The goal is to gain clarity and confidence about a direction for their marriage.

 

mentalization-based children

Mentalization-Based Treatment for Children

A Time-Limited Approach

Nick Midgley, Karin Ensink, Karin Lindqvist, Norka Malberg, and Nicole Muller

Mentalization-based treatment (MBT) promotes clients’ ability to interpret the meaning of others’ behavior by considering their underlying mental states and intentions, as well as clients’ capacity to understand the impact of their own behaviors on others.  This book is the first comprehensive clinical introduction to using this approach with children, 5-12 years old, who suffer from emotional and behavioral problems including anxiety and depression.  Chapters examine problem assessment and case formulation, the therapist’s stance, and treatment termination. The guide also includes a chapter-length case illustration and an appendix that lists measures of reflective functioning in children and parents, as well as validation articles.

 

 

Christopher Keys: On Community Psychology

by Kristen Knight

Communities can assume many forms—from online forums to residential neighborhoods, from large collaborations to small groups of people. The APA Dictionary of Psychology, Second Edition, defines community psychology as a discipline “that encourages the development of theory, research, and practice relevant to the reciprocal relationships between individuals and the social systems that constitute the community context.” But these ideas may seem a bit abstract—so we consulted an expert in the field to put them in context.

Christopher B. Keys served on the editorial board of the APA Handbook of Community Psychology—released last October as part of the APA Handbooks in Psychology ® series—along with fellow Editors-in-Chief Meg A. Bond and Irma Serrano-García and Associate Editor Marybeth Shinn. The handbook spans two volumes, and contains 63 chapters contributed by dozens of authors from around the world. It was the first comprehensive work to be published in the field in more than 15 years. Here, Dr. Keys describes community psychology and talks about why it matters.

Note: The opinions expressed in this interview are those of the authors and should not be taken to represent the official views or policies of the American Psychological Association.

Chris Keys picture 2013

Christopher B. Keys, PhD, is a professor emeritus and former chair of the psychology departments at both the University of Illinois at Chicago and DePaul University. He has also been a founder and chair of the community psychology doctoral program in the psychology department at the University of Illinois at Chicago, and a professor and codirector of the advocacy and empowerment of minorities program in the department of disability and human development at the school. He was the founding associate dean for research in the college of science and health at DePaul University.

Dr. Keys’s research has focused on organizational approaches to community psychology, organizational empowerment, community research issues, and the positive community psychology of disability, and in addition to lecturing and conducting workshops all over the world, he has coauthored and coedited more than 125 articles, chapters, and books on community psychology and disability-related topics. 

KK: How do you define community psychology?

CK: Community psychology is the study of the relationship between person and context and the action taken to improve that relationship by creating a more socially just social contract. More specifically, community psychologists investigate and take action to support and empower persons who have less than their fair share of society’s resources and the variety of community contexts in which they live and by which they are influenced.

KK: How does community psychology apply in our day-to-day lives?

CK: Community psychology examines current social problems and develops constructive ways to address them. Consequently, community psychologists engage in research and action on a variety of important social issues, such as improving educational opportunities, preventing homelessness and enhancing the mental health of those people who are disadvantaged by virtue of society’s marginalization of members of selected groups. These include, but are not limited to, people with disabilities, people living in poverty, people with minority sexual orientations, people of color, and/or women. For example, if you are seeking to improve an after-school program in a low income neighborhood that enhances urban children’s wellbeing and academic performance, then consult relevant work on these issues in community psychology.

KK: What are some of the most important issues that the field is addressing today, and has this changed since the formal recognition of the field more than 50 years ago?

CK: In addition to the topics mentioned above, community psychology issues of particular import today that endure from early in the field’s history include

  • taking an ecological perspective to better grasp the context in which social problems develop and have impact;
  • thinking critically to challenge orthodoxy, such as the assumed preeminence of evidenced-based practice for assessing the quality of interventions; and
  • valuing participation of community members and those from other disciplines as well as partnerships with community organizations in research and action.

Some topics that have grown in importance over the last 50 years since

community psychology was formally established in the United States include (a) celebrating diversity in its many forms; (b) understanding the socioemotional side of community including the psychological sense of community, social capital, and social support; and (c) emphasizing human strengths and resilience in seeking to understand and empower those who face societal prejudice and discrimination.

KK: How do the author demographics and range of topics discussed in the handbook reflect the field?

CK: The handbook authors are a diverse group in terms of demographics, arena of work, and discipline. A notable number are from diverse disciplinary perspectives, nations, races and ethnicities, sexual orientations and career stages. In their diversity, the handbook authors represent the demographic richness of the field of community psychology in the 21st century. The topics addressed by the 63 chapters in 2 volumes include the theoretical foundations of community psychology, the dimensions of context, the methods for research and community change, approaches to social issues, working with diverse groups, emerging challenges, controversies and opportunities, and practical issues related to becoming and being a community psychologist. The handbook also includes topics suggested by experts consulted in an open meeting at the Fourth International Conference of Community Psychology in Barcelona in 2012, and by other thought leaders. Taken together, these topics, while not exhaustive, constitute the most comprehensive coverage of the field to date.

References

Bond, M. A., Serrano-García, I., & Keys, C. B. (Eds.-in-Chief), Shinn, M. (Assoc. Ed.). (2017). APA handbook of community psychology (Vols. 1–2). Washington, DC: American Psychological Association.

VandenBos, G. R. (Ed.). (2015). APA dictionary of psychology (2nd ed.). Washington, DC: American Psychological Association.