By Amy Dockser Marcus

When she was in her early 30s, Katy Mathes decided to check her cancer risk. A genetic test showed a mutation on a BRCA gene, which significantly raises a person’s lifetime risk of developing hereditary breast or ovarian cancer.

Thirteen people in the family got tested—her mother, her sister, cousins and aunts. Eleven had the mutation. Almost all did their testing with Myriad Genetics Inc., which introduced the first BRCA tests in 1996.

“I treated my test results like the Bible,” said Ms. Mathes, now 37, an elementary-school art teacher in Colorado. “There was no questioning the report.”

Ms. Mathes, who has one child, decided she would have no more. To reduce her cancer risk, she underwent surgery to remove her ovaries and fallopian tubes. So did her younger sister, their mother and four other relatives. Ms. Mathes and her sister also had double mastectomies.

This year, their mother sat them down at the table of their parents’ winter home in Florida. Two weeks earlier, her genetic counselor had called. The lab was no longer sure the variant is a significant problem.

A family with many members who thought they were at a high risk of cancer might not be after all.

Myriad, a molecular diagnostics company based in Salt Lake City, now says the variant is “of unknown significance.”

The lab said it made the change as more people with the same variant were tested and added to Myriad’s database. The new analysis showed that specific variant might carry a lower risk of cancer than one categorized as pathogenic, which increases a person’s chance of getting a disease.

Reclassifying a variant from “pathogenic” to “unknown significance,” as Myriad did in this case, is very unusual, the lab said. In a paper published last year, researchers working with Myriad data over a 10-year period said it occurred in less than 1% of pathogenic variants.

“We know these are very difficult situations,” said Susan Manley, a board-certified genetic counselor who is senior vice president of medical services at Myriad. “We make these reclassifications very carefully. The science is evolving.”

The company said it can’t comment on individual cases. Changing a classification from harmful to uncertain “is a rare event, but I understand that rare is of no consolation to the patient when it happens to them,” Ms. Manley said.

After her mastectomy, Ms. Mathes suffered a rolling series of complications. The incision on her left breast didn’t stay closed. She couldn’t pick up her toddler son for over nine months, “didn’t do bath time, couldn’t hug him. He couldn’t sit on my lap, I was in such pain.” Her fifth-grade students took on the job of putting chairs on top of their desks at the end of the school day; Ms. Mathes couldn’t lift them.

She takes daily hormones and supplements to help with the symptoms of early menopause caused by the surgery to remove her ovaries. Her body is disfigured, she said, her relationship with her husband under strain. She underwent two additional surgeries to try to fix the complications from the mastectomy and reconstruction. This month, she had a third.

In the world of genetic testing, the two BRCA genes, BRCA1 and BRCA2, are among the best-studied of those associated with hereditary cancer risk.

Clinicians and patients rely on the test results to guide crucial medical decisions. The BRCA tests have high public recognition, especially after actress Angelina Jolie announced in 2013 that she was BRCA-positive and had a preventive double mastectomy and breast reconstruction; in 2015 she announced she had surgery to remove her ovaries and fallopian tubes to further reduce cancer risk.

Millions of people have taken BRCA tests. National and international genetic-testing guidelines recently expanded the numbers of who should be recommended for genetic counseling and considered for BRCA testing. DNA-testing companies, including 23andMe, Ancestry and MyHeritage, now market tests that allow consumers to get some information about their BRCA1 and BRCA2 genes.

“We are moving towards BRCA screening for the entire population,” says Jay Shendure, a geneticist at the University of Washington.

And as more people get tested, the interpretation of those results is going to change.

Ms. Mathes’s saga began in July 2015 when her aunt called her mother to say that she had a positive BRCA test. Her aunt, Carol Larson, had taken the test when it was offered during a regular gynecological exam. Ms. Larson had told the doctor her sister had breast cancer.

When Ms. Larson learned she tested positive, she researched strategies for managing elevated cancer risk, such as more frequent screening. According to what she read on the website of a breast-cancer-patient organization, preventive surgeries significantly lowered cancer risk.

She called others in her close-knit family. “I worried they were going to panic,” said Ms. Larson.

Even before learning about her aunt’s positive result, Ms. Mathes had decided to take the test. Her family had history with cancer. Now that Ms. Mathes was a mother, she wondered about her own risk.

In August 2015, Ms. Mathes went to a genetic counselor to receive the BRCA results. Her mother, who had breast cancer at age 49, was visiting and went with her to the appointment.

Mother and daughter clasped hands when they learned Ms. Mathes tested positive for a BRCA variant. “I felt horrible knowing I passed this on,” said her mother, Jane Setchell.

Ms. Mathes found it hard to concentrate during the counseling session.

She stared at the big red bar, vivid and jarring, running across the first page of the report that indicated a positive BRCA test.

The statistics in the report scared her. It stated she had up to an 84% risk of breast cancer by age 70, compared with 7.3% in the general population, and up to 27% risk of ovarian cancer by age 70, compared with 0.7% in the general population.

Afterward, she and her parents, and her son, Jaxson, went out for lunch. Ms. Mathes, feeling drained, barely touched her food. She didn’t know what to say to her parents. “I guess it’s better to know rather than not know,” she told them.

Her mother and three aunts—who now range in age from 61 to 65—moved quickly to get surgery. Three of the four women had their ovaries and fallopian tubes removed the same month, November 2015.

“We bonded together. We called it Neuter November,” said Judy Gaebler, one of the aunts. “Myriad’s test is the gold standard. Everyone goes by what they say.”

Ms. Setchell said she encouraged her sisters to get the surgeries. “I felt we had been dealt a tragic blow from which we could only survive by fighting back. That fighting back meant we had to have the surgeries.”

Over the next months, Katy Mathes struggled with what to do.

One day in September 2015, a month after she got her report, she woke up with intense sadness, finally acknowledging that surgery would mean she could have no more children. She cried all the way to work and sat in her car, unable to go in. She called the principal from the parking lot, crying, and said she couldn’t come to work.

She talked to her husband, Kyle Mathes. “We can’t sit here and wait,” said Mr. Mathes. “What if we don’t find it in time?”

Her sister, Tricia Leigh, also had a positive BRCA test and also wrestled with the decision to have surgery. She was breast-feeding her second child. Ms. Leigh said she feared passing on a mutation to future children.

She recalled her college graduation, when their mother was still in treatment for breast cancer. “Mom wore a wig. It wasn’t super cold, but she was freezing, she couldn’t warm up. She couldn’t stay for the whole ceremony,” she said.

Ms. Mathes called her mother to tell her she was proceeding with the surgeries. Her mother cried.

“I would have loved for her to have another child,” she said. “But when you are scared you are going to harm that child, why would you?”

Ms. Mathes had more intense screening, including mammograms and an ultrasound. In June of 2016, she went to see a breast surgeon to discuss surgery. She brought a copy of her Myriad report. “I am so sorry this is happening to you,” she recalls the doctor saying.

They discussed the gene mutation, her family history of breast cancer, and risk-reduction strategies. They discussed the different options. Ms. Mathes told the doctor she wanted to proceed with a prophylactic mastectomy.

Ms. Mathes planned surgery around breaks from teaching. She had a mastectomy in December 2016 during winter break and surgery to remove her ovaries and fallopian tubes in May 2017, at the end of the school year.

Almost two years later, Ms. Mathes and her sister both flew with their children to Sarasota, Fla., to visit their parents. It was the first time taking the children to Disney World and the sisters were excited. “There was so much to be happy about,” Ms. Mathes said.

One evening, after the children were in bed, the sisters sat with their parents at the table having a drink of wine. “I have to tell you something,’” their mother said. That’s when she broke the news that the lab’s interpretation of the family BRCA variant had changed.

The genetic counselor who had called urged her to tell family members right away.

“My brain just shut off,” said Ms. Mathes.

Her mother, who is a pharmacist, had read scientific papers about variants of uncertain significance. She started sharing the information she learned with her daughters.

The technical information felt overwhelming to Ms. Mathes. She didn’t understand the terms. She didn’t want to talk about it. The conversation ended. The next day they went to Walt Disney World.

When Ms. Mathes got home to Colorado, she didn’t immediately tell her husband. “I didn’t want to tell him we did all this for nothing. We could have had more kids,” she said.

In May, she received her own Myriad report from the genetic counselor’s office. The couple read it together after their son was asleep.

The red bar was gone. So were the tables saying she had up to an 84% risk of getting breast cancer.

“Is this saying what I think it is saying?” Mr. Mathes asked.

“I think it means I didn’t have to do what I did,” Ms. Mathes replied.

Myriad’s Ms. Manley said the reports stress that using the results to make clinical decisions should also include a patient’s personal and family history. Researchers are trying to find better ways to personalize individual risk assessment. For now, “The risks we put on the report are general,” she said.

Labs don’t all agree on classifications. Other genetic testing companies still classify the mutation that the family members have as harmful.

Unlike many medical tests, “genetics is murky,” says Stephen J. Chanock, a geneticist and director of the division of cancer epidemiology and genetics at the National Cancer Institute. “It’s not so simple as ‘Doctor, do I have to worry or don’t I have to worry?’ ” he said. “There is a continuum of risk.”

There are tens of thousands of BRCA variants and new research reveals that not all confer the same level of risk to a patient. Some gene variants “have intermediate or moderate levels of risk, not full-blown risk,” says Fergus J. Couch, a professor at the Mayo Clinic, whose lab has worked on some of these studies.

Patients and even genetic counselors may not understand the weight of the evidence behind a lab’s determination that a variant is pathogenic. Labs use algorithms, reports in scientific literature and their own internal databases, among other factors, to predict how likely a variant will affect a body’s cells.

For many variants, “you are making a judgment call but that is not always clear to the public,” said Dr. Couch.

The analysis around the BRCA variant in Ms. Mathes’s family put a lot of weight on a single 2011 paper, says Melissa Cline, a researcher at the University of California Santa Cruz Genomics Institute and project manager of the BRCA Exchange, a shared database for BRCA variants.

The 2011 study indicated that the variant likely alters the BRCA2 protein, which plays a role in suppressing tumors. For a clinical interpretation, Dr. Cline said, researchers would want to see evidence that the change affects the actual function of the protein in the process of DNA repair. Sometimes cells compensate in other ways.

Rien Blok, a clinical laboratory geneticist at Maastricht University Medical Centre and one of the authors of the 2011 paper, said an international consortium of researchers is working on conducting such experiments, but they take time. The experiments in the 2011 study still meet the current standard for deeming a variant deleterious. “I am still behind our conclusions until proven otherwise,” Dr. Blok said.

Seth Marcus, a genetic counselor at Advocate Health Care in Park Ridge, Ill., who counseled Ms. Mathes’s mother and one of her aunts, said when Myriad notified him about the change, he checked a public database to see how other labs classify the variant. The six labs that submitted data report the variant as “likely pathogenic.”

He also called Myriad and three other labs. Most said they had seen the variant in only a very limited number of cases.

“In the end, you give the patient the data and the knowledge you know,” he said. “People have to go from there. You work with the data you have.”

In November, Ms. Mathes and her husband went back to the same medical office to talk to a genetic counselor together. They sat next to each other as Ms. Mathes filled out the paperwork.

“Any chance you are pregnant?” Ms. Mathes read the question aloud.

“No, they took those from you,” Mr. Mathes said.

This time around, Ms. Mathes asked questions that hadn’t occurred to her in 2015.

What Ms. Mathes discovered surprised her. Today, Myriad has 38 people with her variant in its database. Twelve are from her own family. She had no idea in 2015 how many people Myriad had in its database with her variant.

Both she and her husband said had they known about the sample size, they might have asked more questions about whether it made sense to do surgery or wait.

Fear of cancer is what ultimately made the couple opt for surgery. When the initial report stated that Ms. Mathes had “up to 84%” risk of breast cancer by age 70, “we both felt it was ridiculous to play those odds,” said Mr. Mathes.

The genetic counselor said the same estimate was given to anyone with a positive BRCA result and it wasn’t tailored to Ms. Mathes’s specific mutation.

The genetic counselor noted that Myriad didn’t call the variant benign. As more people get BRCA tests, the report might change again.

“Our genetic counselors inform patients about accepted clinical guidelines and the best available scientific data at the time of the patient consultation,” said Cynthia R. Dickerson, vice president, marketing and sales, for Radiology Imaging Associates and Invision Sally Jobe, the Englewood, Colo. clinic that provided the counseling. “When testing facilities provide updated information, we contact the patient.”

Based on Ms. Mathes’s history, and the fact that the significance of her variant is now considered uncertain, the counselor estimated that her lifetime risk of breast cancer is 21%. That is still higher than the average, the counselor said.

“That is not high enough to make me remove organs. I would have had another kid. I would have waited to do surgery,” Ms. Mathes later said.

Recently, Ms. Mathes’s mother told her daughters that Myriad offered to conduct more extensive genetic testing to determine whether she tested positive for mutations on 35 genes. That might predispose her, and by extension, her daughters, to eight types of hereditary cancers.

The company offers such testing to customers whose results are downgraded to see if it can clarify a family’s risk. Her mother took the test.