Caramel apples, a fall food favorite in the US where all apple-related products are synonymous with the season, may have become a little less fun to eat. While the deliciously tart and sweet treats—dipped in caramel and drizzled with toppings—are a (messy) delight to consume, a recent study suggests the handheld sweets are a breeding ground for Listeria if they left unrefrigerated.

According to the study published in mBio, an online open-access journal of the American Society for Microbiology, caramel apples punctured by dipping sticks left unrefrigerated over the course of a couple weeks may harbor Listeria monocytogenes.

The researchers studied Listeria growth on Granny Smith apples dipped in caramel stored at both room temperature and in the refrigerator. After three days, the researchers found the average population of Listeria monocytogenes increased 1,000-fold on the apples with sticks compared to those without sticks when stored at room temperature.

Curiously, neither caramel (because of its low amount of water) nor apples (due to their acidity) are normally breeding grounds for Listeria, according to Kathleen Glass, coauthor of the study. The researchers theorize that by inserting a stick into the apple, a little bit of juice reaches the fruit’s surface and is trapped under a layer of caramel that “creates a microenvironment that facilitates growth of any L. monocytogenes cells already present on the apple surface.” Both moisture transfer and microbial growth are accelerated at room temperature compared to refrigeration, Glass adds.

To decrease the risk, Glass suggests caramel apple manufacturers thoroughly disinfect apples before dipping them in caramel, add growth inhibitors to the caramel coating or apple wax, or use better temperature-time controls to inhibit the growth of L. monocytogenes.

Since the study was published, some retailers are taking precautions to prevent these potentially harmful products from making their way into consumers’ homes.  For example, the Kroger Co. will no longer sell unrefrigerated caramel apples that have been pierced with dipping sticks. “After reviewing the study, we have decided to voluntarily withdraw unrefrigerated caramel apples from our stores,” says Payton Pruett, Kroger’s vice president of food safety. “While we believe the potential health risk is minimal, we are acting out of an abundance of caution on behalf of our customers.”

Kroger says although no illnesses have been reported in connection with these products, all its stores have pulled the products from shelves and distribution centers and disposed of them. However, the company says it is open to carrying these products again in the future and will work with suppliers to reduce the risk of bacterial growth.

Researchers say the study was prompted by an outbreak of Listeriosis in 2014 that, according to CDC, affected 35 people from 12 states. Thirty-one people were hospitalized, and seven deaths were reported. CDC says Listeriosis contributed to at least three of these deaths. Many of the people who were infected reported eating commercially produced, prepackaged caramel apples before becoming sick, prompting a recall from three manufacturers.

The study can be found here.