Coffee grounds are organic in matter, which makes it a perfect food source for worms. Yet, too much of a good thing always has the risk of throwing the conditions in your worm bin off kilter. This can hurt your worms. As usual, there are many factors at play.