It's important to provide your cats with plenty of clean, litter space so that they can take care of their business comfortably. If not, cats may become stressed out and take care of their business somewhere they're not supposed to (like your very own closet).
A general rule of thumb is that you should buy one per cat, plus one more. Thus, if we have n cats, we need n+1 litter boxes. I worked out the complicated math below:
Of course, this all depends on how often you clean, and how big or small your living space is. Personally, we have two cats and just one large litter box (shown below) that we clean out daily.