Baby oil is just pure mineral oil so I suspect that allergies aren't the problem, unless you're using oil from you car or some other mineral oil with additives. Some older plastics are slowly attacked by petroleum products, though, as Lemur suggested.