Can we handle our baby zebra finches?

we are first time breeders and our first batch has brought us 5 little baby zebra finches. I would like to know if and when we can handle them without risk of the parents abandoning them or having some strange effect on them. also, is it a good idea at all? The parents are doing a great job and we are about 3 1/2 weeks in and i don't want to mess anything up.