Food Allergies Treatment

My two sons both had problem digesting milk when they were babies, had to put them both on soy formula. But they grew out of it, at like 2 and 3 years old they were able to drink cow’s milk no problem. But the last bout of the flu we had seemed to mess them both up. Every time they eat cereal with milk for breakfast before school, they either vomit and have diahhrea. My husband had this problem when he was little, and my own mother is lactose intollerant. They can drink milk any other time of the day, have cereal in the evenings no problem.
My question is….does anybody happen to know why this happens? Would this be a milk allergy or lactose intollerance?
Just curious about this, as it seems rather odd to me. The body is very mysterious!