Enough of these old wive's (or even old farmer's) tales
, I think even the Met Office is a better weather forecaster than Elms.
Since Elms are wind-pollinated it is most likely because there was a dry breezy spell when the trees were in flower.
All the best