Here We Go Gathering Nuts in May

Just as I was about to go outside (glorious sunshine today) to pick up hazelnuts, this old nursery rhyme jumped into my head.

Here we go gathering nuts in May, nuts in May, nuts in May,
Here we go gathering nuts in May,
So early in the morning.

But why does it refer to May?  I can't think of any nuts that are ripe then. It is said to be a traditional British song, so I can't explain it away by thinking maybe they are gathering nuts in Australia at that time (would you be?)

Some versions of the song refer to a 'cold and frosty morning' which would be another oddity for that time of year.

An entry in Wikipedia suggests that it might refer to pignuts. The Wild Food School tells us that pignuts have been used as a food source but not commonly.

Maybe pignuts were collected for animal feed though the collecting of them sounds labour intensive.

Anyone else got ideas out there on the matter?  Could it be it was originally Nuts for May, keeping them for May festivals?

Anyhow, here I go gathering nuts in October.... it occurs to me they may just have taken poetic licence as that doesn't sound at all right!