I know this will raise allot of eyebrows, but if we take a closer look there may be some benefit into defining the term for men who garden. I ask this question because there are allot of men who love the idea of "gardening" but they don't fully engulf themselves into it because there is that astigmatism that "gardening" is for women.
We all know it is not, because most of all our produce comes from farms who are owned and operated by men. But maybe that's it, if you have allot of land and you grow things you are considered a farmer, but if you have small plot with little space you are considered a gardener. I can almost bet that if you ask most people if they associate a man or women with farming, they will probably, overwhelmingly pick the guy with the straw hat and overalls instead of the girl with the sundress and straw basket. But nonetheless, gardening and farming are still one in the same, especially if you are growing vegetables. It doesn't matter if you are doing it commerically or personally, they both require the same basic fundamentals.
The term that comes to my mind and that is unisex is "homesteader." To me it seems that under this "homesteader" umbrella you pretty much do it all and gardening, farming, recycling, preserving and being self-sufficient becomes more of a lifestyle instead of a job or a hobby. So if I had a "Man term" for Gardening it would be called Homesteading, because you just don't stop at gardening. I personally don't mind being called a gardener, but I must admit I do get some weird looks from the guys when I talk about gardening instead of Homesteading.
With some thought and a little more attention to this, maybe some women who want there partners to join them out in the field can label their activities under a different title. I will almost guarantee that the increase of homophobic men who hate anything associated with girls will, without a doubt, get their hands dirty in the garden; If they are considered homesteaders or something else other than gardeners.