This is an interesting problem. Here in North America small farmers have pretty much been decimated because it seems that every economic shock of any sort makes their life more difficult. A century or so of that and it's no wonder that most farmers have sold the family plot and taken up web site design or retired to someplace warm. Currently, the percent of US population engaged in agriculture is slightly over 1 percent. And US/Canadian farms mostly weren't/aren't all that small by Asian standards.
China seems to be doing much better agriculture wise than most developing countries, but they seem to have developed a weird system with some family members intensively farming their leased plot while living in a house they own built on land they lease. And other family members work in sometimes distant cities. It's difficult to imagine such a system working at all much less being transplantable to India. Maybe Japan or Europe offers some answers that will work for India.
So Modi is probably correct that Indian agriculture needs massive structural changes. His problem is that any such revolution is likely to involve politicians hanging from lamp posts. I doubt he favors that part.