Mired in history
At its start, the IETF was tied to the US government. Its first meeting consisted of 21 US government sponsored researchers and, since it was in 1986, I would guess that they were all white males. Over the years, the government aspect toned down but the US centrism remained as did the numbers of white male participants.
Because of this history, it is not surprising that potential non-US and/or BAME participants question whether their voice will be heard and their decision not to participate maintains the status quo.