Yeah, from experience, the ones who would complain would probably be those who complains whenever they get a downvote in the first place, and those are generally not common (I think we only ever had three different users maximum doing it each year).
My favorite part of all, tho, is how it is now possible to see our total negative ratings separately from positive ones. With the old system, two users could have been seen as perfect contributors with 200 respect, but one of them actually has +500 and -300, meaning that he contributed even more, but at the same time posted a lot of questionable messages. Of course, though, there are multiple other variables, such as an user having a larger amount of following and thus, getting more votes, or people clicking on the wrong vote icon by mistake (especially mobile users). Even ExtendeD has a few negative votes.