This might turn into a Wikipedia essay one day, right now it's just the beginnings of a page of personal thoughts. Anyone can edit this if they like, although I reserve the right to exercise more editorial control here than I would in article space.
The topic of abuse of administrative powers comes up frequently. Frequently, the accuser is a newbie, troll, or POV pusher and the accusation is completely unwarranted. Many editors have become jaded about such complaints because of this. However, I have seen an unfortunate amount of questionable conduct by admins, including questionable use of admin functions, so I agree that there's a problem here.
Update: The Arbcom now seems willing to remove adminship from those who exhibit a pattern of misuse of the tools. This may well pretty much solve the problem. So I'm not sure how much of what's here is relevant anymore. Another update: I'm not sure at all that the arbcom still shows much willingness to remove adminship except in obvious cases. (Is this changing?) Yet another update: I still think we need a process for admins who have lost the confidence of the community. The crats, despite being charged with determining consensus for adminship, seem unwilling to address the other half of the issue. Many editors seem convinced that only Jimbo or Arbcom can remove adminship, but I think they're only saying that because no one else ever has. Maybe it's a small issue- several of the admins who I'd lost confidence in are no longer admins, so the system may well be working just fine.