I don't like charms, and haven't for years, for a few reasons.
1: Unlike regular CC, you can't damage them, and they'll eventually become un-charmed and attack you again.
2: Many quests have kill requirements (aka: this barrier will not go down until specific enemies are killed) and charms slow this down significantly by forcing you to wait for it to wear off so you can kill what the quest requires you to kill.
3: Unreliable aggro. They generate their own aggro and act with their own crappy AI, making things more chaotic/messy. I like to control my fights, or at least know that those with me have the same goal and we can work together to keep the combats controlled and efficient. Charms, Summons, and Hirelings don't do this.
...and the upcoming planned changes will add another reason. Charms don't count as defeated. I'd guess those who are telling you to stop know that habits don't change on a whim, and want you to get into the habit of not charming enemies early so that once the changes do go live, you'll already be with the program instead of causing grief then.