Mögt ihr ihn? would in fact translate to: Do you like him?
To understand this we need to be aware of the double meaning of “ihr”, just like you pointed out. In this specific case the declension of mögen, in the form of “Mögt” tells us that we are talking to more than one person or a group of people. Therefore, a group of people is asked if they like a person with a male article (ihn). And as you said the “ihn” signifies that we are dealing with a Akkusative pronoun.
Since “her”(ihr) is a Dativ pronoun, it can’t be used in this exact circumstance and we would need to change it to “sie”(she) a Akkusative pronoun. For example: Mag sie ihn? = Does she like him?, again changing the declension of “mögen” to show this, as well.