The confusing part is not se vs le for me but "to" vs "for." Your explanation was that an indirect object means to him/it, etc., but the example is "for him", which is very different in English. I think this needs to be explained. When I speak I usually try to clarify with "para mi" for "for me", but it may not be right.
Tom