Why do Australians call the English poms or pommes?

POHM stands for Prisoner of Her Majesty or Prisoner of Her Majesty's Mother England. Since Australia was first used by the English as a penal colony for it's convicts, shouldn't Aussies refer to themselves as poms? It doesn't make sense otherwise. I'm Scottish by the way so I'm neutral on this.