.jpeg uses an image compression that will alias what you intend to be a hard edge in an image. Since KEY COLOR is an exact RGB number, you will get a halo for the pixels that aren't exactly the key color.
Use .tga or .png
Even then, you need to insure that the alpha channel is only of the pixels that you want to see...
For example, lets suppose you have some text that is red, which you want to decal onto a model. But the red text was drawn on a white background.
If you make an alpha channel based on selecting the text, you will end up with a white halo (because of the tolerance of your selection)...
But if you make the alpha channel for the text, then fill the whole image with red, your alpha channel will clip the text out of the red background giving you no halo.
Hope that clears it up a little