There is a rule, but it’s only of value to somebody who knows which language the root word came from, so it’s really no help at all for most of us. In general, words take un- when they are of English (Germanic) origin and in- if they come from Latin. (The forms im-, il-, and ir– are variations on in-.) Apart from that, there’s really no good guide to which one you should choose. You’re just going to have to stick to learning them by rote.
If it’s any consolation to you, the battle between in– and un– has been going on for centuries, with sometimes one form winning and sometimes the other, which suggests that the problem has been troubling English speakers for a very long time. As an example, for several centuries English had both inability and unability, but the latter disappeared in the eighteenth century for no very obvious reason. Another is familiar from the American Declaration of Independence: “We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable rights …”; these days, it’s inalienable (it should always have been, by the rule, since alien comes from the Latin alienus, of or belonging to another person or place).
A few pairs are still fighting it out, such as inarguable and unarguable. Others have distinct senses, such as unhuman and inhuman, or inartistic and unartistic. Even more confusingly, some pairs of adjectives and nouns have different prefixes: unstable has the noun instability, and uncivil has incivility. All these have to be learned, I’m afraid.
One example that seems to break the rule is inflammable, which looks as though it refers to something fireproof rather than something easily set on fire. The word comes from Latin inflammare, which uses a different in– prefix, one that intensifies the meaning of the root word (so turning flammare, to burn, into inflammare, to burst into flames). It is very confusing; in modern times the old form flammable has had to be revived so that people would not fatally mistake what inflammable meant. If only we’d stuck to the older enflame from French, and used it to make enflammable, we wouldn’t have had the problem.
One thing we can say for sure is that in– and its relatives are not living prefixes. If you want to negate a word today, you are much more likely to use un– (or perhaps non– or a-, but that’s another story.)
Original article: Un- Versus In-