Use good notation

By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems, and, in effect, increases the mental power of the race.

— Alfred North Whitehead, “An Introduction to Mathematics”

Good notation can make the difference between a readable paper and an unreadable one.

Ideally, notation should emphasize the most important parameters and features of a mathematical expression or statement, while downplaying the routine or uninteresting parameters and features. For instance, if one does not care much about the exact values of constants in estimates, then notation which conceals these constants (such as $O()$, $\ll$, or $\lesssim$) are useful; conversely, these notations should be avoided if the precise values of these constants are of importance to the paper.

Notation which is used globally should be defined in a notation section near the front of the paper, or in the introduction; notation which is only used locally (e.g. within a single section, or within a proof of a single lemma) should be defined close to where it is used (possibly with a reminder that this notation is not used elsewhere in the paper); this is helpful when there are many sections, each with their own extensive notation.

Note that notation or statements which are introduced within a proof of a lemma are already understood to be localised to that lemma; it is bad form to then recall that notation or statement outside of that lemma, except perhaps as a remark or as motivation). In some cases it is worthwhile to define the notation once near the start of the paper, and then recall it whenever necessary.

One should strive to make one’s choices of notation compatible and consistent with notation already in the literature, so that the readers who are already familiar with prior notation will adapt easily to your paper and will not be confused. This is particularly important if you plan to cite results from prior literature, and even more so if you want to claim that an argument from a prior paper can be adapted “with only routine changes” to give a slightly different result that you need for the current paper. If the notations are different between the cited work and the current paper, one should at minimum reproduce the key results or statements from that paper, translated into the notation of the current paper, to spare the reader from performing the translation themselves.

Try to avoid notation which is overly “cute” or “clever”. This can be distracting or appear unprofessional. In particular, the notation should not be cleverer than the actual substance of the paper.

One should definitely avoid naming new terms after yourself (or after your family members, your pets, etc.), for the obvious reasons. If other authors name the concepts you introduce after yourself, and that appellation becomes common usage, then you may use that term as well, but in all other cases it gives the rather blatant impression of vanity or narcissism.

There is an issue of where to strike the balance between too little notation and too much notation. A good rule of thumb is that any expression or concept which is used three or more times will probably benefit from introducing some notation to capture that expression or concept; conversely, an expression which is only used once probably does not need its own special notation. (An exception would be for particularly crucial theorems or propositions in the paper; here it might be worthwhile to invest in some notation in order to make the statement of those theorems clean and readable. Conversely, if an expression only appears in multiple locations of the paper because of coincidences of no significance, then it may be better to avoid introducing notation that gives the false impression of a connection between these appearances.)

If one needs to name a certain property or class of objects, one should generally use very bland names (e.g. “good”, “bad”, “Type I”, “Type II”, etc.) for peripheral or technical terms; colorful terms should be used sparingly, and only for those concepts that are quite central to the paper, lest they distract from the main points of that paper. (This is analogous to how, in film and literature, the main characters generally tend to have more memorable names than the secondary ones.)

Sometimes one is unsure what notation to use for a particular concept, because of potential conflicts with other notation in other (as yet unwritten) parts of a paper. One solution here is to introduce a TeX macro for that notation, and force yourself to use that macro exclusively whenever that notation is used. (For instance, if you have a group which you are tentatively naming G, you could define a macro \grp that is set to G, and use \grp instead of G throughout the paper.) That way, if you find a notational conflict later on (e.g. if you discover that you really need G to denote a graph instead), then you only need to change one line in your TeX file – the line that defines the macro – to resolve the notational conflict, rather than to do a tedious (and error-prone) search-and-replace.

For any rigorous component of the paper, the notation should be precise and unambiguous (and for non-rigorous components, ambiguous notation should be pointed out with “scare quotes” or other cautionary phrases such as “roughly speaking” or “essentially”). A certain amount of abuse of notation is permitted, though, as long as this is properly pointed out.

To illustrate some of the ambiguity issues in notation let us use the common example of division. The expression a/bc can mean either (a/b) c or a/(bc). In such situations I would recommend either using parentheses as above to disambiguate, or else use the \frac construction (i.e. $\frac{a}{b}c$ or $\frac{a}{bc}$). The choice of which one to use is up to you, though one should try to be locally consistent (e.g. if a certain fraction a/b appears in ten consecutive equations, I would use either a/b for all of these equations, or $\frac{a}{b}$ for all of these equations). Also, as the \frac construction tends to be larger and thus draw more of the attention of the reader, I would use for “important” fractions, reserving the more inconspicuous ”/” notation for more “minor” fractions. The symbol $\div$ should largely be avoided (its resemblance to symbols such as +, $-$ and $\times$ suggest commutativity, or at least associativity), unless for some reason / is being used for some other purpose unrelated to division or quotienting.

It is also worthwhile to quietly reinforce one’s notational conventions when given the opportunity. For instance, suppose in one’s argument one has a vector space, which one has decided to call V. When referring back to this object, one could say “the vector space”, or “V”, but if the reader does not remember what vector space is being discussed, or what V is, the reader will have to take a minute or so to flip back and figure this out. But if instead you refer to this object consistently as “the vector space V”, then the notational convention is reinforced, and the reader can continue reading without breaking rhythm. (One can also modulate the choice of terminology used here to emphasise different aspects of the object being referred to. If for instance it is the additive structure of V which is currently relevant, you can instead say “the additive group V”; if, later, it is the topological structure which is the most important, one can say “the topological vector space V”, and so forth. This allows one to subtly draw attention to the most important features of the object under consideration, without distracting the reader from the main body of the argument.)

I give some further thoughts on the desirable properties of mathematical notation in this MathOverflow answer.