Overview
The theory of AHP does not demand perfect consistency. AHP allows inconsistency but provides a measure of the inconsistency in each set of judgments. This measure is an important by-product of the process of deriving priorities based on pairwise comparisons. It is natural for people to want to be consistent. Being consistent is often thought of as a prerequisite to clear thinking. However, the real world is hardly ever perfectly consistent, and we can learn new things only by allowing for some inconsistency with what we already know.
A measure of inconsistency is computed for each set of judgments when making paired comparisons. Inconsistency follows the transitive property; for example, if you were to say that A > B, and B > C, then say that C > A, you would have been inconsistent. However, as described below, the AHP and Expert Choice "inconsistency ratio" measures more than just transitive consistency.
It is more important to be accurate when deriving priorities than to be consistent. There are some people who are perfectly consistent but consistently wrong. If we are perfectly consistent (as measured with an AHP inconsistency ratio of zero), we cannot say that our judgments are good, just as we cannot say that there is nothing wrong with us physically if our body temperature is 98.6 degrees. On the other hand, if our inconsistency is, say, 40% or 50% (an inconsistency ratio of 100% is equivalent to random judgments), we can say there is something wrong, just as we can say that there is something wrong if our body temperature is 104 degrees.
Tom Saaty, the creator of AHP and cofounder of Expert Choice, reasoned that consistency is important in our everyday lives, but we must allow for some inconsistencies to learn new things. As a "rule of thumb," he proposed that we should be consistent about an order of magnitude (10 times) more than we are inconsistent. From this, he reasoned that an inconsistency ratio of about 10% or less is reasonable. However, particular circumstances may warrant the acceptance of a higher value, even as much as 20% or 30%. A set of judgments that results in priorities that are more accurate is better than a set of judgments that are more consistent but results in less accurate priorities. But since we are deriving priorities and don't usually know what the real priorities are, we should ask whether the derived priorities are reasonable or not, at the same time that we examine the inconsistency ratio.
Causes of Inconsistency
Let us look at some of the reasons why inconsistency occurs, as well as the useful information that the inconsistency ratio conveys, and ways to address it.
The most common cause of inconsistency is clerical error. When entering one or more judgments into a computer, the wrong value, or perhaps the inverse of what was intended, is sometimes entered. Clerical errors can be very detrimental and often go undetected in many computer analyses. When using Expert Choice Comparion, one can easily find and correct such errors.
A second cause of inconsistency relates to model structure. Ideally, one would structure a complex decision in a hierarchical fashion where factors at any level are comparable, within an order of magnitude or so of other factors at that level. Practical considerations (number of judgments that would be required) frequently preclude such a structuring, but it is still possible to get accurate results. Suppose, for example, we compared several items whose ratios differed by as much as two or three orders of magnitude. One might erroneously conclude that the AHP verbal scale is incapable of capturing such ratio differences since the verbal scale ranges from 1 to 9. (This is not an issue with the Graphical/Numerical pairwise scale.) However, because the resulting priorities are based on second, third, and higher order dominances, priorities derived with the AHP verbal scale can produce ratios that differ far beyond an order of magnitude. A higher than usual inconsistency ratio will result because of the number of extreme judgments, but the priorities will be reasonably accurate. If one recognizes this as the cause (rather than a clerical error, for example), one can accept the inconsistency ratio even though it is considerably greater than 10%.
A third cause of high inconsistency is the inappropriate use of "extreme" judgments. This can occur if an evaluator chooses "extreme" for every or most every judgment rather than using the verbal scale to connote intensity as well as direction. To help reduce the possibility of this occurring, Expert Choice Comparion will inform an evaluator of the meaning of an "extreme" judgment the first time or two that a judgment is made. However, the Project Manager might examine the judgments of evaluators with high inconsistencies to see if this is the case and, if so, ask the evaluator to reconsider their judgments. Asking evaluators to provide comments for their judgments beforehand is a good way ascertain this.
A fourth cause of inconsistency is lack of information. If an evaluator has little or no information about the factors being compared, then judgments will appear to be somewhat random, and a high inconsistency ratio will result. Sometimes we fool ourselves into thinking that we know more than we really do. It is useful to find out that a lack of information exists, although sometimes we might be willing to proceed without immediately spending time and money gathering additional information in order to ascertain if the additional information is likely to have a significant impact on the decision.
A fifth cause of inconsistency is lack of concentration during the judgment process. This can happen if the people making judgments become fatigued or are not really interested in the decision.
A sixth cause of a high inconsistency ratio is an actual lack of consistency in whatever is being modeled, or what we might refer to as real world inconsistencies. The real world is rarely perfectly consistent and is sometimes fairly inconsistent. Professional sports is a good example. It is not too uncommon for Team A to defeat Team B, after which Team B defeats Team C, after which Team C defeats Team A. Inconsistencies such as this may be explained as being due to random fluctuations, or to underlying causes (such as match ups of personnel), or to a combination. Regardless of the reasons, real world inconsistencies do exist and thus will appear in our judgments.
Consistency is Necessary but Not Sufficient
It is important that achieving a low inconsistency does not become the goal of the decision making process. Reasonable consistency is necessary but not sufficient for a good decision. It is possible to be perfectly consistent but consistently wrong. It is worth saying again: It is more important to be accurate than to be consistent. You should NOT change judgments just because of inconsistencies. You should reexamine judgments because of high inconsistency and change only those judgments that you feel were incorrectly recorded or for which you have a change of opinion - regardless of inconsistency.
Background and Theoretical Concepts related to Inconsistency
Inconsistency– measurement and significance
The priorities derived from relative pairwise comparisons are computed as the normalized values of the right hand eigenvector associated with the largest eigenvalue of the reciprocal matrix formed from the pairwise comparisons. (For those non-mathematicians reading this, all you need to know is that the eigenvector computation - which Expert Choice Comparion will do for you - is one of the most powerful and widely applicable concepts in mathematics and physics. If you Google "Eigenvector and billion and Google" you will find several explanations of how Google made billions using Eigenvectors.)
The largest eigenvalue (there are multiple solutions or values just as there are two for the quadratic formula) is referred to as lmax. The closer lmax is to n, the number of elements being compared, the more consistent are the judgments. If judgments are perfectly consistent, lmax will equal n. Thus, the difference between lmax and n, can be used as a measure of inconsistency. Instead of using this difference directly, Saaty defined a consistency index as:
(lmax - n)/(n -1)
since it represents the average of the remaining eigenvalues.
In order to derive a meaningful interpretation of either the difference or the consistency index, Saaty simulated random pairwise comparisons for different size matrices, calculating the consistency indices, and arriving at an average consistency index for random judgments for each size matrix. He then defined the consistency ratio as the ratio of the consistency index for a particular set of judgments, to the average consistency index for random comparisons for a matrix of the same size. Forman performed additional simulations and calculated indices for cases with missing judgments.
Since a set of perfectly consistent judgments produces a consistency index of 0, the consistency ratio will also be zero. A consistency ratio of 1 indicates consistency akin to that which would be achieved if judgments were not made intelligently, but rather at random. This ratio is often called the inconsistency ratio since the larger the value, the more inconsistent the judgments.
E. H. Forman, "Random Indices for Incomplete Pairwise Comparison Matrices" European Journal of Operations Research Vol. 48, #1, 1990, pp. 153-155
