A global, historical rule of law index

Abstract

This article introduces the V-Dem Rule of Law Index, capturing formal-procedural aspects of de facto adherence to rule of law principles. It relies on information from fifteen V-Dem indicators based on systematic expert assessments. The measure covers almost all sovereign and semi-sovereign polities from 1789 until today, which means that its scope by far surpasses that of extant alternatives. All country-year point estimates are supplemented by confidence intervals. The conceptual framework underlying the index builds on a survey of established, thin understandings of the rule of law. Empirical dimensionality tests reveal that the indicators tap into a common trait, which supports the use of Bayesian factor analysis to construct the index values. Comparisons with established rule of law measures demonstrate a very high co-variation. In addition, descriptive overviews and correlations with measures of neighboring concepts indicate that the new measure is able to discriminate between the rule of law performance over time and between countries on the one hand and between adherence to the rule of law and neighboring features of good governance on the other hand. The article closes with an empirical examination of the association between de jure constitutional provisions and de facto observance of the rule of law. The findings, based on a conventional country and year fixed effects regression analysis covering most independent countries from 1850 to 2010, identify a negative, substantively weak association. Hence, the results indicate that constitutional provisions, by themselves, do not promote the rule of law in all contexts, such as under autocratic rule.

Throughout the history of political thought, it has been discussed whether the better form of government is one based on the rule of men or on the rule of law (e.g., Aristotle 2000[350BC]: S. 1287b; Bobbio 1987, p. 138; Hayek August von 1960; see also Ferejohn and Pasquale 2003). These ideals of popular sovereignty and the rule of law can conflict with each other when the will of the majority goes against basic doctrines associated with the rule of law. It has been forcefully argued, however, that this does not need to be the case. Both ideals cherish freedom and equality as their underlying and legitimating principles, and the concepts can be conceived as mutually reinforcing, constitutive attributes of liberal democracy (see, e.g., Diamond 1999; Habermas 1996; Merkel 2004; Møller and Skaaning 2011; O’Donnell 2004).

It has been emphasized that the rule of law, including independent courts, is a crucial institutional feature that can help protect democracy, while it has also increasingly been a target for democratic backsliding (Ginsburg 2019). Contemporary diagnoses indicate that fundamental rule of law principles have not gained a foothold or are under duress in many places around the world, where inauspicious conditions and ill intentions undermine the quality of government (Piccone 2020). The interplay between the political regime and the rule of law can take many forms as illustrated by the recent successful and unsuccessful cases of democratic backsliding attempts in Hungary and Turkey on the one hand and Malawi and the USA on the other.

But to assess this relationship empirically and to investigate developments, causes, and consequences of the rule of law more generally, we need high quality measures with a broad coverage in terms of countries and years. A series of democracy measures, covering most countries of the world for more than two centuries, offer themselves for such endeavors (see Coppedge et al. 2020a; Boix et al. 2012; Marshall et al. 2018; Skaaning et al. 2015). Yet, regarding the rule of law, the recent wave of new governance-related datasets has so far not resulted in high quality measures that go far back in time for a large set of countries. This lack of global, historical data is surprising given the broad interest in the topic and the fact that comprehensive scope in terms of countries and years can help us shed new light on empirical trends and relationships.

Moreover, comprehensive reviews have found that extant rule of law measures also suffer from a number of other shortcomings, including conceptual blurriness and multidimensionality, excessive emphasis on property rights and other substantial rights, lack of transparency, problematical sources, and questionable coding and aggregation procedures (Møller and Skaaning 2014; Ginsburg 2011; Haggard et al. 2008; Merkel 2012; Versteeg and Ginsburg 2017; Voigt 2012). This means that there is room for new measures, which are able to provide a better scope than extant alternatives, while offering satisfactory solutions to issues concerning transparency, reliability, and validity.

Against this backdrop, I discuss the conceptualization, measurement, and aggregation (cf. Munck 2009) underlying the Varieties of Democracy (V-Dem) Rule of Law Index, which captures de facto adherence to rule of law principles and covers most polities of the world from 1789 to today. A number of features sustain the validity and reliability of the resulting index scores (see McMann et al. 2022). The index builds on a theoretically informed conceptual basis, which focuses on a limited number of attributes associated with thin understandings of the rule of law. Moreover, the index scores rely on information from multiple indicators based on a sophisticated Bayesian measurement model, which uses the judgements of multiple experts per country and additional information about coder characteristics to calculate point estimates supplemented with estimates of uncertainty in the form of confidence intervals (see Coppedge et al. 2020b). My focus and approach is quite similar to Gutmann and Voigt (2018), who—based on a focus on practice, a thin conceptual understanding, and a series of indicators—proposes a new de facto rule of law index, which they use to investigate empirical relationships. However, their measure is based on data from the World Justice Project, which only offers very limited time-series, and they use it explore the deep roots of the rule of law rather than the association between de jure and de facto adherence to the rule of law.

The article proceeds in the following way. First, I discuss how to define the rule of law and argue that the best choice is to employ a minimalist version that focuses on formal-procedural features, including fair access to justice and rule-abiding government institutions. Second, I present the V‑Dem dataset, and the indicators used to create the Rule of Law Index. Third, I examine the dimensionality and scalability of these indicators. Fourth, I look at the empirical co-variation with other rule of law measures and measures of neighboring concepts and present some descriptive statistics. Finally, I carry out a tentative assessment of whether it makes a difference for actual adherence to the rule of law that the constitution explicitly includes prescriptive statements regarding the rule of law or direct reference to equality before the law, the equal rights of men, or non-discrimination. The results indicate that de jure regulations are not positively related to de facto practices. The contribution of this article is thus three-fold. It describes the underpinnings of the V‑Dem Rule of Law Index in detail, it assesses its validity and reliability, and it applies it to shed new light on the relationship between formal prescriptions and actual practices.

1 Defining the rule of law

A good measure needs to be rooted in a well-justified and clear concept (Sartori 1970, p. 1038). However, the rule of law is an essentially contested concept (Waldron 2002; cf. Gallie 1956), meaning that scholars and laymen disagree vehemently about the defining attributes. My point of departure will be understandings of the rule of law in a Western context. While it is possible to find elements of Western understandings in the legal traditions of other cultures, these tend to be more disputed and less researched (see Brown 1997; Rutner 1992), and they did not figure prominently when the V‑Dem indicators were developed. In general, the Western debate concerns two issues: 1) Does the rule of law only concern features of the rules and the conduct of public authorities, or does it also require ordinary citizens to respect and comply with legal norms? 2) Is the rule of law merely a formal-procedural ideal or is a substantive rights dimension also required? (Raz 1979; Møller and Skaaning 2014; Tamanaha 2004).

A thin (minimalist) conception seems to offer the best foundation for an empirical rule of law measure. On a first note, it speaks in favor of a thin definition that there is widespread agreement about the importance of formal-procedural features, which arguably constitute a common core in virtually all understandings of the rule of law (Møller and Skaaning 2014). Fortunately, the agreement also concerns the specific formal-procedural features. This is illustrated by the striking overlap between the lists of attributes presented in Table 1.

Table 1 Attributes of formal-procedural rule of law conceptions. Sources: Fuller (1969, p. 39), Raz (1979), Finnis (1980), Rawls (1971), and Lauth and Sehring (2009)

The first four lists of attributes are linked to rule of law definitions by influential scholars. The fifth list concerns criteria of a formal Rechtsstaat. The latter is included to show that this well-known German concept is largely identical to a thin understanding of the rule of law. From Table 1 we see that the formal-procedural definitions largely agree to emphasize particular requirements for the character of the laws (general, prospective, clear, consistent, open, compliable, and relatively stable), access to justice (including independent and accessible courts), and that government institutions are ruled by law and subject to it. Laws need to be applied fairly, which means that the administrative application or judicial adjudication of legal prescriptions is consistent across equivalent cases and that is based on procedures that give a chance for the views and interests at stake in each case to be properly voiced (see O’Donnell 2004, p. 33).

Accordingly, the rule of law has much resemblance with what Rothstein and Teorell (2008; see also Rothstein 2012) have termed quality of government, i.e., the extent to which institutions exercising public authority are impartial. It follows that rule of law principles can be violated in different ways. First, the laws themselves are not general, clear, promulgated, prospective, consistent, and compliable. Second, legal institutions and their procedures are not “available to ordinary people to uphold their rights, settle their disputes, and protect them against abuses of public and private power”. Third, people vested with public authority exercise their power “in an arbitrary, ad hoc, or purely discretionary manner on the basis of their own preferences or ideology” (Waldron 2020).

As mentioned above, some do not consider the rule of law as a purely formal-procedural ideal. They rather think that public order and/or substantive rights need to be added to the list of attributes (e.g., Bingham 2010; Dworkin 1985). However, such extensions mean that we open a can of worms. In the words of Waldron (2020):

“Once we open up the possibility of the Rule of Law having a substantive dimension, we inaugurate a sort of competition in which everyone clamors to have their favorite political ideal incorporated as a substantive dimension of the Rule of Law. Those who favor property rights and market economy will scramble to privilege their favorite values in this regard. But so will those who favor human rights, or those who favor democratic participation, or those who favor civil liberties or social justice. The result is likely to be a general decline in political articulacy, as people struggle to use the same term to express disparate ideals.”

This warning is also relevant for the second reason to opt for a thin definition, namely, to reduce conflation with other concepts (Møller 2018; cf. Przeworski 1999). Conceptual distinctiveness supports analytical work, and it has obvious advantages when we are interested in investigating the relationship between different phenomena. Otherwise, associations become (at least partly) true by definition.

It also speaks in favor of a thin definition that empirical one-dimensionality has some benefits. For example, the loss of valuable information in the aggregation process tends to be lower, and the ranking of units based on the resulting scores is less sensitive to the choice of aggregation rule. Moreover, multidimensional measures have a high risk that dissimilar and maybe even reverse dynamics unfold at a lower level of aggregation (Voigt 2012). Subcomponents of unidimensional measures, in contrast, are more likely to show similar trends and associations with other phenomena, so that complexity can be reduced by a composite measure without losing sight of substantial diversity at more disaggregate levels.

Finally, all else equal, the fewer defining attributes, the less is the risk of errors in the measurement exercise (Møller 2016). That said, given strong theoretical reasons, a thin rule of law measure can always be thickened by combining the scores for the fulfilment of formal-procedural features with scores for the fulfilment of selected substantive rights or popular adherence to the law. Accordingly, it is better to settle for a thin understanding of the rule of law to guide the construction of an index that should be relevant for many users.

2 Varieties of democracy: measurement procedures and relevant indicators

The next step is to marshal indicators from the V‑Dem dataset that reflect the features of a formal-procedural conception of rule of law just emphasized. Even though V‑Dem also offers other types of data (mainly observable indicators hand-coded by in-house research assistants), all the V‑Dem indicators used here are based on expert surveys.

The use of standards-based judgements to measure social scientific concepts has faced critique because personal evaluation can introduce measurement error. The fact that all well-known rule of law measures either fully or partly rely on evaluative data as the concept is very difficult to capture adequately with more objective indicators. Important aspects of the rule of law are not directly observable and readily available information tends to be incomplete, inconsistent, and insufficient (Schedler 2012; Skaaning 2018).

V‑Dem seeks to take advantage of country and field specific expert knowledge to capture many important issues at a disaggregate level, while minimizing the risk of systematic and random errors. The online questionnaire contains detailed questions with well-defined response categories. It is divided into different clusters, where the indicators used to measure adherence to the rule of law are taken from the surveys covering civil liberties, the executive, and the judiciary. The carefully selected international and native country experts (mostly academics) are assigned surveys matching their expertise. Typically, a minimum of five experts code each indicator for country-years going back to 1900, while the country-years before 1900 are coded by one or two experts on political history (see Coppedge et al. 2020a).¹

The expert ratings are turned into indicator scores by a Bayesian item response theory (IRT) model. Ratings by “lateral” coders (who rate multiple countries for a limited time period), “bridge” coders (who code the full time series for more than one country), and anchoring vignettes (coders assign scores to hypothetical scenarios) are used to increase comparability over time and across countries. In addition, the measurement model uses information about self-assigned uncertainty and coder characteristics (e.g., gender, education, position, residence, and personal views about democracy) in the calculations (Pemstein et al. 2020).

There are several advantages to this kind of estimation. The measurement model treats the expert ratings as imperfect reflections of latent traits and provides an interval-level estimate of the latent feature. In addition, the IRT model takes into account that experts might have different thresholds for their ratings and adjusts the estimates according to patterns revealed by the data. Finally, the measurement model employs the information about the level of coder agreement, individual coder characteristics, self-assigned uncertainty levels from the coders, the amount of data, and estimates of coder precision to construct confidence intervals that reflect the uncertainty associated with the point estimates. These features of the data-generating process make V‑Dem data a valuable source for constructing a high-quality, comprehensive rule of law index. It is important to note, however, that this procedure does not eliminate the possibility of biases in expert judgments, and some estimates are based on relatively limited information, especially when considering historical periods or areas where little relevant evidence is available (see Knutsen et al. 2024).

Table 2 lists 15 disaggregate V‑Dem indicators that reflect different de facto aspects of the formal-procedural rule of law conception presented above. The first five are from the executive survey, the next four from the civil liberties survey, and the last six from the judiciary survey. The fact that some of them are included in different surveys increases the number of experts involved in coding each country (for the post-1900 period) as many experts only code one or two of these surveys for a particular country.

Table 2 V‑Dem rule of law indicators

In sum, the selected indicators cover formal legality, access to justice, and the subjection of government institutions to the law, while none of them captures the level of public order or respect for substantive rights (such as property rights and freedom from torture). A closer look at the indicators supports that they are in accordance with a series of principles associated with good measurement practices (see Goertz 2005; Munck 2009). They are theoretically relevant and coherent as all indicators refer to core aspects of a thin, formal-procedural understanding of the rule of law.

Moreover, the individual indicators capture rather distinct issues and jointly they tend to cover the defining attributes in a relative exhaustive fashion, although only a single indicator directly addresses the series of features linked to the character of laws. Whether all the indicators tap into a common empirical dimension is another question.

3 Empirical dimensionality and aggregation

The collection of V‑Dem indicators captures relevant variation in adherence to the rule of law across and within countries over the past century. But to what extent do they tap into a common dimension? Although the main purpose is to establish a high degree of concept-measure consistency, there are some analytical advantages associated with empirical one-dimensionality. Highly correlated indicators would therefore lend further support aggregating the selected indicators into a general index.²

To examine the dimensionality more systematically, I calculated the bivariate correlations (Pearson’s r) between the selected indicators for sovereign countries (as defined by the independent states variable in the V‑Dem dataset) in three periods, i.e., 1789–2019, 1789–1899, and 1900–2019. Year 1900 is chosen as threshold because of the different measurement approaches regarding the number and type of experts and to get an impression of whether the patterns of covariation have changed over time. Moreover, I also report the correlations for all polities included in the V‑Dem dataset, which also covers many dependencies and semi-sovereign states.

The results show that the indicators are highly correlated. None of the correlation coefficients are lower than 0.48 and most of them are higher than 0.6 (see Table 7 in the Appendix). The correlations between the indicators directly reflecting corruption and theft of public goods are some of the highest. Unsurprisingly, this also applies to some of the indicators that reflect very similar issues (e.g., access to justice for women/men, judicial independence for high courts/lower-level courts, and compliance with high court/lower-level court decisions). The covariation is slightly higher after 1900 than for the 1789–1899 period, which could be an effect of more coders and more available information. Some of the differences might also be partly due to changes in the context and the sample. The relevance of context and sample is further indicated by the fact that the correlations drop a bit when also including non-independent polities among the observations. That said, all coefficients remain at a high level despite these changes to the sample.

As a further test of empirical dimensionality, I supplement the bivariate correlation with standard factor analyses of similar samples. The loadings and uniqueness scores linked to the first factor are presented in Table 3.

Table 3 Factor loadings

As expected, the eigenvalues are very high, and the first factor accounts for no less than 84–87% of the variation, while the loadings range from 0.69 to 0.91. The factor analyses thus lend additional support to the index construction by showing that the indicators reflect a common latent trait.

The four factor analyses also extract a second factor with eigenvalues slightly above 1. However, it only accounts for less than 11% of the variation. The variables with relatively high loadings on this factor are those paying special attention to corruption. This indicates that the construction of separate index capturing public authority corruption could be valuable (see McMann et al. 2022). Nonetheless, this finding does not undermine the construction of an overarching rule of law index given the overwhelming evidence that also the selected corruption indicators also tap into a common underlying dimension.

To further probe the scalability of the items, I employ the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy. It provides a summary of the size of the partial correlations for each pair of variables relative to the original (zero-order) correlations. As a rule-of-thumb, KMO values larger than 0.8 indicates that factor analysis is useful to combine a set of items. Table 4 shows that all the individual KMO values are at least 0.89, and the overall KMO value is 0.93–0.95.

Table 4 KMO values

Thus, the KMO scores lend strong support for combining the items via factor analysis. Overall Cronbach’s alpha value in the range of 0.96 to 0.97 lends further support to the aggregation of indicators as they corroborate that the internal consistency (scalability) of the indicators is extremely high. Furthermore, the overall value is not improved by excluding any of the indicators. Against this backdrop, the V‑Dem rule of law index is constructed by fitting a Bayesian factor analysis model (including all observations) to the indicators. This procedure takes into account the measurement uncertainty associated with the individual indicators via the method of composition. The model estimates the posterior distribution of the latent factor score for each country-year observation, and these posterior distributions are used to produce point estimates (posterior averages) and confidence intervals (standard deviations and highest posterior density regions).

The index scores are subsequently standardized to a probability (0–1) score by using the normal cumulative distribution function based on the assumption that the latent trait follows a normal distribution. This procedure, which tends to squish values—and smooth over random error—at the extremes, is used to ease interpretation and increase comparability across different V‑Dem indices. It deserves mentioning that while this standardization spreads out the observations, it also leads to some variance truncation in the upper end.³

Given the strong one-dimensionality of the indicators, we would expect the distribution of index scores not to be so sensitive to the method of aggregation. I assess this expectation by correlating the rule of law index based on Bayesian factor analysis with indices based on alternative procedures to aggregate the same list of indicators. Alternative options include using the minimum value (reflecting non-substitutability between the subcomponents), using the arithmetic mean (reflecting partial substitutability), or using the maximum value (reflecting full substitutability) to construct a formative index (see Goertz 2005).⁴ The correlation (Pearson’s r) with the alternative based on averaging the scores is a stunning 0.997, meaning that the relative distributions of index scores are nearly identical. The correlation with the other alternatives is lower but still very high, namely, 0.908 (minimum rule) and 0.866 (maximum rule).⁵ These results indicate that it does not make a big difference whether the subcomponents are combined via factor analysis or through alternative aggregation methods.

Figure 1 shows line graphs with point estimates and confidence intervals for four countries (Sweden, Russia, Japan, and Argentina) from different parts of the world with long time-series and dissimilar trajectories regarding political regimes and socio-economic development. They illustrate that the V‑Dem Rule of Law Index is able to capture long-term as well as short-term developments but also that the estimates are characterized by varying degrees of uncertainty as indicated by the size of the confidence intervals.

Fig. 1

Rule of law trends in four selected countries, 1789–2022

It is beyond the scope of this article to discuss the plausibility of these estimates in detail. It is, however, comforting to note that major events and distinct episodes, which we would expect to be related to the rule of law performance, are reflected in the data. Examples include the unification and initiation of the liberal period in 1862, the military coups in 1939, 1966, and 1976, and Carlos Menem’s corrupt presidency in Argentina; the judicial reform of Alexander II in the 1860s, subsequent reactionary policies under Alexander III, totalitarian rule under Stalin, the fall of the communist regime in 1991, and Putin’s concentration of power after ascending the presidency in 1999 in Russia; the end of the Shogunat in 1868 and reforms following the World War II defeat in Japan; and the abolition of aristocratic prerogative for higher state positions in 1845 and a major revision of the wage system in the civil service in the late 1850s in Sweden. In general, the relative position of these countries also makes sense based on what we know from qualitative accounts: Sweden has shown the overall best performance followed by Japan, while Argentina and especially Russia have mostly been worse off.

Figure 2 broadens the geographical perspective and captures the index scores for all countries in 2022, where the darkest color indicates the strongest adherence to rule of law principles. The overview does not deliver many implausible surprises. There is a lot of variation between and within world regions. Asian, African, and Middle Eastern countries tend to receive lower scores than European (particularly EU members), North American, and Pacific countries, and Latin American countries are generally placed somewhere in the middle. Among the top performers, we find the Nordic countries, Benelux, the Anglophone countries, Germany, Switzerland, Estonia, Singapore, and Uruguay. At the other end of the scale are countries such as North Korea, Equatorial Guinea, Venezuela, South Sudan, Azerbaijan, Eritrea, and Yemen.

Fig. 2

World map depicting rule of law scores for 2022

4 Comparison with extant measures

When introducing a new measure, it is common practice to compare it to available alternatives. I here limit this exercise to three indices with a global reach, where the data providers explicitly refer to them as rule of law measures. One of them is a component of the well-known civil liberties index from Freedom House (FH); the second one of the six worldwide governance indicators (WGI), and the third is the rule of law index by the World Justice Project (WJP). These indices—with some of their most important features summarized in Table 5—have been highlighted as some of the most influential measures today (Versteeg and Ginsburg 2017).

Table 5 Features of selected rule of law measures

One striking difference between the V‑Dem index and the alternatives is the scope. WGI, FH, and WJP cover some microstates not included in the V‑Dem dataset. Then again, V‑Dem includes no less than 231 historical and contemporary polities for the period 1789 until today (average coverage is 113 years). It is clear that the temporal coverage is unmatched as the other measures only go back to 1996, 2005, and 2014, respectively. It is also worth mentioning that, strictly speaking, it is questionable to compare the WGI scores and the WJP scores over time as their estimates are calculated separately for every year based on unit standard variations or normalized values. This means that yearly changes in samples, global means, and ranges make temporal comparisons rest on a shaky foundation.

There are also some noteworthy differences regarding the defining attributes of the rule of concept. Apart from sharing the formal-procedural core, substantive rights and public order are—to various degrees and with somewhat different emphasis—captured by all of the three alternatives. Despite the differences in conceptualization and measurement procedures, the rule of law indices are very highly correlated. The pairwise correlations with the V‑Dem index are 0.88 (FH), 0.89 (WGI), and 0.91 (WJP), respectively. If we recognize the alternatives as good measures of the rule of law, the high levels of covariation validate the V‑Dem Rule of Law Index. It is also possible, though, that all measures are biased in similar ways and then large correlations merely signify a high degree of interchangeability without providing a valid measure of the key concept.

Based on the components, we would expect the Rule of Law Index to offer a relatively clean measurement of core aspects of the rule of law. A correlation of the V‑Dem index with the disaggregate scores for the WJP components supports this claim as the strongest associations are 0.92 with constraints on government (i.e., the extent to which those who govern are bound by law) and 0.88 with regulatory enforcement (i.e., the extent to which regulations are fairly and effectively implemented and enforced). It is also comforting to note that the weakest association is 0.58 with order and security (i.e., the extent to which society assures the security of persons and property), which is conceptually distinct from the thin definition underlying the Rule of Law Index.

The thin understanding of the rule of law means that the index scores should also be distinguishable from measures of other neighboring concepts focusing on democracy and substantive rights. To examine this issue, I use the ten-point Political Competitiveness⁶ measure from Polity IV (Marshall et al. 2018) and the dichotomous democracy measure from Boix et al. (2012) to proxy electoral (thin) democracy. They cover most country-years from 1800 until today. To measure respect for substantive rights, I use the Latent Human Rights Protection Scores (v3) provided by Fariss (2019), which cover the period from 1946 to 2013. This index on physical integrity rights violations combines information from several datasets, including the CIRI Human Rights Database (Cingranelli et al. 2014), which I use a as source for separate indicators on freedom of speech, freedom of assembly and association, and freedom of religion. I use the updated versions included in the CIRIGHTS dataset (Cingranelli et al. 2018). It covers most countries in the period 1980–2017.

The correlations are substantively lower than those between the V‑Dem Rule of Law Index and other rule of law measures (0.52 for political competitiveness; 0.63 for electoral democracy, 0.63 for the latent human rights protection scores; 0.57 for freedom of speech; 0.58 for freedom of assembly and association; and 0.42 for freedom of religion). On the one hand, this finding lends some support to the distinctiveness of the Rule of Law Index. On the other hand, they are still relatively high. This could indicate that experts are influenced by widespread perceptions about how particular countries are doing well on all aspects of “good governance” and vice versa. However, it is also likely that these phenomena are causally connected (they could, for example, be mutually reinforcing) or have common determinants (such as socio-economic development), which would also lead to high covariation. Taken together, the empirical differences tend to be sufficiently large to discriminate between the different concepts and to carry out statistical investigations of their relationships.

5 De jure vs. de facto rule of law

The different tests and overviews presented above support that the emphasized V‑Dem indicators capture a common, latent trait that reflects a minimalist conception of the rule of law in a valid and nuanced way. But do we really need another de facto measure when we already have de jure measures with high reliability and broad coverage available? One way to provide an answer to this question is to examine the relationship between de jure and de facto measures.

Several studies have investigated the connection between de jure constitutional provisions and the de facto observance of human rights (Melton and Ginsburg 2014; Chilton and Versteeg 2016; Law and Versteeg 2013; Keith et al. 2009; Hayo and Voigt 2007; Elkins et al. 2013; Bjørnskov and Mchangama 2019). Keith et al. (2009: 644) phrase the relevance of such studies nicely, when they state that,

“If the law is an effective barrier to human rights abuse (repression), then those who wish to prevent it are well advised to invest their time in establishing the kinds of law that deter it. If the law is no more than a flimsy parchment barrier to rights abuse, then efforts to create legal barriers to repression may be irrelevant: those who wish to prevent human rights abuse should probably concentrate their efforts on other prophylactic measures.”

However, previous studies do not focus on the rule of law as such. Some of them study civil liberties or social rights, while others analyze judicial independence, fair trials, or other partial aspects of the rule of law. These works have found mixed results regarding the association between constitutional provisions and the actual fulfilment of the legal obligations. In addition, they only cover relatively short time spans, meaning that a lot of relevant information is not considered.

Here I carry out a tentative assessment of whether, regarding the rule of law, the degree to which the constitution explicitly includes prescriptive statements is associated with de facto observance. To do so, I make use of the LEGAL_ORDER index from Nardulli et al. (2013), which is a de jure rule of law measure. It covers most country-years from 1850 to 2010 and thus covers almost the entire period of written constitutions inspired by liberal ideas about individual rights and constraints on the exercise of state power.⁷ More particularly, they employ indicators from the CCP, which capture a country’s formal, constitutional commitment to the following issues in a given year: equality before the law; office holders take oath to obey constitution; affirmative duty to obey the constitution; protection from illegal or ultra-vires administrative actions; habeas corpus; ban on punishment without law; and interpretations of the constitution fall to the judiciary. These indicators have been combined into an index via factor analysis.

A simple bivariate correlation between LEGAL_ORDER and the Rule of Law Index produces a coefficient of −0.005. The finding indicates that—regarding the rule of law—de jure provisions and de facto adherence are not strongly associated. This might be a surprising finding to many, but also other studies have tended to find no or very weak relationships between legal prescriptions and actual human rights performance (e.g., Law and Versteeg 2013; Chilton and Versteeg 2016; Bjørnskov and Mchangama 2019; Hayo and Voigt 2007). But the genuine relationship could be suppressed by confounders. What happens if we include the variables in a conventional cross-sectional time-series statistical analysis with the V‑Dem measure as the predicted variable and the LEGAL_ORDER index as the predictor variable? In order to answer this question, I run linear panel regression models, including country and year fixed effects, where standard errors are clustered by country. Some of the models include a battery of standard controls.

The requirement for extensive coverage limits the number of relevant control variables. I use the comprehensive data on GDP/cap. and population size from Fariss et al. (2017), the political competitiveness measure from Polity IV described above, and the total oil income per capita variable from Haber and Menaldo (2012).

Because of the risk of introducing post-treatment bias by the inclusion of control variables, which might to some extent be influenced by the de facto rule of law level, I first run two clean models; one only including the LEGAL_ORDER variable lagged one year and the other including the same index lagged ten years. Subsequently, I run a model with the two control variables expected to be most exogenous (population size and oil income per capita) and a model with all controls (all control variables lagged one year). Two additional models separate the time-series into two—before and after 1945, respectively, where the threshold is inspired by the common practice in comparative politics and international relations to focus on developments after World War II.

The results of the regression analyses lend further evidence to the absence of a strong, positive relationship between de jure and de facto measures of the rule of law. Indeed, we see from Table 6 that the association is consistently negative and statistically significant in all but one model (full model covering coun/try-years before 1945).

Table 6 Results from Regression Analysis

Note that a causal interpretation is not defensible due to risks of unobserved variables bias and endogeneity not accounted for by the analysis. The results might, for example, reflect that in many places, constitutions are merely parchment barriers and those regimes with high propensity for violating rule of law principles have been inclined to introduce such principles as window dressing. However, the predictive power is very low—as indicated by the R² of 1% in the clean models—which indicates that the association is not of substantial importance.

The control variables largely behave as expected. GDP/cap. and especially political competitiveness show a strong, positive relationship with the V‑Dem rule of law index. The inclusion of these variables has a large impact on the predictive power. In contrast, population size does not show a robust relationship with adherence to the rule of law; only in the full model for the period after 1945, the relationship is statistically significant. Oil production also demonstrates a weak covariation with the outcome of interest. Interestingly, the sign flips from positive to negative when analyzing different periods. This result is generally in line with the findings from comprehensive studies on the resource curse, which show that the relationship between natural resource wealth and democracy is insignificant in a long-term perspective (Haber and Menaldo 2012), but that a negative relationship can be identified in the most recent decades (Andersen and Ross 2014).

In sum, the results suggest that there is not a strong relationship between de jure and de facto measures. This means that scholars should not proxy actual rule of law observance with formal constitutional provisions and, accordingly, that there is a need for de facto measures such as the proposed index. Moreover, the findings also question the practice of combining de jure and de facto indicators in aggregate measures of the rule of law.

6 Conclusion

The rule of law is one of the most widely acclaimed ideals. Hardly anyone openly speaks up against principles emphasizing the legitimacy and importance of formal legality, equal access to justice, and the subjection of state power to the law. The rule of law is crucial for our understanding of justice, human well-being, and democracy. Concerning the intersection with the latter, it has been argued that “the rule of law cannot survive without democratic support, even as democracy cannot survive without the rule of law” (Ginsburg 2019, p. 369). However, empirical analyses have hitherto suffered from a shortage of global, historical rule of law data that support the creation of descriptive overviews as well as large-scale explanatory analyses. The V‑Dem Rule of Law Index fills this gap. It rests on a sound conceptual foundation, focusing on a limited set of formal-procedural attributes that most scholars agree on. The new measure lends itself to systematic comparative studies by offering fine-grained aggregate and disaggregate scores for independent countries, semi-sovereign units, and colonial dependencies since 1789.

Empirical tests have shown that the rule of law indicators from V‑Dem tap into a shared latent dimension and that the resulting index is highly correlated with extant rule of law measures with much lower historical coverage. Descriptive illustrations indicate that the new measure is able to identify changes over time and similarities and differences across countries. Moreover, correlations with measures of concepts related to other aspects of good governance have demonstrated that the index is able to discriminate between them although they tend to move in parallel to a substantial degree. Although the V‑Dem data are certainly not perfect, this article offers good news to scholars and other people interested in the rule of law in the form of a new, methodologically solid measure with a comprehensive scope.

The absence of a positive, significant, and substantial covariation between de jure constitutional provisions and de facto observance of the rule of law is bad news for people believing in the power of formal-institutional reforms.⁸ Most likely, the result is mainly driven by the neglect of constitutional rules in autocracies. While a causal interpretation of this finding is not warranted, the result indicates that constitutional provisions, in themselves, are often merely parchment barriers that do little else than function as window dressing in autocratic settings. This is surely not the last word on the topic, but at least we now have access to a global, historical measure of de facto adherence to rule of law principles that—used with caution—can help us to investigate this and other questions about developments, causes, and consequences of the rule of law in large-scale comparative studies.

Appendix

Table 7 Bivariate correlations between V‑Dem rule of law indicators