Validation

A very important and basic task is the validation of our data: which data is valid in a certain typology? For example, if we have a data source from 2014 that describes data within Europe in 2014 and we currently download data from the Eurostat website in 2020, we may find a discrepancy in how the regions are described. The dataset from 2014 is likely to represent sub-national territories according to their NUTS2013 boundary definition, and the new dataset according to the currently valid NUTS2016 boundary definition. We may not be able to correctly join the two datasets.

The validate function family helps in finding out to conformity of your regional coding with various typologies. In the first CRAN release, only Eurostat typologies are coded, but some of the functions can be used with other typologies, for example, with the ISO-3166-1 and ISO-3166-2 typologies. OECD uses the NUTS typologies for the EU member states, so these typologies are the most frequently used in international statistics.

Eurostat offers so-called correspondence tables to follow boundary changes, recoding and relabelling for all NUTS changes since the formalization of the NUTS typology. Unfortunately, these Excel tables do not conform with the requirements of tidy data, and their vocabulary for is not standardized, either. For example, recoding changes are often labelled as recoding, recoding and renaming, code change, Code change, etc.

The data-raw library contains these Excel tables and very long data wrangling code that unifies the relevant vocabulary of these Excel files and brings the tables into a single, tidy format , starting with the definition NUTS1999. The resulting data file nuts_changes is included in the regions package. It already contains the changes that will come into force in 2021.

The vignette Validating Your Typology gives a real-life example to the use of these functions. Validations are the starting points for Recoding and re-labelling, Aggregation And Re-Aggregation and Disaggregation And Projection operations.

Recoding and re-labelling

In the European Union, the most frequent regional statistical change is recoding and relabelling. Such changes do not affect the statistical data, only the statistical metadata.

For example, the overseas department and region of France, Réunion, was officially renamed La Réunion in 2010: this is relabelling. In earlier sources, you may find data about this region under the identifier Réunion, and later you may find it extended with the French article La Réunion. But the data from Réunion in 2009 and La Réunion in 2012 refers to the same territory. Furthermore, when France changed its administrative boundaries, it changed their official code abbreviations. Although the typology of La Réunion did not change, to indicate that a dataset is comparable across French departments according to the new administrative borders (which did change in many other regions) the geolocational code of this region changed from FR94 to FRA4 in 2013 and to FRY4 in 2016. This is recoding.

The geocodes of Eurostat are elements of a controlled metadata vocabulary. To avoid confusion, when a part of the vocabulary changes, for example, when France is changing some elements in its NUTS regions and their coding, it often changes the complete vocabulary, including for unchanged elements, for clarity. This is why FR94 becomes FRA4 even though it is describing the same overseas department and region, La Réunion.

If you want to work with French or international data that include any statistical information about the overseas departments, you must make sure that observations coded with FR94, FRA4, FRY4 are treated as a time-series of the La Réunion and not as different entities.

Names are often dubious identifiers. La Réunion remains in the ISO-3166-1 and ISO-3166-2 libraries Réunion, which is no longer used as name in NUTS. What makes working with ISO and NUTS vocabularies is that while NUTS is “versioned”, ISO is not, so a correspondence for a given year, for example, for the year 2013 is almost impossible to create. This is why the use of Eurostats NUTS geo codes, which are usually unique in a given year, should be preferred over ISO.

Many of Eurostat’s regional datasets contain recoded geographical IDs that violate the definition of tidy data, because under the same geo column they may contain observations for FR94, FRA4, FRY4. However, this is not a tidy description of the territory. Like in our nuts_changes data, they should be reported in three different columns, code_2010, code_2013 and code_2016. The descriptive metadata code, i.e. row identifier FR94 is compatible with FR71 for example, but not with FRY4.

If you are joining data from different tables, you must not join by their geo code, but by the explicit code_2013 geo codes for the observation described or aggregated according to the NUTS2013 regional boundaries and code_2016 for the NUTS2016. In other words, you must make sure that you join by valid elements in a given version of NUTS.

Another important note: if you join with the raw geo data, you will often have seemingly missing data. You may have data available for FR93, FR94, FRY3, FRY4, with missingness for the first two in 2016 and missingness for the latter two in 2010, but in fact, these observations or related to two, unchanged territorial units (Guyane and La Réunion.) You must choose one consistent coding, either using only FRY4 for La Réunion or FR93 for La Réunion. You can also identify FR93, FRY3 explicitly with the name Guyane. However, you must choose between the use of La Réunion or Réunion.

data(nuts_changes)
overseas <- nuts_changes[ 
  which(nuts_changes$geo_name_2016 %in% c("La Réunion", "Guyane")),
  ]

knitr::kable(
  overseas[, c("code_2010", "code_2013", "code_2016",
               "geo_name_2010", "geo_name_2013", "geo_name_2016")]
  )

This is in fact a simpler recoding than the earlier example of Budapest. La Réunion did change its name and code, so you find its data under different labels, but it is still within the NUTS2 typology. Its comparators are likely to be found in the same statistical table. In the case of Budapest, the name did not change, but the change of the geo label reflects the fact that Budapest was moved from the NUTS3 typology to the NUTS2 typology, because it can be much better compared with NUTS2 regions. For data concerning the time period before 2017, you need to look for data about Budapest in datasets that follow the NUTS3 typology, or in national data sources, but you can fill in these historical data to pre-2017 cells of your NUTS2 regional data. The size of Budapest did not change that much - it should have been compared with NUTS2 regions in 2014, like in 2020, and not with smaller NUTS3 units.

See in more detail the vignette Recoding & Relabelling for our programmatic solution.

Disaggregation And Projection

Often, we have missing data about smaller territorial units, and we would like to impute their wider region’s representative data to a higher density dataset. For example, we do not have data for Schwabia, which is a region of the German federal state of Bavaria, and therefore we would like to impute the Bavarian (NUTS1) level data to this NUTS2 level missing observation for Schwabia. This is a valid imputation technique for non-aggregating descriptive statistics, such as mean or median values derived from surveys.

A typical problem of missingness is that we are comparing NUTS1 or NUTS2 level regions of Europe, and we do not find information about the small members states of Cyprus, Estonia, Luxembourg, Malta, because, due to their size, these countries do not have such divisions. Because of their size and territorial integrity, they can be well compared with Schwabia, for example, but the data is missing in a NUTS2 data table. In this case, we may want to impute the technical NUTS0 (country) level data into the NUTS2 data table, creating the technical NUTS2 data of CY00, EE00, LU00 or MT00, which is identical to the national CY, EE, LU, MT observations. In this case, the imputation is valid to any type of data, even to count data like population, because EE = (NUTS0) EE0 (NUTS1) = EE00 (NUTS2).

The function impute_down creates a generic solution to the problem when we want to project data from a larger territorial unit to a smaller one. For example, some state level data is missing for certain Australian or German federal states, and we want to use the national (average) values for imputation.

The function impute_down_nuts gives and EU-specific solution, with a single data frame as input. The mixed_nuts_example data table shows a mixture of potential problems in a small, fictional dataset with random numbers. For simplicity, we take the fictional, country-level data from Malta, and input it down to all potential sub-divisions.

Malta is a very transparent example, because the country has only 2 subdivisions on the lowest, NUTS3 level, i.e. MT001 and MT002. You can find Malta among countries (technical NUTS0 level) as MT, among NUTS1 regions as MT0 (because there are no subdivisions on this level), and among NUTS2 regions again as MT00. For any data that cannot be assigned for a geographical location or area, the technical codes ending with Z are used.

So, what are our possibilities if we have only country-level MTdata, but we would like to use it in a NUTS3 database and impute this variable to both MT001 and MT002. (You can safely do this with average values taken from a survey, but you should not do this with additive values such as the population.)

data("mixed_nuts_example")
mixed_nuts_example %>%
  dplyr::filter(substr(geo,1,2) == "MT") %>%
  knitr::kable()

Malta offers two technical imputation slots on NUTS1 level, MT0 for territorially attributed data, and MTZ for non-territorial exceptions. On NUTS2 level, MT00 and MTZZ. And at last, what is a real imputation, you can impute the MT = MT0 = MT00 values to MT001, should you have only data for MT002, etc. (In this simple example, we impute the MT values to both subdivisions, MT001 and MT002.)

data("mixed_nuts_example")
impute_down_nuts(mixed_nuts_example) %>%
  dplyr::filter(substr(geo,1,2) == "MT") %>%
  knitr::kable ()

Aggregation And Re-Aggregation

The European Union sometimes publishes data on NUTS2 level regions, but not on the larger NUTS1 level macroregions or provinces. In many cases, this information is fully present in the NUTS2 datasets, and a simple aggregation can produce a NUTS1 level dataset, because each NUTS2 region is strictly assigned to one broader NUTS1 macroregion. For count-type data, we can simply aggregate the constituent NUTS2 level data to NUTS1 level.

A frequent problem is the re-assignment of a smaller territorial units, such as lower level administrative unit or a lower NUTS unit to another larger unit, or a pre-existing larger unit.

For example, LT02, i.e. Vidurio ir vakarų Lietuvos regionas in Lithuania consists of the former LT00 (undivided) Lithuania minus LT00A, i.e. Vilnius. LT01, or Sostinės regionas, i.e. the metropolitan area of Vilnius is a separate region from 2016. If you have national and NUTS3 or Vilnius municipality level data from Lithuania, you can present it according to the undivided NUTS2013 definition or the more granular NUTS2016 definition.

The Southern region of Ireland, IE05 was created in the same year from the earlier NUTS3 level regions IE023, IE024, and IE025. You can fill in the missing data for earlier years with the simple additive formula IE05=IE023+IE024+IE025 for count/additive variables, or you can create a re-weighted average for average values.

This information can be found in the change_2016 of the nuts_changes dataset.

data(nuts_changes)
southern <- nuts_changes[ 
  which(nuts_changes$geo_name_2016 == "Southern"), ]
knitr::kable(
  southern[, c("code_2013", "code_2016", "geo_name_2016", "change_2016")]
  )

Boundary Changes With No Solution

In some cases the boundary change cuts across all territorial typologies that re-configuration is impossible from aggregated data. Unless we have access to the individual level data (which is not within scope of the tasks helped by this package), it is not possible to create time-wise comparable data for these units. We must treat the observations on these units logically missing, which may or may not be imputed with any techniques. It is usually helpful to mark territorial observations in panels or time series that have such a structural change.

Time-series

Because of the complexity of sub-national data, there are likely to be missing elements in time-wise comparisons. For example, we may find that some data is available for many years on a German federal level, and for most states, but a particular annual data is missing for Bavaria. It would be tempting to use a simple interpolation of the available data, but often a naïve interpolation is not the best method, because, as we have seen, often the data is available at a different aggregation level or under a different metadata label. It is imperative to first make sure that we exploit all information that is present in a different territorial structure for any given year, and then start inter-temporal imputation with interpolation, or carry forward/backward, forecasting and backcasting algorithms.

At last, we may go back to the cross-sectional data, and may fill in some cross-sectional data gaps with interpolated or extrapolated data. Aggregating an estimate for Bavaria from previous or next-years constituent lower level (NUTS2 or NUTS3) data is very likely to be a far better estimate than any imputation results, for example, a simple substitution of a median value from all German or EU NUTS1 units. The territorial aggregation structure gives plenty of exploitable information.

Citing the data sources

Eurostat data: cite Eurostat.

Administrative boundaries: cite EuroGeographics.

Citing the eurostat R package

For main developers and contributors, see the package homepage.

This work can be freely used, modified and distributed under the BSD-2-clause (modified FreeBSD) license:

citation("eurostat")
#> 
#> Kindly cite the eurostat R package as follows:
#> 
#>   (C) Leo Lahti, Janne Huovari, Markus Kainu, Przemyslaw Biecek.
#>   Retrieval and analysis of Eurostat open data with the eurostat
#>   package. R Journal 9(1):385-392, 2017. Version 3.7.5 Package URL:
#>   http://ropengov.github.io/eurostat Manuscript URL:
#>   https://journal.r-project.org/archive/2017/RJ-2017-019/index.html
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Misc{,
#>     title = {eurostat R package},
#>     author = {Leo Lahti and Janne Huovari and Markus Kainu and Przemyslaw Biecek},
#>     journal = {R Journal},
#>     volume = {9},
#>     number = {1},
#>     pages = {385-392},
#>     year = {2017},
#>     url = {https://journal.r-project.org/archive/2017/RJ-2017-019/index.html},
#>     note = {Version 3.7.5},
#>   }

Citing the regions R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("regions")
#> 
#> To cite package 'regions' in publications use:
#> 
#>   Daniel Antal (2021). regions: Processing Regional Statistics. R
#>   package version 0.1.8. https://regions.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {regions: Processing Regional Statistics},
#>     author = {Daniel Antal},
#>     year = {2021},
#>     note = {R package version 0.1.8},
#>     url = {https://regions.dataobservatory.eu/},
#>   }

Contact

For contact information, see the package homepages.