Introduction: Retrieval of Storting data in R

A wide variety of parliamentary data have been made available to the public in several countries over the last decade. Be it through frontend websites or back-end APIs, researchers on parliaments have never had easier access to large amounts of data than they do now. However, both frontend and API scraped data often come in formats (.html, .xml, .json, etc) that require substantial structuring and pre-processing before they are ready for subsequent analyses.

In this vignette, I present the stortingscrape package for R. stortingscrape makes retrieving data from the Norwegian parliament (Stortinget) through their easily accessible back-end API. The data requested using the package require little to no further structuring. The scope of the package, discussed further below, ranges from general data on the parliament itself (rules, session info, committees, etc) to data on the parties, bibliographies of the MPs, questions, hearings, debates, votes, and more.

Although this is the first attempt to make data on Stortinget more easily accessible, stortingscrape does not live in a vacuum. A variety of parliamentary data for different countries are available for researchers to use freely. For parliamentary debates, Thomas, Pang, and Lee (2006) were one of the first to gather and make available data. Their data cover the proceedings of the 2005 House debates. Eggers and Spirling (2014) structured the UK Hansard speech data, which spans from 1802 to 2010. Beelen et al. (2017) provided continuously updated data for the Canadian parliament, Rauh and Schwalbach (2020) made available a collection of speech data from 9 countries, and Turner-Zwinkels et al. (2021) developed a day-by-day dataset of MPs in Germany, Switzerland, and the Netherlands, in the period between 1947 and 2017. These examples are, however, different from stortingscrape in that they are finished datasets ready for download and have limited scope.

The main goal of stortingscrape is to allow researchers to access any data from the Norwegian parliament easily, but also still be able to structure the data according to ones need. Most importantly, the package is facilitated for weaving together different parts of the data.stortinget.no API.

I will start this vignette by briefly discussing the openly accessible data.stortinget.no API. Next, I will describe the philosophy, scope and general usage of the stortingscrape package. Finally, I will present some minimal examples of possible workflows for working with the package, before I summarize.

Stortinget’s API

The Norwegian parliament was comparatively early in granting open access to their data through an API when they launched data.stortinget.no in 2012. The general purpose of the API is to priovide transparency in the form om raw data, mirroring the frontend web-page information from stortinget.no. The format of the API has been fairly consistent over the time of its existance, but there have been some small style changes over different versions.¹ stortingscrape was built under version \(1.6\) of the API.

Except for content that is blocked for the public (e.g. debates behind closed doors), the API contains all recorded data produced in Stortinget. These data include data on individual MPs, transcripts from debates, voting results, hearing input, and much more. For a exhaustive list of all data sources in the API.² The data available in the API can be accessed through XML of JSON format³, both of which are flexible formats for compressing data in nested lists.

As an exmple, the raw data input for general information about a single MP⁴ looks like this:

#> <person>
#>   <respons_dato_tid>2021-08-13T14:59:48.2114895+02:00</respons_dato_tid>
#>   <versjon>1.6</versjon>
#>   <doedsdato>0001-01-01T00:00:00</doedsdato>
#>   <etternavn>Aasen</etternavn>
#>   <foedselsdato>1967-02-21T00:00:00</foedselsdato>
#>   <fornavn>Marianne</fornavn>
#>   <id>MAAA</id>
#>   <kjoenn>kvinne</kjoenn>
#> </person>

This is the typical XML structure in the API, although other parts of the data are more complex in that the XML tree can be nested multiple times. This will be discussed further in the next section.

Package philosophy, scope, and usage

stortingscrape aims to make Norwegian parliamentary data easily accessible, while also being flexible enough for tailoring the different underlying data sources to ones needs. Indeed, contrary to most open source parliamentary speech data, stortingscrape aims at giving the user as much agency as possible in tailoring data for specific needs. In addition to user agency, the package is built with a core philosophy of simplifying data structures, make seamless workflows between different parts of the Storting API, and limit data duplication between functions.

Because a lot of analysis tools in R requires 2 dimensional data formats, the stortingscrape package prioritize converting the nested XML format to data frames, when possible. However, some sources of data from the Storting API are nested in a way which makes retaining all data in a 2 dimensional space either impossible or too verbose. For example, the get_mp_bio() function, which extract a specific MP’s biography by id, has data on MP personalia, parliamentary periods the MP had a seat, vocations, literature authored by the MP, and more. In order to make all these data workable, the resulting format from the function call is a list of data frames for each part of the data. The different list elements are, however, easily combined for different applications of the data.

One of the core thoughts behind the workflow of the package is to make it easy to combine different parts of the API and to extract the data you actually need. To facilitate this, most functions within stortingscrape are built to work seemlessly with the apply() family or control flow constructs in R. Because we do not want to call the API repeatedly, functions that are expected to often be ran repeatedly have a good_manners argument. This will make R sleep for the set amount of seconds after calling the API. It is recommended to set this argument to 2 seconds or higher on multiple calls to the API. Generally, the package is built by the recommendations given by the httr2 package (Wickham 2023)⁵.

Most of the data from Stortinget’s API and frontend web page are interconnected through ids for the various sources (session id, MP id, case id, question id, vote id, etc.). stortingscrape core extraction methods are based around these. One of the major benefits of this is that whether you want to extract, for instance, a single question found on the frontend web page, or all questions for a parliamentary session, the package is flexible enough to suit both needs (see the workflow section). It will also enable users to quickly retreive data from the frontend web-page.⁶

Because of the interconnectedness of the API’s data, there are some overlapping sources of data. For instance, both retreival of MP general information (get_mp()), biography (get_mp_bio()), and all MPs for a session (get_parlperiod_mps()) have the name of the MP in the API, but only get_mp() will return MP names in stortingscrape, because these two data sources are easily merged by the MP’s id (see the workflow section).

The scope of stortingscrape is almost the entire API of Stortinget, with some notable shortcomings. First, there are no functions for dynamically updated data sources, such as current speaker lists (https://data.stortinget.no/dokumentasjon-og-hjelp/talerliste/). Second, as mentioned above, duplicated data i avoided whenever possible. Third, certain unstandardized image sources – such as publication attachment figures – are not supported in the package. And finally, publications from the get_publication() function can be retrieved, but are returned in a parsed XML data format from the rvest package because these data are not standardized across different publications.

There are three overarching sources of data in stortingscrape: 1) Parliamentary structure data, 2) MP data, and 3) Parliamentary activity data. These are, in some/most cases, linked by various forms of ID tags. For example, retrieving all MPs for a given session (get_parlperiod_mps()) will give access to MP IDs (mp_id) for that session, which can be used to extract biographies, pictures, speech activity, and more for those MPs. Next, I will showcase some examples of how a typical workflow for using stortingscrape could look like.

Workflow

In the following section, I will discuss some examples of data extraction with stortingscrape. I start by showing basic extraction of voting data based on vote IDs from the frontend web-page – stortinget.no. Next, I exemplify the large set of period and session specific data by retrieving all MPs for a specific parliamentary period and all interpellations for a specified parliamentary session. Finally, I show how the different functions of the stortingscrape package works together – merging data on cases with their belonging vote results. Note that the vignette is built using the examples in the data folder of the package.⁷

data_files <- data(package = "stortingscrape")$results[,"Item"]
data(list = data_files)

## covid_relief <- get_vote("85196")

head(covid_relief[, c("case_id", "vote_id", "n_for", "n_against", "adopted")])
#>   case_id vote_id n_for n_against adopted
#> 1   85196   17631     1        87   false
#> 2   85196   17632     6        81   false
#> 3   85196   17633    14        74   false
#> 4   85196   17634    42        46   false
#> 5   85196   17635    40        48   false
#> 6   85196   17636    15        73   false

covid_relief$vote_id[which(grepl("217", covid_relief$vote_topic))]

## covid_relief_result <- get_result_vote("17689")

head(covid_relief_result[, c("vote_id", "mp_id", "party_id", "vote")])
#>   vote_id mp_id party_id          vote
#> 1   17689   SSA        H           mot
#> 2   17689   EAG        H ikke_tilstede
#> 3   17689   PTA      FrP           mot
#> 4   17689   DTA        A ikke_tilstede
#> 5   17689  KAAN       SV           for
#> 6   17689  KAND       Sp           for

table(covid_relief_result$party_id, 
      covid_relief_result$vote) |>
  addmargins()
#>      
#>       for ikke_tilstede mot Sum
#>   A    27            21   0  48
#>   FrP   0            12  14  26
#>   H     0            22  23  45
#>   KrF   0             5   3   8
#>   MDG   1             0   0   1
#>   R     1             0   0   1
#>   SV    5             6   0  11
#>   Sp    8            12   0  20
#>   Uav   0             0   1   1
#>   V     0             5   3   8
#>   Sum  42            83  44 169

## parl_periods <- get_parlperiods()
## parl_sessions <- get_parlsessions()

tail(parl_periods[,c("id", "years")])
#>         id     years
#> 15 1965-69 1965-1969
#> 16 1961-65 1961-1965
#> 17 1958-61 1958-1961
#> 18 1954-57 1954-1958
#> 19 1950-53 1950-1954
#> 20 1945-49 1945-1950
tail(parl_sessions[,c("id", "years")])
#>         id     years
#> 34 1991-92 1991-1992
#> 35 1990-91 1990-1991
#> 36 1989-90 1989-1990
#> 37 1988-89 1988-1989
#> 38 1987-88 1987-1988
#> 39 1986-87 1986-1987

parl_periods$id[nrow(parl_periods)]
#> [1] "1945-49"

## mps4549 <- get_parlperiod_mps("1945-49")
head(mps4549[, c("mp_id", "county_id", "party_id", "period_id")])
#>   mp_id county_id party_id period_id
#> 1  AAKU        VA        A   1945-49
#> 2  AARY        AA        A   1945-49
#> 3  ALKJ        He        H   1945-49
#> 4  ALVÅ        Fi      NKP   1945-49
#> 5  AMSK        ST        A   1945-49
#> 6  ANBØ        SF        V   1945-49

## interp0203 <- get_session_questions("2002-2003", q_type = "interpellasjoner")
dim(interp0203)
#> [1] 22 26

## cases <- get_session_cases("2019-2020")

# The case titles are, unfortunately, not translated
cases$root$title_short[48]
#> [1] "Representantforslag om å reversere avgiftsøkninger på alkoholfrie drikkevarer og bevare arbeidsplasser i norsk næringsmiddelindustri og varehandel"

## vote <- get_vote(cases$root$id[48])

vote[, c("case_id", "vote_id", 
         "alternative_vote", 
         "n_for", "n_absent", "n_against")]
#>   case_id vote_id alternative_vote n_for n_absent n_against
#> 1   78686   15404               -1     1       82        86
#> 2   78686   15405            15406    46       82        41
#> 3   78686   15406            15405    41       82        46

## vote_result <- lapply(vote$vote_id, get_result_vote, good_manners = 5)
names(vote_result) <- vote$vote_id

vote_result <- do.call(rbind, vote_result)
head(vote_result[, 3:ncol(vote_result)])
#>         vote_id mp_id party_id          vote permanent_sub_for sub_for
#> 15404.1   15404   SSA        H           mot              <NA>    <NA>
#> 15404.2   15404   EAG        H           mot              <NA>    <NA>
#> 15404.3   15404   PTA      FrP           mot              <NA>    <NA>
#> 15404.4   15404   DTA        A           mot              <NA>    <NA>
#> 15404.5   15404  KAAN       SV           mot              <NA>    <NA>
#> 15404.6   15404   MAA       Sp ikke_tilstede              <NA>    <NA>

table(vote_result$vote, vote_result$party_id,
      dnn = c("Vote result", "Vote ID")) |>
  prop.table(margin = 2) |>
  round(digits = 2)

Summary

In this vignette, I have presented the philosophy, scope, usage, and workflow of the stortingscrape package for R. In sum, stortingscrape makes retrieving data from the Norwegian parliament (Stortinget) more accessible through the back-end API (data.stortinget.no). One core philosophy of the package is to let the user tailor the data to ones needs, while at the same time extracting minimal overlapping data. The scope of the package ranges from general data on the parliament itself (rules, session info, committees, etc) to data on the parties, bibliographies of the MPs, questions, hearings, debates, votes, and more.

List of functions

A list of all functions and their description can be found in the package documentation within R or from github.

Bibliography