Introduction DescriptiveStats.OBeu

Aikaterini Chatzopoulou, Kleanthis Koupidis, Charalampos Bratsas

2017-09-17

Introduction

This package was created in order to enable the calculation of descriptive statistical measures, for the needs of a European project. “DescriptiveStats.OBeu” includes functions for measuring central tendency and dispersion of numeric variables along with their distributions and correlations and the frequencies of categorical variables for a given dataset. This document introduces you to DescriptiveStats.OBeu’s basic set of tools.

Input

The input dataset of the main function is a JSON link or a JSON file or a text in JSON format. There are different parameters that a user could specify (see the package specification) and interact with the results.

The parameters: “dimensions”, “measured.dimensions” and “amounts” should be defined by the user, to form the dataset. Then an automated process calculates the basic descriptive measures of tendency and spread, boxplot and histogram parameters in order to describe and visualize the distribution characteristics of the desired dataset.

A sort description about the inputs.
Input Description

json_data

The json string, URL or file from Open Spending API

dimensions

The dimensions of the input data

amounts

The measures of the input data

measured.dimensions

The dimensions to which correspond amount/numeric variables

coef.outl

Determines the length of the “whiskers” plot. Default is 1.5.

box.outliers

If TRUE the outliers will be computed at the selected “coef.outl” level

box.width

The width level is determined 0.15 times the square root of the size of the input data.

cor.method

The correlation coefficient method to compute: “pearson” (default),“kendall” or “spearman”.

freq.select

One or more nominal variables to calculate their corresponding frequencies.

Pre-processing of input

DescriptiveStats.OBeu includes functions that automatically calculates the central tendency and spread measures, the boxplot, histogram and barplot visualization parameters and the correlation matrix of the input fiscal dataset.

The final returns are the parameters needed for forming summary tables of central tendency and dispersion measures and visualizing boxplot, histogram, barplot and correlation matrix of the input data.

Central Tendency Measures

Central Tendency Measures describe the central position of a distribution for a group of data. The basic measures are the mean and the median.

Dispersion Measures (Measures of Spread)

Dispersion measures describe how similar or varied the data are. The range, quartiles and the interquartile range, variance and standard deviation are measures of spread.

Output

The output of this process is a list in json format divided into four components of parameters and results with the first subcomponents.

Here is a sort description of the outputs in each function:

Output descriptions in each function.
Function Output Description

statistics
  • Min
  • Max
  • Range
  • Mean
  • Median
  • Quantiles
  • Variance
  • StandardDeviation
  • Skewness
  • Kurtosis
  • The minimum observed value of the input data
  • The maximum observed value of the input data
  • The difference between maximum and minimum
  • The average value of the input data
  • The median value of the input data
  • The 25%, 75% percentiles
  • The variance of the input data
  • The standard deviation of the input data
  • The Skewness of the input data
  • The Kurtosis of the input data

boxplot
  • lo.whisker
  • lo.hinge
  • median
  • up.hinge
  • up.whisker
  • box.width
  • lo.out
  • up.out
  • n
  • Lower horizontal line out of the box
  • Lower horizontal line of the box
  • Horizontal line in the box
  • Upper horizontal line of the box
  • Upper horizontal line out of the box
  • The box width of each variable
  • Lower outliers
  • Upper outliers
  • The number of non-NA observations

histogram
  • cuts
  • counts
  • mean
  • median
  • The boundaries of the histogram classes
  • The frequency of each histogram class
  • The average value of the input vector
  • The median value of the input data

frequencies
  • Variable name
  • frequencies
  • “_row"
  • relative.frequencies
  • The name of the calculated variable
  • The frequency value
  • Name of the categories of the variable
  • Relative frequency values

correlation
  • Variable name
  • Correlation value
  • “_row"
  • The name of the calculated variable
  • The correlation value
  • The corresponding correlation variable

Examples

Simple examples for each function are provided, in order for the user to understand the use and how to deal with these functions.

In some examples specific columns of the dataset are selected in order for smaller amount of results to be presented. The dataset that is being used is available in DescriptiveStats.OBeu package and represents the budget for Wuppertal for 2009 to 2018.

Statistics

We often need the basic descriptive statistics for the data We are working with. The ds.statistics() function can provide these results.

library(DescriptiveStats.OBeu)
ds.statistics(Wuppertal_df[,-4])
## $Min
## $Min$Amount
## [1] -2040681
## 
## $Min$ProduktbereichNR
## [1] 11
## 
## $Min$ProduktgruppeNR
## [1] 1101
## 
## 
## $Max
## $Max$Amount
## [1] 507995000
## 
## $Max$ProduktbereichNR
## [1] 71
## 
## $Max$ProduktgruppeNR
## [1] 7101
## 
## 
## $Range
## $Range$Amount
## [1] 510035681
## 
## $Range$ProduktbereichNR
## [1] 60
## 
## $Range$ProduktgruppeNR
## [1] 6000
## 
## 
## $Mean
## $Mean$Amount
## [1] 6171229
## 
## $Mean$ProduktbereichNR
## [1] 30.11422
## 
## $Mean$ProduktgruppeNR
## [1] 3020.384
## 
## 
## $Median
## $Median$Amount
## [1] 736038.1
## 
## $Median$ProduktbereichNR
## [1] 31
## 
## $Median$ProduktgruppeNR
## [1] 3103
## 
## 
## $Quantiles
## $Quantiles$Amount
##       25%       75% 
##  243696.1 2653000.0 
## 
## $Quantiles$ProduktbereichNR
## 25% 75% 
##  11  51 
## 
## $Quantiles$ProduktgruppeNR
##  25%  75% 
## 1130 5102 
## 
## 
## $Variance
## $Variance$Amount
## [1] 7.771069e+14
## 
## $Variance$ProduktbereichNR
## [1] 313.2779
## 
## $Variance$ProduktgruppeNR
## [1] 3116183
## 
## 
## $StandardDeviation
## $StandardDeviation$Amount
## [1] 27876637
## 
## $StandardDeviation$ProduktbereichNR
## [1] 17.69966
## 
## $StandardDeviation$ProduktgruppeNR
## [1] 1765.272
## 
## 
## $Kurtosis
##           Amount ProduktbereichNR  ProduktgruppeNR 
##       160.151906        -1.413781        -1.412865 
## 
## $Skewness
##           Amount ProduktbereichNR  ProduktgruppeNR 
##       11.4762126        0.2965627        0.2976537

Histogram

ds.hist() gives the parameters of the numeric input vector.

histogram <- ds.hist(Wuppertal_df$Amount)
histogram
## $cuts
##  [1] -5.0e+07  0.0e+00  5.0e+07  1.0e+08  1.5e+08  2.0e+08  2.5e+08
##  [8]  3.0e+08  3.5e+08  4.0e+08  4.5e+08  5.0e+08  5.5e+08
## 
## $counts
##  [1]   46 6032   83   30   10    0    1   11    2    4    4    2
## 
## $mean
## [1] 6171229
## 
## $median
## [1] 736038.1
str(histogram)
## List of 4
##  $ cuts  : num [1:13] -5.0e+07 0.0 5.0e+07 1.0e+08 1.5e+08 2.0e+08 2.5e+08 3.0e+08 3.5e+08 4.0e+08 ...
##  $ counts: int [1:12] 46 6032 83 30 10 0 1 11 2 4 ...
##  $ mean  : num 6171229
##  $ median: num 736038

Boxplot

The ds.boxplot() function gives the statistics needed for a boxplot to be drawn.

ds.boxplot(Wuppertal_df$Amount,out.level = 0)
## $data
## $data$lo.whisker
## [1] -2040681
## 
## $data$lo.hinge
## [1] 243696.1
## 
## $data$median
## [1] 736038.1
## 
## $data$up.hinge
## [1] 2653000
## 
## $data$up.whisker
## [1] 507995000
## 
## $data$box.width
## [1] 11.83
## 
## $data$lo.out
## NULL
## 
## $data$up.out
## NULL
## 
## $data$n
## [1] 6225

Correlation

This function calculates the correlation coefficient of the input data.

ds.correlation(Wuppertal_df)
##                  Year Amount ProduktbereichNR ProduktgruppeNR
## Year                1   0.02             0.01            0.01
## Amount              0   1.00             0.15            0.15
## ProduktbereichNR    0   0.00             1.00            1.00
## ProduktgruppeNR     0   0.00             0.00            1.00

Frequency

ds.frequency() provides the frequencies and the relative frequencies of factors/characters of the input dataset.

ds.frequency(Wuppertal_df[,2:3])
## $frequencies
## $frequencies$Kontotyp
##         Var1 Freq
## 1 Aufwendung 3197
## 2     Ertrag 3028
## 
## $frequencies$Art
##       Var1 Freq
## 1 Ergebnis 2739
## 2     Plan 3486
## 
## 
## $relative.frequencies
## $relative.frequencies$Kontotyp
##         Var1      Freq
## 1 Aufwendung 0.5135743
## 2     Ertrag 0.4864257
## 
## $relative.frequencies$Art
##       Var1 Freq
## 1 Ergebnis 0.44
## 2     Plan 0.56

Kurtosis

This function calculates kurtosis of the input data.

ds.kurtosis(Wuppertal_df)
##             Year           Amount ProduktbereichNR  ProduktgruppeNR 
##       -0.8371997      160.1519064       -1.4137810       -1.4128654

Skewness

This function calculates skewness of the input data.

ds.skewness(Wuppertal_df)
##             Year           Amount ProduktbereichNR  ProduktgruppeNR 
##        0.1669875       11.4762126        0.2965627        0.2976537

Open spending

The last function open_spending.ds() extracts and analyze the json data provided from Open Spending API, using ds.analysis() function.

Wuppertal_openspending
## [1] "http://next.openspending.org/api/3/cubes/4b6d969e07ef7a86aa54e539fc127a14:wuppertalhaushalt/facts"
descript = open_spending.ds(
  json_data =  Wuppertal_openspending, 
  dimensions ="functional_classification_3.Produktgruppe|date_2.Year",
  amounts = "Amount"
  )
# Pretty output using prettify of jsonlite library
jsonlite::prettify(descript,indent = 2)
## {
##   "descriptives": {
##     "Min": {
##       "Amount": [
##         533.21
##       ]
##     },
##     "Max": {
##       "Amount": [
##         2997043.49
##       ]
##     },
##     "Range": {
##       "Amount": [
##         2996510.28
##       ]
##     },
##     "Mean": {
##       "Amount": [
##         659132.4457
##       ]
##     },
##     "Median": {
##       "Amount": [
##         476400.565
##       ]
##     },
##     "Quantiles": {
##       "Amount": [
##         313924.26,
##         656962.815
##       ]
##     },
##     "Variance": {
##       "Amount": [
##         469375540712.697
##       ]
##     },
##     "StandardDeviation": {
##       "Amount": [
##         685109.8749
##       ]
##     },
##     "Kurtosis": [
##       5.7675
##     ],
##     "Skewness": [
##       2.5221
##     ]
##   },
##   "boxplot": {
##     "Amount": {
##       "lo.whisker": [
##         533.21
##       ],
##       "lo.hinge": [
##         306296.49
##       ],
##       "median": [
##         476400.565
##       ],
##       "up.hinge": [
##         658308.4
##       ],
##       "up.whisker": [
##         1185907.2
##       ],
##       "box.width": [
##         1.5
##       ],
##       "lo.out": [
## 
##       ],
##       "up.out": [
##         2954238.51,
##         2979998.49,
##         2992244.95,
##         2916160.36,
##         2885816.5,
##         2997043.49,
##         2875275.56,
##         1252420.49,
##         1248584.45
##       ],
##       "n": [
##         100
##       ]
##     }
##   },
##   "histogram": {
##     "Amount": {
##       "cuts": [
##         0,
##         500000,
##         1000000,
##         1500000,
##         2000000,
##         2500000,
##         3000000
##       ],
##       "counts": [
##         54,
##         32,
##         7,
##         0,
##         0,
##         7
##       ],
##       "mean": [
##         659132.4457
##       ],
##       "median": [
##         476400.565
##       ]
##     }
##   },
##   "frequencies": {
##     "frequencies": {
##       "functional_classification_3.Produktgruppe": [
##         {
##           "Var1": "",
##           "Freq": 2
##         },
##         {
##           "Var1": "(entfallen in 2013) Geschäftsbereichsleitung GB 1.1 ",
##           "Freq": 5
##         },
##         {
##           "Var1": "Beschäftigtenvertretung",
##           "Freq": 1
##         },
##         {
##           "Var1": "Bezirksvertretungen",
##           "Freq": 7
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 1",
##           "Freq": 15
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 2.1",
##           "Freq": 7
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 2.2",
##           "Freq": 7
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 4",
##           "Freq": 28
##         },
##         {
##           "Var1": "Gleichstellung von Frau und Mann",
##           "Freq": 7
##         },
##         {
##           "Var1": "Politische Gremien",
##           "Freq": 7
##         },
##         {
##           "Var1": "Verwaltungsführung",
##           "Freq": 14
##         }
##       ],
##       "date_2.Year": [
##         {
##           "Var1": "2009",
##           "Freq": 16
##         },
##         {
##           "Var1": "2010",
##           "Freq": 15
##         },
##         {
##           "Var1": "2011",
##           "Freq": 14
##         },
##         {
##           "Var1": "2012",
##           "Freq": 14
##         },
##         {
##           "Var1": "2013",
##           "Freq": 15
##         },
##         {
##           "Var1": "2014",
##           "Freq": 13
##         },
##         {
##           "Var1": "2015",
##           "Freq": 13
##         }
##       ]
##     },
##     "relative.frequencies": {
##       "functional_classification_3.Produktgruppe": [
##         {
##           "Var1": "",
##           "Freq": 0.02
##         },
##         {
##           "Var1": "(entfallen in 2013) Geschäftsbereichsleitung GB 1.1 ",
##           "Freq": 0.05
##         },
##         {
##           "Var1": "Beschäftigtenvertretung",
##           "Freq": 0.01
##         },
##         {
##           "Var1": "Bezirksvertretungen",
##           "Freq": 0.07
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 1",
##           "Freq": 0.15
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 2.1",
##           "Freq": 0.07
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 2.2",
##           "Freq": 0.07
##         },
##         {
##           "Var1": "Geschäftsbereichsleitung GB 4",
##           "Freq": 0.28
##         },
##         {
##           "Var1": "Gleichstellung von Frau und Mann",
##           "Freq": 0.07
##         },
##         {
##           "Var1": "Politische Gremien",
##           "Freq": 0.07
##         },
##         {
##           "Var1": "Verwaltungsführung",
##           "Freq": 0.14
##         }
##       ],
##       "date_2.Year": [
##         {
##           "Var1": "2009",
##           "Freq": 0.16
##         },
##         {
##           "Var1": "2010",
##           "Freq": 0.15
##         },
##         {
##           "Var1": "2011",
##           "Freq": 0.14
##         },
##         {
##           "Var1": "2012",
##           "Freq": 0.14
##         },
##         {
##           "Var1": "2013",
##           "Freq": 0.15
##         },
##         {
##           "Var1": "2014",
##           "Freq": 0.13
##         },
##         {
##           "Var1": "2015",
##           "Freq": 0.13
##         }
##       ]
##     }
##   },
##   "correlation": {
## 
##   }
## }
##

Conclusion

These seven functions provide the basic statistics of data. Even inexperienced users can handle them, in order to evaluate their data and be able to visualize them.