Price data
Data for spatial price comparisons usually have three dimensions:
- the region \((r=1,\ldots,R)\), which can be a country, city, or any other regional entity,
- the basic heading \((b=1,\ldots,B)\), which is a group of similar products for which expenditure weights, \(w_{b}\) or \(w_{b}^{r}\), are available,
- and the individual product \((n=1,\ldots,N_b)\), for which prices \(p_{bn}^r\) are recorded.
For transaction and supermarket scanner data, purchased quantities \(q_{bn}^r\) of the individual products are often available.
Having such detailed data can pose some additional challenges. Not every individual product might be available in each region. Hence, there are gaps in the data. In the worst case these gaps can result in non-connected price data (World Bank 2013, 98), where price level comparisons of all regions are no longer possible.
Characteristics
The pricelevels-package offers various functions to
characterize price data and to deal with the issue of data gaps. As a
first step, it’s good to check the sparsity (share of gaps) of
a dataset and if the dataset is connected.
Connected price data
Suppose we have prices for 5 products (all belonging to the same
basic heading) in 4 regions. These data are stored in the data.table
dt. Transforming dt into a price matrix (rows:
products; columns: regions) shows that the data exhibit gaps as products
4 and 5 are not priced in regions
c and d, respectively.
# example price data with gaps:
dt <- data.table(
"r"=c(rep(c("a","b"), each=5), rep(c("c","d"), each=3)),
"n"=as.character(c(rep(1:5, 2), rep(1:3, 2))),
"p"=round(runif(16, min=10, max=20), 1))
# price matrix:
dt[, tapply(X=p, INDEX=list(n,r), FUN=mean)]
#> a b c d
#> 1 15.1 17.9 18.3 15.0
#> 2 13.7 19.6 18.1 17.5
#> 3 16.6 15.6 14.2 16.8
#> 4 12.2 18.3 NA NA
#> 5 14.2 17.5 NA NADue to the gaps, the sparsity() of this dataset is
greater 0. As there are only a few gaps, however, function
is.connected() indicates that the dataset is connected. The
function comparisons() provides some further details. It
shows that all four regions form one ‘block of connected regions’ since
they are connected via direct or indirect links. Moreover, the function
output highlights that each region can be directly compared with the
other regions using at least one price ratio.
dt[, sparsity(r=r, n=n)] # sparsity
#> [1] 0.2
dt[, is.connected(r=r, n=n)] # is connected
#> [1] TRUE
dt[, comparisons(r=r, n=n)] # one group of regions
#> Key: <group_id>
#> group_id group_members group_size total direct indirect n_obs
#> <int> <char> <int> <int> <int> <int> <int>
#> 1: 1 a;b;c;d 4 6 0 6 16Non-connected price data
Let’s assume now that prices for products 1,
2, and 3 are no longer available in regions
a and b (for example because these products
are not sold there anymore). This situation is captured in data.table
dt2. In total, the price data still include 4 regions and 5
products. However, the price matrix reveals that the pattern of gaps
changed.
# non-connected example price data:
dt2 <- dt[!(n%in%1:3 & r%in%c("a","b")),]
# price matrix:
dt2[, tapply(X=p, INDEX=list(n,r), FUN=mean)]
#> a b c d
#> 1 NA NA 18.3 15.0
#> 2 NA NA 18.1 17.5
#> 3 NA NA 14.2 16.8
#> 4 12.2 18.3 NA NA
#> 5 14.2 17.5 NA NAThe sparsity increased to 0.5. Even worse, the dataset is no longer
connected. This can be easily seen from the price matrix, where regions
a and b form one block of connected regions,
and regions c and d another but separate
block. This is also highlighted in the output of
comparisons(), which now has two rows.
dt2[, sparsity(r=r, n=n)] # sparsity
#> [1] 0.5
dt2[, is.connected(r=r, n=n)] # is not connected
#> [1] FALSE
dt2[, comparisons(r=r, n=n)] # two groups of regions
#> Key: <group_id>
#> group_id group_members group_size total direct indirect n_obs
#> <int> <char> <int> <int> <int> <int> <int>
#> 1: 1 a;b 2 1 0 1 4
#> 2: 2 c;d 2 1 0 1 6The fact that the dataset dt2 is not connected does not
mean that no price comparison is possible. It only means that a price
comparison of all regions cannot be done. The function output
of comparisons() shows that separate price comparisons
within each block of regions could be done. However, it is clear that a
comparison of the price levels between the two blocks is still not
possible. To follow this approach, the regions in dt2 can
be classified into blocks of connected regions using the function
neighbors(). Otherwise, if one wants to carry out the price
comparison only for the biggest block of connected regions,
dt2 can be connected using function connect(),
as shown below.
# group regions into groups of connected regions:
dt2[, "block" := neighbors(r=r, n=n, simplify=TRUE)]
dt2[, is.connected(r=r, n=n), by="block"]
#> block V1
#> <fctr> <lgcl>
#> 1: 1 TRUE
#> 2: 2 TRUE
# subset to biggest group of connected regions:
dt2[connect(r=r, n=n), is.connected(r=r, n=n)]
#> [1] TRUEBoth approaches allow to perform a price comparison on a subset of the data by excluding some regions.
Simulation
The quality of price data is of utmost importance for the reliability
of a regional price comparison. To assess the impact of gaps on the
reliability of price index numbers, it is helpful to control the amount
of gaps in the price data. At the same time, however, it must be ensured
that the price data stay connected. A simplistic approach of simulating
price data would not guarantee this. Therefore, the
pricelevels-package offers some functions for data
sampling.
In the following, we want to simulate random price data for 5 products within the same basic heading in 4 regions. This can be easily achieved if there are no data gaps. First, products and regions are sampled. Second, random prices are added.
R <- 4 # number of regions
N <- 5 # number of products
# set frame:
dt <- data.table("r"=as.character(rep(1:R, each=N)),
"n"=as.character(rep(1:N, times=R)))
# add random prices:
set.seed(1)
dt[, "p":=round(x=rnorm(n=.N, mean=as.integer(n), sd=0.1), digits=1)]
# price matrix:
dt[, tapply(X=p, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 0.9 0.9 1.2 1.0
#> 2 2.0 2.0 2.0 2.0
#> 3 2.9 3.1 2.9 3.1
#> 4 4.2 4.1 3.8 4.1
#> 5 5.0 5.0 5.1 5.1Data gaps
It is clear that the price data dt are connected.
However, to get more realistic data, we have to introduce gaps. This can
be done randomly as well. If there are only few gaps, the likelihood to
end up with non-connected price data is relatively low. However, with
more gaps, it can happen that the resulting price data are no longer
connected. To overcome this issue, the function rgaps()
introduces gaps into an existing (complete) dataset while ensuring that
the data stay connected. In the following, we introduce approx. 20% gaps
into the price data dt.
# introduce gaps:
set.seed(1)
dt.gaps <- dt[!rgaps(r=r, n=n, amount=0.2), ]
# price matrix:
dt.gaps[, tapply(X=p, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 NA 0.9 1.2 1.0
#> 2 2.0 2.0 2.0 2.0
#> 3 2.9 3.1 2.9 3.1
#> 4 4.2 NA 3.8 NA
#> 5 5.0 NA 5.1 5.1From the price matrix, we see that there is no price for product
1 in region 1. The
rgaps()-function also allows to exclude specific regions
and/or products from the sampling of gaps. If we want, for example, that
there are generally no gaps in region 1, we can implement
this as follows:
# introduce gaps but not for region 1:
set.seed(1)
dt.gaps <- dt[!rgaps(r=r, n=n, amount=0.2, exclude=data.frame(r="1", n=NA)), ]
# price matrix:
dt.gaps[, tapply(X=p, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 0.9 0.9 1.2 NA
#> 2 2.0 2.0 2.0 2.0
#> 3 2.9 3.1 2.9 3.1
#> 4 4.2 NA 3.8 NA
#> 5 5.0 NA 5.1 5.1We see from the price matrix that no price is missing in region
1.
If the gaps should occur with a higher likelihood for specific
regions and/or products, this can be defined as well. The
prob-argument in rgaps() allows to specify for
each observation the (relative) likelihood that gaps occur at this
position in the data. In the following example, gaps will occur more
often for products 4 and 5.
Expenditure weights
The price data in dt represent the data that statistical
offices usually collect as quantities or weights on individual products
are not available. However, transaction data often include purchased
quantities while data at the basic heading level exhibit expenditure
weights as an indication of the relative importance for household
consumption.
As the purchased quantities can be expected to heavily depend on the
prices of products, there is no stand-alone function for the sampling of
random quantities. However, random expenditure weights for basic
headings can be added to some dataset using the function
rweights(). These weights can be the same for each basic
heading in each region, differ across basic headings but are the same
for each region, or differ across basic headings and regions. This is
shown below, where we sample the weights for the individual products
(and not basic headings):
# constant expenditure weights:
dt.gaps[, "w1" := rweights(r=r, b=n, type=~1)]
dt.gaps[, tapply(X=w1, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 0.2 0.2 0.2 0.2
#> 2 0.2 0.2 0.2 0.2
#> 3 0.2 0.2 0.2 0.2
#> 4 NA NA 0.2 0.2
#> 5 NA 0.2 0.2 NA
# weights different for basic headings but same among regions:
dt.gaps[, "w2" := rweights(r=r, b=n, type=~n)]
dt.gaps[, tapply(X=w2, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 0.19621397 0.19621397 0.19621397 0.19621397
#> 2 0.08129106 0.08129106 0.08129106 0.08129106
#> 3 0.50029883 0.50029883 0.50029883 0.50029883
#> 4 NA NA 0.05927477 0.05927477
#> 5 NA 0.16292136 0.16292136 NA
# weights different for basic headings and regions:
dt.gaps[, "w3" := rweights(r=r, b=n, type=~n+r)]
dt.gaps[, tapply(X=w3, INDEX=list(n,r), FUN=mean)]
#> 1 2 3 4
#> 1 0.01418186 0.01418186 0.01418186 0.01418186
#> 2 0.40005184 0.40005184 0.40005184 0.40005184
#> 3 0.12417969 0.12417969 0.12417969 0.12417969
#> 4 NA NA 0.33378579 0.33378579
#> 5 NA 0.12780082 0.12780082 NAIf there are no gaps in the data, the weights always add up to 1.
With gaps, however, the type of weights is prioritized,
meaning that constant weights remain constant even though a
renormalization would have led to different weights across basic heading
and regions. Therefore, the weights do not necessarily add up to 1 in
each region. Of course, the weights can easily be renormalized if
necessary.
Price data
The previous steps to simulate random price data are put together in
the function rdata(). This function easily simulates random
price data for a specified number of regions, basic headings, and
individual products including gaps and weights. However, beside this
easier usage, there are some additional benefits:
- Prices and quantities for each individual product are simulated.
- Sales can be introduced. Some prices and quantities are adjusted by a random factor. The sales are flagged in the data.
- The generation of prices follows a more sophisticated model (see also the NLCPD method below): \[\begin{equation} p_{bn}^r = \pi_{bn} \ (P^r)^{\delta_{bn}} \ u_{bn}^r \ , \end{equation}\] where the price of product \(n\) in region \(r\), \(p_{bn}^r\), is explained by the general product price, \(\pi_{bn}\), the product-specific elasticities \(\delta_{bn}\), and the regional price level, \(P^r\). The error term \(u_{bn}^r\) follows a lognormal distribution.
- The parameters used for the data generating process (i.e., the model
above) are stored by
rdata()and can be added to the function output if desired. This is particularly useful for simulations where some calculated price levels are compared to the ‘true’ price levels underlying the data generation.
The following code snippet shows how easily a connected price dataset with gaps, sales and expenditure weights for \(B=4\) basic headings with \(N_b=(2,2,3,3)\) products in \(R=9\) regions can be simulated.
# simulate random price data:
set.seed(123)
srp <- rdata(
R=9, # number of regions
B=4, # number of product groups
N=c(2,2,3,3), # number of individual products per basic heading
gaps=0.2, # share of gaps
weights=~b, # same varying expenditure weights for regions
sales=0.1, # share of sales
settings=list(par.add=TRUE) # add true parameters to function output
)
# true parameters:
srp$param
#> $lnP
#> 1 2 3 4 5 6
#> 0.01351101 -0.06729194 -0.03758062 -0.02719543 -0.13336083 0.11908720
#> 7 8 9
#> 0.05064581 -0.07850519 0.16068999
#>
#> $pi
#> 01 02 03 04 05 06 07 08
#> 2.151639 2.109964 1.372202 1.754044 5.870784 5.714721 5.906962 1.797949
#> 09 10
#> 1.728033 1.619993
#>
#> $delta
#> 01 02 03 04 05 06 07
#> 1.2247660 1.2530426 1.0527690 0.5926094 2.2100173 1.3200648 -0.3813724
#> 08 09 10
#> 1.4605623 0.6499459 0.2391432
# price data:
head(srp$data)
#> Key: <group, product, region>
#> group weight region product sale price quantity
#> <fctr> <num> <fctr> <fctr> <lgcl> <num> <num>
#> 1: 1 0.07939543 1 01 TRUE 8.13 53705
#> 2: 1 0.07939543 2 01 FALSE 7.90 9765
#> 3: 1 0.07939543 3 01 FALSE 8.29 10992
#> 4: 1 0.07939543 5 01 FALSE 7.36 9829
#> 5: 1 0.07939543 6 01 FALSE 10.02 38338
#> 6: 1 0.07939543 7 01 FALSE 9.20 22230
The graph shows that the individual products’ prices vary across regions and within the basic heading. However, while they are at similar levels within the same basic heading, they can considerably differ between basic headings.