Introduction

The metaggR package implements the knowledge-weighted estimate proposed in Palley and Satopää (2021). This procedure aggregates judges’ estimates of a continuous outcome. To do so, each judge is asked to provide two responses:

1. An estimate of the outcome, and
2. a prediction of the other judges’ average estimate of the outcome.

Aggregation is done with the function knowledge_weighted_estimate(E,P) that inputs J judges’ estimates of the outcome (E) and predictions of others (P), and outputs the knowledge-weighted estimate. In the following two sections we will illustrate this aggregator on a simple example and on experimental data included in the package.

Example 1: Three Gorges Dam

This section illustrates metaggR on the Three Gorges Dam example in Palley and Satopää (2021). First, we will upload the library:

library(metaggR)

Next, we can aggregate the judges’ estimates:

# Judges' estimates:
E1 = c(50, 134, 206, 290, 326, 374)
# Judges' predictions of others:
P1 = c(26, 92, 116, 218, 218, 206)
# Knowledge-weighted estimate:
knowledge_weighted_estimate(E1,P1)
#> [1] 329.305

Therefore the final knowledge-weighted estimate is 329.305.

Example 2: Calorie Counts

This section illustrates metaggR on real-world data collected in an experiment in Palley and Satopää (2021). In this experiment participants were presented with 36 different pictures of food from different restaurants and were asked to estimate the total number of calories in these dishes. Each response involves three steps:

1. Initial Estimates: On the first screen the participant was presented with a picture of a meal and asked How many calories do you think are in this meal?
2. Predictions of Others: On the second screen the participant saw the same picture, was reminded of their previous estimate, and given the statement: We will be showing this picture to other participants as well. Just as we did with you, we will ask them how many calories they believe are in this meal. The participant was then asked to predict How many calories do you think that others will guess on average?
3. Final Estimates: On the third screen the participant saw the same picture again and was asked After having reflected on others, what is y our own final best estimate of the number of calories in this meal?

The metaggR package includes the data in the calories dataset. This dataset is a list with 4 elements:

1. true_calories: A vector of true calorie counts of each 36 meals.
2. estimates_initial: A list of the judges’ initial estimates of the calorie counts in each of the 36 meals.
3. estimates_final: A list of the judges’ final estimates of the calorie counts in each of the 36 meals.
4. predictions_of_others: A list of the judges’ predictions of the others’ average estimate of the calorie counts in each of the 36 meals.

The elements of each member of calories correspond to the same meal. Specifically, the jth elements of true_calories, estimates_initial, estimates_final, and predictions_of_others represent the true calories, initial estimates, final estimates, and predictions of others of the jth meal.

To illustrate, we will consider the responses given for the first meal. First, we will load the package and the calories dataset:

library(metaggR)
data("E_CALORIES_INITIAL")
data("E_CALORIES_FINAL")
data("P_CALORIES")
data("THETA_CALORIES")

Next, we will pick out the responses given for the first meal:

meal = 1
# True number of calories in the first meal:
(theta = THETA_CALORIES[meal])
#> [1] 990
# Judges' initial estimates of the number of calories in the first meal:
(E_initial = E_CALORIES_INITIAL[[meal]])
#>  [1]  140  351  450   20  800 1200   50  475  850  330 1000  600  340  950  600
#> [16]  300 1250  750 1200  670  500 1200  520   75  300  900   25  800 1000  500
#> [31]  900  825 1200  300  600 5000  800  600  800  780 1000  900  630 2000  950
#> [46]  600  800  550 1200  900  950  250  900  800  900 2000  800 1200   40  760
#> [61]  350 1200 1800  725  650 1200  800  500  700  450 1000  400  200
# Judges' final estimates of the number of calories in the first meal:
(E_final = E_CALORIES_FINAL[[meal]])
#>  [1]  122  654  400   10  850 1200   35  475  850  375  900  550 2000  950  600
#> [16]  300 1250  750 1200  670  700 1200  540   95  310  950   25  850 1200  500
#> [31]  900  850 1250  300  600 5000  800  600  800  810  500  900  625 1800  950
#> [46]  750  800  500 1150  900  950  250  950  800  900 2150  800 1150   60  760
#> [61]  350 1100 1800  725  600 1200  800  500  600  475 1000  400  200
# Judges' predictions of others' average estimate of the number of calories in the first meal:
(P = P_CALORIES[[meal]])
#>  [1]  112  654  300   19  900 1200   40  545  800  250  800  500  300 1200  600
#> [16]  300 1250  599 1000  700  600 1000  580   80  360 1000   30  900 1800  700
#> [31]  900  650 1600  300  650 5000  750  600  500  860  500  900  600 1500  950
#> [46]  700  800  500 1000 1000  900  250 1000  800  700 2200  600 1000   50  980
#> [61]  400 1000 1700  750  450 1200  600  500  500  400  800  300  200

A total of 73 judges provided responses for this meal. The true calorie count is 990 calories. The first judge under-estimated the calorie count and provided an initial and final estimates of 140 and 122 calories, respectively. This judge predicted that the others’ average estimate is 112 calories. The root-mean-squared-errors (RMSE) of the initial and final estimates, and their knowledge-weighted estimates are:

# RMSE of the initial estimates:
sqrt(mean((E_initial-theta)^2))
#> [1] 670.6013
# RMSE of the final estimates:
sqrt(mean((E_final-theta)^2))
#> [1] 674.0261

# RMSE of the knowledge-weighted estimate based on judges' initial estimates:
(KWE1 =  knowledge_weighted_estimate(E_initial, P, no_inf_check = TRUE))
#> [1] 814.6392
sqrt((KWE1 - theta)^2)
#> [1] 175.3608
# RMSE of the knowledge-weighted estimate based on judges' final estimates:
(KWE2 =  knowledge_weighted_estimate(E_final, P, no_inf_check = TRUE))
#> [1] 865.0712
sqrt((KWE2 - theta)^2)
#> [1] 124.9288

This shows that the knowledge-weighted estimate can improve the average accuracy of an individual judge. Specifically, based on the judges’ initial and final estimates, the knowledge-weighted estimates are 814.6392 and 865.0712 calories, respectively. Given that the true calorie count is 990 calories, both aggregate estimates are too low but the knowledge-weighted estimate based on the judges’ final estimates is more accurate. In this example, it improves the individual judges’ RMSE from around 670 calories to around 125 calories.