2. Experiment Design:

“MacBehaviour” can implement an experiment in two types of designs.

1) multiple-trials-per-run design resembles typical psychological experiments, where a human participant encounters multiple trials in an experiment. Here, you present multiple experimental trials, one by one, to an LLM in a single conversation. Note that earlier input and output will serve as the context for a current trial.

2) one-trial-per-run design, you only present a single trial of prompt and stimulus to an LLM in a conversation, and you present another trial in a new conversation.

3. Demo Experiments:

To illustrate these designs and how to construct the experimental stimuli, we next use a demo experiment.

Cassidy et al. (1999) showed that speakers of English can infer the gender of novel personal names from phonology.

In particular, when asked to complete a sentence fragment:

After Corlak/Corla went to bed …

People tend to use a masculine pronoun for names ending in a closed syllable (e.g., Corlak) but a feminine pronoun for those ending in an open syllable (e.g., Corla).

In our demo, we ask an LLM to complete sentence fragments and observe how the model refers to the novel personal name (e.g., using masculine pronouns such as he/him/his or feminine ones such as she/her/hers).

3.1 multiple-trials-per-run design

Before using this package, users should prepare one Excel/CSV file/data frame containing the experimental stimuli and other information for experiment design (see Table 1).

Table 1. The data frame structure

Note. Each row stands for a unique stimulus in the data frame/sheet.

The Excel file/data frame should exhibit a structured format, defining columns for “Run”, “Item”, “Condition”, and “Prompt”, with each row standing for a unique stimulus (see Table 1 for a description of these terms and Table 2 for an example).

In the multiple-trials-per-run design, multiple trials (four trials in our demo) are presented in a single conversation (Run).

In each Run, the package will send the stimulus based on the index of row. Users can randomize item order within Runs in the function “experimentDesign” later. The LLM will use input (prompts and stimuli) and model output (responses) in earlier trials as its context (see Figure 1 for an example of conversation/Run).

Table 2. An exemplar stimulus file in a multiple-trials-per-run design

There are two roles in the context: “user” (for sending stimuli) and “assistant” (as a participant to provide responses). To achieve the above conversation, this package sends the stimuli in the following format for OpenAI GPT series/open-source models and Llama2:

​

# OpenAI/Open-source models

# For the first trial: 

list(role = "user", content = "Please repeat the fragment and complete it into a full sentence: Although Pelcra was sick … ") 

 

# For the second trial:

[list (role = "user", content = "Please repeat the fragment and complete it into a full sentence: Although Pelcra was sick … "),

list (role = "assistant", content = " Although Pelcra was sick, she remained determined to finish her project on time. "),

list (role = "user", content = " Please repeat the fragment and complete it into a full sentence: Because Steban was very careless …")]

​

The conversational context was provided as the beginning of the next trial’s prompt. In this example, the context included the first stimulus Please repeat the fragment and complete it into a full sentence: Although Pelcra was sick … and its response Although Pelcra was sick, she remained determined to finish her project on time. The prompt then presented the second stimulus Please repeat the fragment and complete it into a full sentence: Because Steban was very careless … after the conversational context. We implemented this function for Llama-2-chat-hf series in the same way (see here for details).

​

3.2 One-trial-per-run Design

In the one-trial-per-run design, an LLM will be presented only one trial of the experiment in a Run/conversation. In our demo experiment (see Table 3), for instance, each conversation with the LLM involves only one stimulus. In this design, each stimulus is given a unique Run number, indicating that each one is to be presented in a separate conversation with the LLM. This design eliminates the potential for previous context to influence the response of current stimulus, ensuring that each stimulus is evaluated independently.

​

Table 3. Stimuli for one-trial-per-run design

​

Load your stimuli from an Excel file.

df = read.xlsx("/path/to/excel/demo.xlsx")

The read.xlsx function from the openxlsx package reads the file, converting it into a data frame within R. To accurately use the stimuli within the R environment, the loadData function is utilized, which translates the structured data from an Excel file/data frame into an organized data frame within R:

ExperimentItem = loadData(runList=df$Run, itemList=df$Item, conditionList=df$Condition, promptList=df$Prompt)

Arguments: This function prepares the stimuli from your Excel data.

The “loadData” function maps vectors or data frame columns to specific keywords. These keywords are then recognized by subsequent functions in our framework. This mapping streamlines the automatic identification and processing of relevant data collection:

1. The “runList”, required, a numeric vector, matches the column for “Run” in the CSV file and denotes the conversation/run index. It is utilized in loops for interactions with LLMs. The vector’s name (e.g., df$Run) can be arbitrary; what’s important is the content specified by users for the runList. This applies to subsequent parameters in this function as well.

2. The “itemList”, required, a numeric vector, refers to the column for “Item”, indicating the item index of stimuli. This is for the researcher’s reference and does not interact with the model’s operation. It will be used in loops for interactions with LLMs.

3. The “conditionList”, required, a numeric/character vector, represents the column for “Condition”, which specifies the experimental condition associated with each stimulus. Similar to “itemList”, it is for the researcher’s reference and does not interact with the model’s operation.

4. The “promptList”, required, a character vector, maps to the column for “Prompt”, which contains the actual prompts that will be presented to the model during the experiment. Each element under this column is a unique prompt the language model will process and respond to.

The output of this function, “ExperimentItem”, is a data frame generated by “loadData”, which includes all the necessary details for each stimulus. The accuracy of “loadData” in mapping the CSV spreadsheet/data frame to the “ExperimentItem” is of pivotal importance, as it ensures that each stimulus is precisely presented according to the experimental design.

Next, the “experimentDesign” function allows users to define the structure and sequence of the experimental Runs (conversation):

Design = experimentDesign(ExperimentItem, session = 1, randomItem = F)

1. “ExperimentItem”, required, a data frame, is the output of function “loadData”, which is a structured data frame for storing stimuli and experimental information (e.g., item, condition, and prompt for each stimulus).
2. The “Session”, optional, an integer, specifies the number of iterations for all stimuli. The default value is 1. It adds a new column named “session” to your data frame, where each session includes all original stimuli. If the “Session” is set to 2, the package collects data for one session and then repeats all stimuli for a second session.
3. “randomItem”, optional, a logical vector, is available to randomize the order of item presentation within a run (conversation). It automatically remains “FALSE” for the one-trial-per-run design.

gptConfig = preCheck (data = Design, checkToken = F, systemPrompt = "You are a participant in a psychological experiment", max_tokens = 500, temperature = 0.7, n = 1)

[list(role = "system", content = " You are a participant in a psychological experiment, please follow the task instruction carefully."),
(role = "user", content = "Please repeat the fragment and complete it into a full sentence: Although Pelcra was sick …"),
…]

runExperiment (gptConfig, savePath = "./demo.xlsx")

1) "gptConfig" is the configuration list object containing all the details of the experiment setup, including the system prompt, chosen model, maximum tokens, temperature, the number of responses and other parameters. This object is crafted in the preceding steps "preCheck".
2) "savePath" is the file path where the experiment results will be saved. This should be an accessible directory on the user's machine with the appropriate write permissions. A file name in the path with either the ".xlsx" or ".csv" extension indicates that its contents are saved in "xlsx" or "csv" format, respectively. These formats are particularly friendly for users who may wish to perform additional data manipulation or visualization within spreadsheet software or import the data into statistical software packages for further analysis.