The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Getting Started with AutoGenAI

AutoGenAI treats a generative AI workflow as a configuration containing a prompt, a provider, and a generation strategy. The package can benchmark these configurations using a task-specific scorer and then select configurations under quality, cost, and latency objectives.

Offline example

library(AutoGenAI)
ex <- autogenai_example()
fit <- optimize_ai(
  ex$task,
  ex$data,
  ex$providers,
  ex$prompts,
  temperatures = 0,
  strategies = "single"
)
fit
#> AutoGenAI optimization
#> Task: sentiment 
#> Configurations: 12 
#> 
#> Best configuration
#>  Provider: mock-balanced 
#>  Prompt ID: 3 
#>  Strategy: single 
#>  Temperature: 0 
#>  Quality: 1.0000 
#>  Cost: 0.000045 
#>  Latency: 0.0000 sec
#>  Utility: 0.9138

Pareto-efficient choices

pareto_ai(fit)
#>                         config_id prompt_id
#> 1  3::mock-balanced::0::single::1         3
#> 5  1::mock-balanced::0::single::1         1
#> 9     2::mock-cheap::0::single::1         2
#> 12    1::mock-cheap::0::single::1         1
#>                                                                                                                        prompt
#> 1  Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label. Check the answer carefully before responding.
#> 5                                                Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#> 9                  Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label. Return only the final answer.
#> 12                                               Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#>         provider temperature strategy samples   quality structured_score
#> 1  mock-balanced           0   single       1 1.0000000               NA
#> 5  mock-balanced           0   single       1 1.0000000               NA
#> 9     mock-cheap           0   single       1 0.6666667               NA
#> 12    mock-cheap           0   single       1 0.6666667               NA
#>    success_rate     cost      latency n   utility robustness
#> 1             1 4.50e-05 0.0000000000 6 0.9137905         NA
#> 5             1 3.35e-05 0.0001666667 6 0.8146759         NA
#> 9             1 8.20e-06 0.0000000000 6 0.3091236         NA
#> 12            1 6.70e-06 0.0001666667 6 0.1875000         NA

Robustness

st <- stress_test(
  ex$task,
  ex$data,
  ex$providers[[1]],
  ex$prompts[[1]]
)
st
#>   perturbation quality
#> 1     original       1
#> 2   whitespace       1
#> 3         case       1
#> 4         typo       1
#> 5   distractor       1
robustness_score(st)
#> [1] 1

Real providers

A real provider is any R function accepting prompt, input, and params and returning one text value. This deliberately keeps model-specific credentials and network behavior outside the package core.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.