The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

image-annotation

Maximilian Weber

Ollama also supports multimodal models, which can interact with (but not create) images.

We start by loading the package:

library(rollama)

After loading the package, we need to pull a model that can handle images. For example, the llava model. Using pull_model("llava") will download the model, or just load it if it has already been downloaded before.

pull_model("llava")
#> ✔ model llava pulled successfully!

We can use textual and visual input together. For instance, we can ask a question and provide a link to a picture or a local file path, such as images = "/home/user/Pictures/IMG_4561.jpg".

In the first example, we ask the model to describe the logo of this package:

query(
  "Excitedly desscribe this logo",
  model = "llava",
  images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png"
)
#> 
#> ── Answer from llava ─────────────────────────────────────────────────
#> This is an image of a logo for "Rollama." The logo features a playful
#> and creative design, with a cartoon-style character resting on a bed
#> of green grass. The character is anthropomorphic, having arms and
#> legs like a human, but it has animal-like ears and is wearing a blue
#> helmet. The helmet seems to have a visor, and there's a badge
#> attached to it that reads "Rollama."
#> 
#> The background of the logo is light blue with a faint cloud pattern,
#> which adds to the whimsical feel of the design. The use of bold
#> colors and simple shapes gives the logo a friendly and approachable
#> vibe, suggesting that whatever "Rollama" represents could be fun and
#> enjoyable.
#> 
#> Without additional context, it's not possible to determine the exact
#> nature or purpose of Rollama from this image alone. However, the
#> playful design and the badge suggest that it might be related to a
#> game, an application, or possibly a branding for something
#> entertaining and engaging.

The second example asks a classification question:

query(
  "Which animal is in this image: a llama, dog, or walrus?",
  model = "llava",
  images = "https://raw.githubusercontent.com/JBGruber/rollama/master/man/figures/logo.png"
)
#> 
#> ── Answer from llava ─────────────────────────────────────────────────
#> The image shows an animated character that resembles a llama. It has
#> distinctive features of a llama, such as the large head with two
#> forward-facing horns and a long, curved neck with short, rounded ears
#> at the top.

Annotating Several Images with Structured Output

You can combine this setup with the structured output workflow to get a systematic classification of images. First, define the schema for the fields you want back for each image:

animal_schema <- create_schema(
  dog = type_boolean(),
  cat = type_boolean(),
  human = type_boolean(),
  bird = type_boolean()
)
animal_schema
#> <rollama structured output schema>
#> ├─object: <NULL> (required)
#> └─properties
#>   ├─boolean: <dog>  (required)
#>   ├─boolean: <cat>  (required)
#>   ├─boolean: <human>  (required)
#>   └─boolean: <bird>  (required)

I put some CC images from Wikipedia in a vector:

examples <- c(
  "https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Huskiesatrest.jpg/1920px-Huskiesatrest.jpg",
  "https://upload.wikimedia.org/wikipedia/commons/thumb/6/68/Orange_tabby_cat_sitting_on_fallen_leaves-Hisashi-01A.jpg/960px-Orange_tabby_cat_sitting_on_fallen_leaves-Hisashi-01A.jpg",
  "https://upload.wikimedia.org/wikipedia/commons/c/c4/Puffin_(Fratercula_arctica).jpg"
)

To annotate each image on its own, rather than sending all of them into a single query, pass images as a list: each element is then paired up with q (recycling the prompt for every image), producing one separate query - and one separate result - per image.

results <- query(
  q = "What can you see in this image?",
  model = "qwen3-vl",
  format = animal_schema,
  images = as.list(examples),
  output = "text",
  stream = FALSE,
  think = FALSE
)

Because the output is guaranteed to be valid JSON, you can parse all results at once1:

purrr::map(results, jsonlite::fromJSON) |>
  dplyr::bind_rows()
#> # A tibble: 3 × 4
#>   dog   cat   human bird 
#>   <lgl> <lgl> <lgl> <lgl>
#> 1 TRUE  FALSE FALSE FALSE
#> 2 FALSE TRUE  FALSE FALSE
#> 3 FALSE FALSE FALSE TRUE

  1. Note that not all models have the ability to use structured outputs and some will ignore the schema.↩︎

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.