is attractive when working with private data or building workflows that should run on our own machine.
And those workflows no longer need to be limited to text.
In this post, we’ll build such a workflow with Gemma 4 and Ollama. We’ll place Gemma 4’s multimodal capability inside a larger process, where downstream steps can consume its structured output.
I recently took a trip to Finland, and I took quite some photos during my journey. In this post, I’ll show you how I use the workflow to analyze them, and then show how the same workflow can power a small application.
1. The Multimodal Workflow
There are two ideas associated with the term “multimodal workflow”.
First, multimodal. This means instead of working only with text, the LLM also receives inputs of other types, such as images. In this post, we consider the local LLM of Gemma 4 family from Google. This model family is capable of processing both image and text.
Then, workflow. This means instead of deciding autonomously what to do next as in an agentic loop, the LLM operates inside a pre-defined sequence, where each stage receives an input and produces an output for the next stage. LLM serves as a function that performs semantic transformations.
For our case study, we want to turn a folder of trip photos into organized memory records. To achieve that, we need to analyze each photo and translate its content into a consistent, structured record. Once those records are available, we can combine them to understand the entire collection.
This naturally gives us a three-stage workflow:
photos = prepare_photos("Finland_trip")
photo_memories = [
analyze_photo(
image=photo.image,
metadata=photo.metadata,
)
for photo in photos
]
trip_memory = synthesize_trip(photo_memories)First, Python prepares the individual images and extracts useful metadata such as their capture times and GPS coordinates.
Then, our local LLM Gemma 4 can analyze each photo and return a structured record of the visual content of the photo.
Finally, we pass the individual records again to Gemma 4 to produce a structured summary of the entire collection.
In what follows, we build the individual stages.
2. Building the Workflow with Gemma 4
2.1 Running Gemma 4 Locally
First things first, we need to make Gemma 4 available locally. For that, we use Ollama, which gives us a local runtime and interface for interacting with the model.
In this post, we use the E4B variant of Gemma, which is one of the edge-friendly models in the family.
On Windows and macOS, you can download and run the installer from the Ollama website. On Linux, you can install from the terminal:
"curl -fsSL https://ollama.com/install.sh | sh"Once Ollama is installed, we can pull the Gemma model:
ollama pull gemma4:e4bWe also need to install the Python packages needed for the workflow:
pip install ollama pillow pydanticHere, ollama is needed for connecting our Python code to the local LLM, pillow handls the image processing, and pydantic is for structured output.
2.2 From Photo to Structured Record
For each photo, we want to produce a structured record describing what the model sees.
Before the images reach Gemma 4, we need to preprocess them. In our workflow, we implemented deterministic logic to resize the image and extract available EXIF metadata, and we wrap them inside prepare_photo().
From the workflow’s perspective, we only need its outputs:
from pathlib import Path
image, metadata = prepare_photo(
Path("Finland_trip/photo.jpg")
)Here, image contains the prepared image bytes, while metadata is a dictionary containing the information extracted from the file.
You can find the full implementation details in the repo attached in the end of the post.
Next, we define the structured record we want Gemma 4 to return:
from pydantic import BaseModel, Field
class PhotoMemoryAnalysis(BaseModel):
scene_summary: str
memory_caption: str
place_type: str
visible_activities: list[str]
visible_objects: list[str]
inferred_interest_signals: list[str]
mood: str
uncertainty_notes: list[str]
confidence: float = Field(ge=0, le=1)This is what we call structured output. Effectively, we pre-define this schema and ask the LLM to output according to this shape. This way, downstream code can access the result through typed attributes or convert it into a dictionary with model_dump().
Then, we build the instruction and prompt:
import json
PHOTO_ANALYSIS_INSTRUCTION = """
Analyze the supplied travel photo and its metadata.
Return a structured record grounded in the provided inputs.
"""
def build_photo_prompt(metadata: dict) -> str:
return f"""Photo metadata:
{json.dumps(metadata, indent=2)}
"""Now we can put everything together:
import ollama
MODEL = "gemma4:e4b"
def analyze_photo(
image: bytes,
metadata: dict,
) -> PhotoMemoryAnalysis:
response = ollama.chat(
model=MODEL,
messages=[
{
"role": "system",
"content": PHOTO_ANALYSIS_INSTRUCTION,
},
{
"role": "user",
"content": build_photo_prompt(metadata),
"images": [image],
},
],
format=PhotoMemoryAnalysis.model_json_schema(),
options={"temperature": 0},
)
return PhotoMemoryAnalysis.model_validate_json(
response.message.content
)Note that we used images field to supply the visual input, and used format to ask Ollama to follow the defined schema. We can then parse the response into a regular Python object:
photo_analysis = analyze_photo(
image=image,
metadata=metadata,
)This way, we have the routine to transform a photo into one structured memory record.
2.3 A Compatibility Issue with Image Input
A note worth mentioning: with my current setup for Ollama (0.32.5) on my Windows machine with gemma4:e4b, it seems that Ollama accepted the multimodal request, but the model didn’t manage to use its visual content.
A simple workaround I adopted is to load Gemma 4 using two separate files from Unsloth’s Gemma 4 E4B GGUF repository:
mmproj-BF16.gguf, which contains the multimodal projector.
gemma-4-E4B-it-UD-Q4_K_XL.gguf, which contains the quantized model.
The projector is the component that allows the model to consume the visual information.
After placing both files in the same folder, I created the following Modelfile:
FROM ./gemma-4-E4B-it-UD-Q4_K_XL.gguf
FROM ./mmproj-BF16.ggufI then imported the model into Ollama:
ollama create gemma4-e4b-split-test -f ModelfileFinally, I updated the model name in Python:
MODEL = "gemma4-e4b-split-test"With this setup, the Gemma model can correctly consume the image.
Of course, If gemma4:e4b already responds correctly to image inputs on your machine, this workaround is not needed.
2.4 From Photo Records to Trip Memory
At this point, we can understand each photo independently.
For the final stage, we want to understand the trip as a whole. Towards that end, we can pass those records to Gemma 4 once more and ask it to connect the individual moments into a trip memory.
First, we apply analyze_photo() to the entire folder:
from pathlib import Path
photo_memories = []
for image_path in Path("Finland_trip").glob("*.jpg"):
image, metadata = prepare_photo(image_path)
analysis = analyze_photo(
image=image,
metadata=metadata,
)
photo_memories.append({
"photo_id": image_path.name,
"analysis": analysis.model_dump(),
"metadata": metadata,
})Note that each photo memory now combines two sources of information: the metadata extracted from the file, and the semantic interpretation of the image provided by Gemma 4.
We define another schema for the final output:
class MemorableMoment(BaseModel):
title: str
description: str
evidence_photo_ids: list[str]
class TripMemorySynthesis(BaseModel):
narrative_summary: str
inferred_interests: list[str]
recurring_themes: list[str]
memorable_moments: list[MemorableMoment]
uncertainty_notes: list[str]We then define the instruction and prompt:
TRIP_SYNTHESIS_INSTRUCTION = """
Synthesize the supplied photo records into a structured trip memory.
Use only the information contained in those records.
"""
def build_trip_prompt(
photo_memories: list[dict],
) -> str:
return f"""Photo memory records:
{json.dumps(photo_memories, indent=2)}
"""The final model call follows the same pattern as before. Note that this call is text-only:
def synthesize_trip(
photo_memories: list[dict],
) -> TripMemorySynthesis:
response = ollama.chat(
model=MODEL,
messages=[
{
"role": "system",
"content": TRIP_SYNTHESIS_INSTRUCTION,
},
{
"role": "user",
"content": build_trip_prompt(
photo_memories
),
},
],
format=TripMemorySynthesis.model_json_schema(),
options={"temperature": 0},
)
return TripMemorySynthesis.model_validate_json(
response.message.content
)We can now complete the workflow:
trip_memory = synthesize_trip(photo_memories)2.5 Results
For this case study, I picked 7 photos from my Finland trip and ran the workflow over them.
Let’s first look at two of the photo records.
This is what Gemma 4 returned for the first image:
{
"scene_summary": (
"A large Ferris wheel, prominently displaying the "
"name 'Helsingfors,' dominates the frame against "
"a clear twilight sky."
),
"memory_caption": (
"Evening views from the Helsingfors Ferris Wheel ride."
),
"visible_objects": [
"Ferris wheel",
"Signage (Helsingfors)",
"Sky",
"Foreground railing/platform",
],
"mood": "Festive",
"uncertainty_notes": [
"The specific location within Helsingfors is not "
"provided, only the name on the attraction."
],
}The model recognized the main attraction. It also read the visible text and incorporated the evening setting into the record.
Here is another photo and model output:
{
"scene_summary": (
"A large owl or raptor is standing near a wooden "
"enclosure and wire fence in a lush, green outdoor "
"setting."
),
"memory_caption": (
"An encounter with wildlife: A majestic bird "
"observing its surroundings amidst the dense greenery."
),
"visible_objects": [
"Owl/Raptor",
"Wooden structure",
"Wire mesh fence",
"Dense foliage",
],
"mood": "Awe-inspiring, Peaceful",
"uncertainty_notes": [
"The precise species of the bird cannot be "
"definitively identified from the image alone.",
"The specific function of the wooden structure "
"is ambiguous.",
],
}After analyzing all seven photos, the workflow passed the structured records to Gemma 4 for the final trip synthesis. Here’s what I got:
This summer trip across Finland was a diverse journey blending
cultural immersion with natural exploration. The traveler
experienced grand architectural settings, from modern concert
halls to majestic neoclassical landmarks, and enjoyed urban life
through city views and dining experiences. Leisure time included
visiting amusement parks and observing local history. A significant
portion of the trip focused on nature, featuring wildlife encounters
and dramatic riverfront sunsets. Travel logistics were also a part
of the experience, marked by transit stops between destinations.3. From Workflow to Application
To make it more interesting, I built a small travel-memory application powered by the workflow we just walked through.
This app allows users to upload their photos and then start the analysis workflow from the interface. Once the analysis is done, it arranges the photos along a timeline and places them on a map using their GPS coordinates.
For each photo, the user can see the caption, scene tags, and other details generated by Gemma 4. At the trip level, the user can also see the summary, model-inferred interests, and memorable moments.
In the app, the user can also search the photo memories and ask questions about the trip. All is done with a local LLM.
You can find the code here: https://github.com/ShuaiGuo16/gemma4-multimodal-workflow