Restaurant menus are messy data.
They come as photos, screenshots, PDFs, scans, different languages, different layouts, and sometimes with several prices for the same item.
For Pinjoy, we wanted to turn that unstructured input into structured restaurant data without forcing restaurant owners to manually enter their entire menu.
So we built an AI-powered menu ingestion pipeline that takes menu images and turns them into normalized JSON that can be stored and consumed by the rest of the platform.
The interesting part wasn't simply calling a vision model.
The challenge was designing the system around the model so that the output was predictable, observable, cost-efficient, and testable.
The Pipeline
At a high level, the pipeline looks like this:
Menu Images
│
▼
Client-side Image Compression
│
▼
GPT-4.1 Vision
│
▼
Compact Intermediate Representation
│
▼
Deterministic Parser / Transformer
│
▼
Normalized JSON
│
▼
Validation
│
▼
Database
There are two additional layers around the pipeline:
┌─────────────────────┐
│ Observability │
│ tokens / cost / │
│ latency / status │
└──────────┬──────────┘
│
Menu → AI → Parser → JSON → Database
┌─────────────────────┐
│ Evaluation │
│ extraction accuracy │
│ regression tests │
└─────────────────────┘
1. Start With Image Preprocessing
Restaurant owners don't necessarily upload optimized images.
A phone can produce a large image that contains far more pixels than the model needs to understand a menu.
We therefore compress the images on the client before sending them to the AI pipeline.
This reduces the amount of data being transferred and helps control the cost and latency of processing large menus.
The important architectural decision here is that preprocessing happens before the AI call rather than making the model pipeline responsible for unnecessarily large inputs.
2. Don't Ask the Model for the Final JSON
This was one of the more important decisions in the implementation.
A natural approach would be:
"Here is a menu image. Return this large JSON schema."
But that creates a lot of unnecessary output.
Instead, we defined a compact intermediate representation.
The model was instructed to produce something conceptually like:
#Burgers
Classic Burger||Beef patty, cheese||75
Chicken Burger||Chicken, cheese||70
#Pizza
Margherita||Tomato, mozzarella, basil||60
For items with multiple variants or prices, the compact format can represent those values without repeating verbose JSON property names.
The prompt explicitly defines the format:
#Section
Name||Desc||Title1:Value1;Title2:Value2
Name||Desc||Value1;Value2
Name||||SinglePrice
The model therefore performs the difficult part:
understanding the image and extracting the information.
It doesn't need to spend tokens repeatedly generating:
{
"category": "...",
"name": "...",
"description": "...",
"prices": [...]
}
for every item.
3. Deterministic Transformation
The compact representation is not stored directly.
We pass it through our own deterministic transformation layer.
AI output
↓
Parser
↓
Normalized object
↓
Validation
↓
Database
This is an important separation.
The LLM is responsible for probabilistic interpretation.
Our application code is responsible for deterministic transformation.
For example:
#Burgers
Classic Burger||Beef patty, cheese||75
becomes something like:
{
"name": "Burgers",
"items": [
{
"name": "Classic Burger",
"description": "Beef patty, cheese",
"prices": [
{
"value": 75
}
]
}
]
}
This gives us control over the final schema instead of trusting the model to consistently produce our entire database representation.
4. Observability From the Beginning
Once an LLM becomes part of a production pipeline, the API response alone isn't enough.
We need to know what happened during every execution.
So I added AI execution logging to the database.
For each execution, we track information such as:
Model
Tool / operation
Status
Latency
Input tokens
Output tokens
Total tokens
Estimated cost
Execution metadata
This gives us visibility into questions such as:
How long does menu extraction take?
How many tokens are we consuming?
What does a menu extraction cost?
Are failures increasing?
Did a prompt change increase token usage?
Did changing the model affect latency?
Which stage of the pipeline is failing?
This also makes optimization measurable.
Instead of saying:
"The compact format seems cheaper."
We can actually compare token usage and cost across executions.
5. Evaluation Instead of Manual Testing
One of the biggest challenges with AI systems is that traditional unit tests aren't enough.
A parser can be tested deterministically.
An LLM cannot.
So we created an evaluation approach around a fixed set of representative restaurant menus.
The dataset contains menus covering different layouts and extraction challenges.
For each menu, we define expected results and evaluate the pipeline against them.
For example:
Expected items: 52
Extracted items: 50
Price accuracy: 98%
Category accuracy: 96%
Item recall: 96%
Schema validity: 100%
We can then run the same evaluation whenever we change:
the prompt
the model
the extraction logic
the intermediate format
This turns AI changes into something measurable rather than relying entirely on manual inspection.
6. Deterministic Tests + AI Evaluations
An important distinction is that not everything needs an LLM judge.
The deterministic transformation layer can have traditional software tests:
compact input
↓
parser
↓
expected JSON
Those tests are fast and deterministic.
The AI extraction layer is evaluated against real menu images.
So the system effectively has two testing layers:
Traditional tests
│
▼
Parser / transformation correctness
AI evaluations
│
▼
Model extraction behavior
This separation makes debugging much easier.
If an evaluation fails, we can determine whether the problem came from the model's extraction or from our own transformation code.
7. Production Failures Become New Evaluation Cases
The evaluation dataset shouldn't be static.
If a restaurant menu produces an interesting failure in production, that case can become a regression test.
The process becomes:
Production failure
↓
Reproduce / anonymize
↓
Add to evaluation dataset
↓
Fix prompt / pipeline
↓
Run complete evaluation
↓
Prevent the same regression
Over time, the evaluation dataset becomes a collection of the actual edge cases encountered by the product.
That is much more valuable than testing only ideal menus.
8. Why We Didn't Just Generate JSON
The architecture ultimately comes down to a simple principle:
Use the LLM where probabilistic reasoning is useful, and use deterministic software everywhere else.
The model is very good at looking at a menu image and understanding:
"This is a category, these are menu items, this is a description, and these are prices."
But once we have extracted that information, there is little reason to ask the model to repeatedly generate our application's verbose data structure.
The deterministic layer gives us:
predictable output
schema control
easier validation
easier testing
lower model output overhead
And the observability and evaluation layers give us:
measurable cost
measurable latency
extraction quality metrics
regression detection
production feedback loops
The Result
The final architecture isn't simply:
Image → GPT → Database
It's:
Image
↓
Preprocessing
↓
Multimodal AI extraction
↓
Compact representation
↓
Deterministic transformation
↓
Validation
↓
Database
+ Observability
+ Evaluation
+ Regression testing
That distinction matters when building AI features for production.
The model is only one component.
The real engineering challenge is building the system around it so that you can measure it, control it, debug it, optimize it, and safely evolve it.
That's the approach we took with Pinjoy's menu ingestion pipeline.