The Gemini / Vertex Manifesto
Gemini as the model brain and Vertex AI as the industrial control plane: one ecosystem for multimodal intelligence, agents, grounding, governance and production deployment.
VisionGoogle AIEnterprise
IDEO-Lab 2026 Guide Gemini / Vertex AI Enterprise AI control plane
Gemini / Vertex The Enterprise Multimodal AI Stack
A premium IDEO-Lab guide dedicated to Google's Gemini and Vertex AI ecosystem: Gemini models, AI Studio, Gemini API, Vertex AI, agents, grounding, RAG, long context, multimodal documents, function calling, built-in tools, context caching, Model Garden, evaluation, security, FinOps and production governance.
Gemini
Vertex AI
Grounded
Governed
Vertex AI
Grounded
Governed
01
Models + API
02
Vertex control
03
Agents + Tools
04
RAG + Evals
Guide angle: Gemini is treated as the intelligence layer and Vertex AI as the enterprise control plane. The goal is not a model catalog; the goal is a production architecture for multimodal AI, source-grounded answers, tool-using agents, governed data paths, release gates and sustainable unit economics.
Platform Map: Gemini API, AI Studio and Vertex AI
Understand the three main surfaces: AI Studio for exploration, Gemini API for application developers, and Vertex AI for governed enterprise deployment.
APIStudioVertex
Gemini Model Family 2026
A dense map of current Gemini model families: Pro for deep work, Flash for speed and multimodal interaction, Flash-Lite for low-cost scale, plus embeddings and specialized models.
ProFlashLite
Gemini 3.1 Pro Preview
The high-capability Gemini route for complex reasoning, software engineering behavior, grounded analysis and precise multi-step agentic workflows.
Deep workAgentsCode
Gemini 3 Flash Preview
The high-speed Gemini route for interactive multimodal understanding, agentic applications, developer workflows and responsive production assistants.
FastMultimodalAgentic
Gemini 3.1 Flash-Lite Preview
The economical Gemini route for high-frequency extraction, classification, routing, lightweight agentic tasks and low-latency production flows.
Low costHigh volumeFast
Long Context and Repository-Scale Reasoning
How to use Gemini long context responsibly for large documents, repositories, logs, transcripts, research packs and enterprise knowledge bundles.
2M tokensDocsCodebases
Native Multimodal Gemini
Gemini can understand mixed inputs across text, image, document, audio and video workflows, enabling rich enterprise analysis beyond text-only chat.
ImagesVideoAudio
Document Understanding and PDF Workflows
Use Gemini for native PDF understanding, table extraction, contracts, technical manuals, invoices, diagrams, forms and RAG-ready document analysis.
PDFTablesForms
Grounding with Google Search
Connect Gemini to real-time web content through Google Search grounding for fresher answers, research workflows and cited source-backed outputs.
Fresh webCitationsResearch
Vertex AI Search Grounding and Enterprise RAG
Ground Gemini with enterprise documents and websites using Vertex AI Search, RAG Engine, metadata filters and permission-aware knowledge systems.
RAGSearchSources
File Search and Multimodal Embeddings
Use Gemini file search and multimodal embeddings to retrieve across text, image, video, audio and PDF content in a unified knowledge layer.
FilesEmbeddingsRAG
Function Calling and Tool Contracts
Make Gemini call controlled functions and APIs with strict schemas, validation, approval gates and audit logs instead of free-form action guesses.
ToolsSchemasActions
Built-in Tools: Search, URL Context, Maps, File Search, Code Execution
Use Gemini built-in tools for grounded web retrieval, URL analysis, file search, Google Maps workflows and code execution where supported.
Built-inCodeURL
Live API and Real-Time Voice / Video
Design low-latency Gemini applications for streaming audio, video, voice agents, live assistants, coaching and multimodal interaction.
RealtimeVoiceVideo
Structured Outputs and JSON Schemas
Constrain Gemini outputs into JSON, schemas and deterministic contracts for APIs, extractors, workflows, migration diagnostics and agents.
JSONSchemaContracts
Context Caching and Cost Engineering
Use implicit and explicit context caching on Vertex AI to reduce repeated-prefix cost and latency for large prompts, repositories, videos and document packs.
CacheLatencyFinOps
Gemini Embeddings and Semantic Search
Build semantic and multimodal search layers using Gemini embeddings, hybrid retrieval, metadata filters and evaluation-driven ranking.
VectorsHybridRetrieval
Vertex AI Agents, Agent Engine and Agent Builder
Use Vertex AI agent infrastructure to build, deploy, connect and evaluate governed agents with tools, state, orchestration and enterprise controls.
AgentsADKRuntime
Deep Research and Research Agents
Design source-grounded Gemini research workflows for technical watch, competitive analysis, documentation synthesis and long-form reports.
ResearchSourcesReports
Vertex AI Model Garden and Open Models
Discover, test, tune, deploy and govern Google, partner and open models through Model Garden and managed model infrastructure.
Model GardenGemmaMaaS
OpenAI-Compatible Integration and Migration
Design adapters that let existing OpenAI-style applications call Gemini or Vertex AI while preserving governance, schemas and model routing.
SDKMigrationCompatibility
Gemini for Software Engineering
Use Gemini for repository understanding, code review, patch planning, unit tests, migration analysis, documentation and developer assistants.
CodeReposTests
Batch, Webhooks and Asynchronous AI Workflows
Use batch processing and event-driven completion patterns for large-scale extraction, summarization, embeddings, document ingestion and offline AI jobs.
BatchAsyncWebhooks
Security, IAM, Data Residency and Governance
Use Google Cloud controls around projects, IAM, service accounts, organization policy, logging, regions, retention and enterprise data boundaries.
IAMPolicyAudit
Gen AI Evaluation and Release Gates
Evaluate Gemini models, prompts and agents with explainable metrics, task-specific datasets, regression tests and production release gates.
EvalsQualityRegression
Quotas, Rate Limits and Provisioned Throughput
Plan capacity for Gemini applications using rate limits, quotas, Provisioned Throughput, fallbacks and workload shaping.
QuotasPTScale
Token Economics and FinOps
Control Gemini and Vertex AI economics with model routing, token budgets, caching, batch strategy, media limits and feature-level profitability.
BudgetTokensMargin
Observability, Logs and AI Incident Response
Make Gemini applications observable with request logs, model IDs, tool traces, retrieval evidence, error taxonomy and incident playbooks.
LogsTracingIncidents
Prompt Design, System Instructions and Prompt Ops
Treat prompts as software artifacts with system instructions, constraints, examples, schemas, tests, versions and rollout strategy.
PromptsVersionsOps
Vertex AI Media Models: Image, Video and Creative Workflows
Place Gemini alongside Vertex AI media generation and understanding capabilities for product content, visuals, video summaries and multimodal applications.
ImageVideoCreative
Vertex AI as an Enterprise AI Control Plane
Use Vertex AI as the control plane for AI projects: datasets, endpoints, models, evaluation, monitoring, deployment, governance and cost management.
MLOpsControlCloud
Reference Enterprise Architecture
A full-stack blueprint for a Gemini / Vertex AI enterprise application with UI, API, retrieval, tools, evals, security, logging and FinOps.
BlueprintLayersControls
IDEO-Lab Gemini / Vertex Playbook
A concrete deployment path for IDEO-Lab: Django integration, guide factory, MigrateSafe, SRDF, glossary search, OCR ingestion and AI productization.
DjangoMigrateSafeSRDF
Production Checklist
The final GO/NO GO checklist for launching Gemini / Vertex AI applications with controlled data, models, tools, evals, monitoring and cost limits.
ChecklistGO/NO GORunbook
Future of Gemini / Vertex AI
Where the ecosystem points next: stronger multimodal reasoning, deeper agents, unified search, live interfaces, enterprise governance and AI-native software factories.
FutureAgentsMultimodal
Grounding vs RAG vs Long Context
A decision framework for choosing Google Search grounding, Vertex AI Search RAG, file search, context caching or direct long-context prompts.
ChoiceEvidenceDesign
Data Pipeline for Gemini Applications
How to build ingestion pipelines for Gemini applications: file intake, parsing, metadata, embeddings, storage, retention and re-indexing.
IngestNormalizeIndex
Human-in-the-Loop and Approval Design
Design review gates for Gemini-generated code, data changes, support replies, financial outputs, legal summaries and production actions.
ReviewApprovalSafety
Product UX for Limits, Errors and Degraded Mode
Design user-facing behavior when Gemini is slow, rate-limited, uncertain, unavailable or too expensive for the requested task.
UXFallbackTrust
Procurement, Terms and Vendor Governance
Track Google AI and Google Cloud terms, data processing, billing ownership, support routes, model lifecycle and enterprise procurement constraints.
TermsLegalVendor
Official references used for this guide
Google AI Gemini API docsGemini API modelsGemini 3 Developer GuideVertex AI Generative AIVertex AI release notesFunction callingGemini toolsStructured outputsDocument understandingGrounding with Google SearchVertex AI Search groundingContext caching on Vertex AIVertex AI Model GardenGen AI evaluation
References
Official Google references
Reference map
This guide is based on official Google AI and Google Cloud documentation. Always verify model IDs, pricing, supported regions, model lifecycle, preview status, quota and data governance before production deployment.
Google AI Gemini API docsGemini API modelsGemini 3 Developer GuideVertex AI Generative AIVertex AI release notesFunction callingGemini toolsStructured outputsDocument understandingGrounding with Google SearchVertex AI Search groundingContext caching on Vertex AIVertex AI Model GardenGen AI evaluation
Production rule: pin exact model IDs, prompt versions, schema versions and source links in your internal registry because model names, endpoints and capabilities evolve.
Chapter 01
The Gemini / Vertex Manifesto
The Gemini / Vertex Manifesto
Gemini is the model family: multimodal reasoning, long-context understanding, code, tool use, document processing, structured output and real-time interaction. Vertex AI is the enterprise platform layer that turns those capabilities into governed production systems.
The strategic difference is not only model quality. It is the integration of Google AI Studio, Gemini API, Vertex AI, Cloud IAM, Model Garden, Agent Engine, Vertex AI Search, context caching, evaluation services, monitoring patterns and enterprise controls.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates The Gemini / Vertex Manifesto into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Model layer | Gemini Pro, Flash, Flash-Lite, embeddings, media-capable models and specialized task models. |
| Developer layer | Gemini API, Google Gen AI SDK, AI Studio, prompts, schemas, tools and code execution. |
| Enterprise layer | Vertex AI, IAM, projects, regions, quotas, governance, Model Garden, Agent Engine and evaluation. |
| Knowledge layer | Grounding with Google Search, Vertex AI Search, File Search, RAG Engine and enterprise data stores. |
| Operations layer | Rate limits, Provisioned Throughput, context caching, logging, cost controls, evals and release gates. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Start with the exact user workflow: research, coding, document extraction, support, DevOps, analytics, agent automation or multimodal search.
- Choose Gemini API for fast developer experiments; choose Vertex AI when production controls, projects, IAM, regionality, governance and enterprise procurement matter.
- Pin exact model IDs and build a model registry because preview endpoints, aliases, retirement dates and capabilities can change.
- Add structured outputs, tool allowlists, test sets, logging and fallback behavior before opening the workflow to real users.
- Treat every agent as a production integration: model plus tools plus permissions plus audit plus rollback.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A powerful Gemini answer can still be wrong when grounding, source selection or tool arguments are weak.
- Large context can hide poor retrieval discipline; long input is not the same as correct evidence.
- Preview models and preview endpoints require lifecycle monitoring before enterprise adoption.
- Tool execution and code execution must be isolated, scoped and audited.
- Cloud AI systems need IAM, region, retention, logging and billing policy from day one.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Django guide factory | Use Gemini to transform large technical material into dense IDEO-Lab modals and structured documentation. |
| MigrateSafe | Use long context, code reasoning and structured output to inspect migrations, diffs, DDL plans and failure catalogs. |
| SRDF orchestrator | Use Vertex AI for governed summaries of capture, batch, send and apply phases with auditable logs. |
| Knowledge base | Use Vertex AI Search and embeddings for source-grounded answers over guides, PDFs, code and runbooks. |
| Productization | Build a Gemini-backed toolbox with cost budgets, evals, user roles and release gates. |
The Gemini / Vertex Manifesto principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 02
Platform Map: Gemini API, AI Studio and Vertex AI
Platform Map: Gemini API, AI Studio and Vertex AI
The ecosystem has several entry points. AI Studio is the fast experimentation surface. Gemini API is the developer API surface. Vertex AI is the Google Cloud production and governance surface.
A mature architecture often starts in AI Studio, moves to the Gemini API for prototypes, then graduates to Vertex AI when IAM, quotas, evaluation, data governance, partner models, Model Garden and enterprise controls become mandatory.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Platform Map: Gemini API, AI Studio and Vertex AI into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Google AI Studio | Prompt experiments, API key management, model trials, rate limit visibility and quick demos. |
| Gemini API | Developer-friendly API for text, images, audio, video, documents, tools, structured outputs and interactions. |
| Vertex AI | Google Cloud integration for enterprise projects, IAM, regions, evaluations, Model Garden, grounding and operations. |
| SDK layer | Google Gen AI SDK plus Vertex AI SDK patterns for Python, Node, Go and Java. |
| Production boundary | Use Vertex AI when workloads need policy, billing, private network, observability and enterprise accountability. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use AI Studio to compare prompts and model behavior quickly.
- Move promising workflows into a small API prototype with typed inputs, outputs and error handling.
- Switch to Vertex AI when the application requires IAM, project isolation, organization policy, evaluation, regional constraints or enterprise support.
- Maintain one model-routing table so developers do not hard-code preview IDs everywhere.
- Document which surface is allowed for which data class.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- API keys from experiments can leak into production code if project separation is weak.
- A proof of concept built in AI Studio may not include IAM, logging, caching, failover or billing controls.
- Model IDs and aliases can shift over time; unmanaged references create hidden technical debt.
- Different environments can produce different latency, quota and pricing behavior.
- Governance must cover both Google AI developer accounts and Google Cloud projects.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Prototype lane | AI Studio for fast guide experiments and prompt tuning. |
| Django lane | Gemini API for product features inside IDEO-Lab. |
| Enterprise lane | Vertex AI for protected workloads with logs, IAM, budgets and model registry. |
| Runbook lane | A single internal page mapping which Google AI surface is approved for each use case. |
| FinOps lane | Separate dev, staging and production budgets. |
Platform Map: Gemini API, AI Studio and Vertex AI principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 03
Gemini Model Family 2026
Gemini Model Family 2026
The Gemini family should be treated as a routing portfolio. Not every workflow deserves the most capable model. Some workloads need deep reasoning, some need low latency, some need long-context document understanding, and some need high-volume cheap extraction.
A production Gemini architecture should keep model selection outside business logic, document endpoint lifecycle, and evaluate each model against real tasks before upgrades.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini Model Family 2026 into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Pro class | Highest reasoning and reliability targets for complex analysis, software engineering and multi-step agents. |
| Flash class | Balanced speed, multimodal ability and tool use for interactive applications and agents. |
| Flash-Lite class | Low-latency and high-volume tasks where cost and throughput dominate. |
| Embedding models | Semantic search, RAG, multimodal retrieval and knowledge indexing. |
| Specialized task models | Image, video, live interaction, research and other task-specific surfaces depending on availability. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create a model matrix with task, input type, latency target, cost target, quality target and fallback model.
- Use small models for routing, classification and extraction; use stronger models for final reasoning and high-risk decisions.
- Run evals before model upgrades because a newer model can be better globally but worse on a specific workflow.
- Keep aliases out of mission-critical code unless they are intentionally controlled by a registry.
- Monitor deprecations and retirement dates through official release notes.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A preview model may be attractive but not stable enough for regulated production.
- Using Pro everywhere can destroy unit economics.
- Using Lite everywhere can increase hidden review cost if error rates rise.
- Context window, media support and tool support differ by model and version.
- Fallback behavior must be tested, not improvised after an outage.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide generation | Use stronger models for dense architecture and critical technical writing. |
| Keyword extraction | Use faster models for bulk classification and tagging. |
| MigrateSafe analysis | Route complex migration failures to Pro-class models. |
| SRDF summaries | Use Flash-class models for repeated operational summaries. |
| Economics | Store model cost per feature and surface margin risks early. |
Gemini Model Family 2026 principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 04
Gemini 3.1 Pro Preview
Gemini 3.1 Pro Preview
Gemini 3.1 Pro Preview is positioned for stronger thinking, improved token efficiency, reliability and grounded behavior compared with earlier Gemini 3 Pro preview usage.
Use this class of model when the task combines code, policy, long context, tool precision and multi-step reasoning. Do not use it as the default for every chat message unless the business economics justify it.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini 3.1 Pro Preview into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Best fit | Complex software engineering, deep analysis, agent planning, tool-heavy workflows and high-stakes synthesis. |
| Prompt shape | Detailed system instruction, clear acceptance criteria, evidence requirements and typed output contract. |
| Tool shape | Narrow tools with schemas, validation, retries and human confirmations for write actions. |
| Evaluation | Repository tasks, multi-document questions, code patch correctness, tool call precision and refusal behavior. |
| Operations | Preview lifecycle watch, fallback model, timeout budget and cost ceiling. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Reserve Pro-class usage for tasks that need depth, precision or difficult trade-off reasoning.
- Give explicit criteria: what to optimize, what to avoid, what evidence to cite and when to refuse.
- Pair with structured outputs for patch plans, JSON diagnostics and deployment checklists.
- Run repository-level evals for coding workflows before letting it edit production code.
- Keep a fallback path to a stable GA model or a smaller model for degraded mode.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Preview availability and pricing may change.
- Reasoning quality does not remove the need for tests and peer review.
- Long responses can hide unsupported assumptions.
- Tool precision must be measured with real tool schemas, not anecdotal demos.
- Cost concentration can occur if every workflow escalates to Pro.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Patch generator | Use for full-function Django patches with testbench context and risk analysis. |
| Migration doctor | Use for complex diagnosis where multiple migrations, tables and states interact. |
| SRDF design | Use for orchestration design, failure-mode analysis and runbook synthesis. |
| Review cockpit | Use Pro only after a first-pass cheaper model identifies high-risk cases. |
| Release gate | Require test evidence before accepting generated code. |
Gemini 3.1 Pro Preview principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 05
Gemini 3 Flash Preview
Gemini 3 Flash Preview
Flash-class models are the workhorses of interactive AI systems. They are designed for applications that need strong capability with better responsiveness and lower cost than the top Pro route.
Use Flash for assistants, interactive coding support, workflow routing, multimodal understanding, fast summarization, customer-facing help and agent loops where latency matters.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini 3 Flash Preview into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Interaction layer | Chat, web assistants, internal copilots, multimodal Q&A and agent loops. |
| Media layer | Images, PDFs, audio/video inputs where fast multimodal interpretation is needed. |
| Tool layer | Function calling, URL context, Google Search, File Search and code execution where allowed. |
| Routing layer | Escalate only difficult failures or high-risk tasks to Pro. |
| Cost layer | Use caching, token caps and max output limits to protect unit economics. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Make Flash the default interactive route when quality is sufficient.
- Add escalation criteria to Pro for uncertain, complex or high-risk cases.
- Use short structured outputs for routing decisions and extraction tasks.
- Measure p95 latency and cost per completed workflow, not only model response time.
- Cache repeated context and keep common instructions at the front of prompts.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Fast models can be overused for tasks requiring deeper reasoning.
- Interactive agents can multiply token usage through loops.
- Multimodal input cost can grow quickly with large files and video.
- Preview endpoints need lifecycle monitoring.
- Latency targets can be broken by external tools and retrieval, not just model speed.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Admin assistant | Use Flash for internal search and operational summaries. |
| Guide editor | Use Flash for chapter rewrites, cards, summaries and modal drafts. |
| Code triage | Use Flash to classify errors before escalation. |
| RAG Q&A | Use Flash as a default answerer grounded by Vertex AI Search. |
| Product UX | Keep user-facing response latency under control. |
Gemini 3 Flash Preview principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 06
Gemini 3.1 Flash-Lite Preview
Gemini 3.1 Flash-Lite Preview
Flash-Lite is the economics layer of the Gemini portfolio. It is suitable for tasks where throughput, latency and cost matter more than maximum reasoning depth.
Typical uses include classification, simple extraction, lightweight summarization, routing, metadata generation, deduplication, content labeling and first-pass triage.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini 3.1 Flash-Lite Preview into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Bulk layer | High-volume jobs: tags, labels, summaries, metadata and normalization. |
| Router layer | Determine whether a task should be answered directly, retrieved, escalated or rejected. |
| Extraction layer | Convert simple inputs into JSON with schemas and validation. |
| Guard layer | Use for pre-checks, policy labels and confidence screening. |
| Escalation layer | Send uncertain or high-risk outputs to Flash or Pro. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use Flash-Lite as a first-pass worker in pipelines.
- Keep prompts short and schemas strict.
- Add confidence scoring and escalation rules.
- Benchmark error rate against human or stronger-model labels.
- Batch repetitive jobs and monitor aggregate cost.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A cheap model can become expensive when failures require manual cleanup.
- Do not ask Lite models to handle legal, financial, medical or production-change decisions alone.
- Schema compliance must be validated in code.
- Bulk jobs can silently amplify systematic mistakes.
- Preview status requires caution in long-lived production commitments.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Glossary extraction | Use for bulk page scanning, keyword tagging and category suggestions. |
| Migration catalog | Use for initial error classification before deeper diagnosis. |
| Guide factory | Use for card summaries and tags, not final high-stakes architecture. |
| FinOps | Track cost per thousand processed pages. |
| Quality gate | Sample outputs and route failures to stronger models. |
Gemini 3.1 Flash-Lite Preview principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 07
Long Context and Repository-Scale Reasoning
Long Context and Repository-Scale Reasoning
Gemini long context can ingest very large inputs, including long documents, codebases, transcripts, logs and multimodal material. This is a strategic advantage for enterprise analysis.
Long context should not replace retrieval architecture blindly. The right design chooses between direct long-context loading, RAG, context caching and staged summarization based on cost, latency, freshness and auditability.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Long Context and Repository-Scale Reasoning into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Direct long context | Best for bounded packages: one repo snapshot, one contract set, one incident bundle or one export. |
| RAG | Best for large, changing knowledge where source-level retrieval and permissions matter. |
| Context cache | Best for repeated analysis over the same large prefix. |
| Chunked pipeline | Best for extraction, classification and summarization at scale. |
| Evidence layer | Always return source pointers, line references, document IDs or file paths where possible. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use long context when the full input structure matters and fits the model limits.
- Use RAG when the corpus is large, dynamic or permission-sensitive.
- Use context caching when many questions reuse the same corpus.
- Ask for evidence tables and uncertainty notes, not just prose.
- Measure whether long context improved accuracy enough to justify cost.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Large input does not guarantee the model attends to the right evidence.
- Repeated long prompts can become financially dangerous without caching.
- Sensitive data may be duplicated in logs or prompts if governance is weak.
- Long outputs can become vague if the task is underspecified.
- Source verification is still required for production decisions.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Codebase reviews | Load a bounded repository snapshot for architecture explanations and risk mapping. |
| MigrateSafe | Analyze migration files, models, SQL previews and testbench logs in one context package. |
| SRDF | Load complete run logs and event flows for incident analysis. |
| Guide conversion | Ingest large PDFs or HTML pages to generate dense chapter structures. |
| Cache strategy | Cache stable project context during patch sessions. |
Long Context and Repository-Scale Reasoning principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 08
Native Multimodal Gemini
Native Multimodal Gemini
Gemini is designed around native multimodality. A production system can ask questions over screenshots, PDFs, diagrams, images, video frames, audio, charts, invoices, forms and mixed text-image material.
The engineering challenge is to control media resolution, file preparation, token cost, source traceability and failure handling.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Native Multimodal Gemini into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Image understanding | Screenshots, UI analysis, diagrams, charts, forms and photos. |
| Document understanding | PDFs with text, layout, tables, images, equations and annotations. |
| Video understanding | Summaries, key moments, compliance reviews, training material and incident footage. |
| Audio understanding | Transcripts, meeting summaries, voice interfaces and call analytics. |
| Multimodal retrieval | Embedding text, images, audio, video and PDFs into a shared search architecture. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Normalize file inputs and metadata before sending them to the model.
- Set media resolution according to task quality requirements.
- Ask for structured outputs with page, timestamp, region or object references.
- Use specialized pipelines when extraction accuracy must be audited.
- Separate exploratory multimodal analysis from regulated extraction workflows.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Visual interpretation can be confident but wrong for tiny text, dense diagrams or ambiguous charts.
- Media-heavy prompts can increase latency and cost.
- User-uploaded files may contain secrets or personal data.
- Source traceability is harder with images and video than plain text.
- OCR-like extraction must be validated if used for finance, legal or operational decisions.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Screenshots | Diagnose UI errors, console screenshots and admin pages. |
| PDF guides | Convert dense PDFs into IDEO-Lab modal content. |
| Database diagrams | Explain schemas, flows and migration impacts from diagrams. |
| Video lessons | Summarize technical recordings into structured guides. |
| Audit mode | Require page and image evidence for extracted claims. |
Native Multimodal Gemini principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 09
Document Understanding and PDF Workflows
Document Understanding and PDF Workflows
Gemini document understanding goes beyond text extraction by using native vision to interpret document layout and context. This matters for PDFs with tables, figures, signatures, equations, screenshots and mixed content.
A production document workflow should separate ingestion, extraction, validation, indexing and question answering.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Document Understanding and PDF Workflows into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Ingestion | Upload PDFs, images or Cloud Storage references with metadata and access policy. |
| Understanding | Ask Gemini to interpret layout, sections, tables, figures and cross-references. |
| Extraction | Use JSON schema for tables, fields, contract clauses or invoice records. |
| Validation | Run deterministic checks for totals, dates, IDs, references and required fields. |
| Indexing | Store extracted content, source positions and embeddings for later RAG. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Define the document contract before extraction: fields, tables, optional fields, confidence and source references.
- Use medium or high media resolution only when the task requires it.
- Validate extracted numbers and dates with normal code.
- Store original file hash and page references for auditability.
- Use RAG only after ingestion quality is measured.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Layout-heavy PDFs can create extraction mistakes even when text looks obvious to humans.
- Tables can be misaligned; totals and columns must be validated.
- OCR-like output should not be trusted without checks in financial or legal contexts.
- Sensitive documents require retention and access policy.
- Prompt-only extraction is fragile without schemas and validators.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Tool docs | Ingest installation guides and manuals into modal-based HTML guides. |
| MigrateSafe reports | Extract DDL tables and error matrices from PDFs. |
| Contracts | Summarize licensing and deployment obligations with source pages. |
| Invoice-like docs | Validate structured fields with Python after Gemini extraction. |
| Knowledge base | Use PDF ingestion plus embeddings to enrich IDEO-Lab search. |
Document Understanding and PDF Workflows principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 10
Grounding with Google Search
Grounding with Google Search
Grounding with Google Search allows Gemini applications to use current web information and return more source-aware answers. It is a major feature when answers depend on fresh public facts.
Grounding is not the same as perfect truth. The application still needs citation display, source selection rules, user-facing uncertainty and refusal behavior.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Grounding with Google Search into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| User query | Ask a question that may require fresh public information. |
| Search grounding | Gemini retrieves and uses relevant web content through Google Search grounding. |
| Answer synthesis | The model integrates sources and returns a final response with grounding metadata. |
| UI layer | Display sources, caveats and date-sensitive warnings. |
| Policy layer | Block or review sensitive workflows where fresh web content is insufficient evidence. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use Search grounding for public, current, factual topics.
- Prefer enterprise data grounding for internal facts.
- Display citations or source references in the product UI.
- Add a fallback when grounding returns weak or conflicting sources.
- Log which workflows used web grounding for audit and cost analysis.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Search results can be stale, promotional, incomplete or contradictory.
- The model can over-synthesize from weak sources.
- Public web grounding is not enough for internal policy or confidential company knowledge.
- Source display must be designed; hidden grounding is not transparent enough.
- Freshness-dependent answers need dates and caveats.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| LLM guides | Use web grounding to refresh model names, pricing pages and release notes. |
| Tech watch | Generate summaries of new Django, PostgreSQL or security changes. |
| Public research | Build cited guide sections with official docs first. |
| Risk rule | Never use generic web grounding for private IDEO-Lab internals. |
| Audit | Store source URLs and retrieval timestamp for generated guides. |
Grounding with Google Search principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 11
Vertex AI Search Grounding and Enterprise RAG
Vertex AI Search Grounding and Enterprise RAG
Vertex AI Search grounding connects Gemini to enterprise data stores, websites or document corpora. This is the serious route for internal knowledge systems because it can be designed around data sources, permissions, metadata and source-backed responses.
Enterprise RAG is not just vector search. It includes ingestion, chunking, metadata, ranking, access control, citations, evaluation and feedback loops.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Vertex AI Search Grounding and Enterprise RAG into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Data sources | Websites, PDFs, manuals, tickets, runbooks, SQL exports, code docs and enterprise repositories. |
| Indexing | Chunking, parsing, embeddings, metadata and refresh schedule. |
| Retrieval | Vertex AI Search or RAG Engine retrieves candidate evidence. |
| Grounded generation | Gemini answers using retrieved evidence and cites sources. |
| Governance | Source access, freshness, retention, logging, evaluation and redaction. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Start with one corpus and one business question family.
- Design metadata fields before ingestion: app, version, date, owner, confidentiality, language, file type.
- Evaluate retrieval separately from generation.
- Require citations and source snippets in the UI.
- Monitor unanswered questions and add content, not only prompt hacks.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Poor chunking can destroy answer quality.
- Embeddings alone may fail on exact IDs, dates and technical names without hybrid search.
- Users may receive answers from documents they should not access if permissions are not enforced.
- Outdated documents must be demoted or archived.
- RAG answers can still hallucinate if the prompt allows unsupported synthesis.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guides search | Index all IDEO-Lab guides with app, technology and modal metadata. |
| Migration KB | Ground MigrateSafe answers in error catalog, patch logs and testbench results. |
| SRDF KB | Ground DR answers in runbooks, event tables and phase logs. |
| DB tuning | Index Oracle, PostgreSQL and MariaDB tuning guides for DBA assistants. |
| User portal | Expose cited answers with source modal links. |
Vertex AI Search Grounding and Enterprise RAG principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 12
File Search and Multimodal Embeddings
File Search and Multimodal Embeddings
Gemini embedding models can support semantic search across multiple modalities, which changes the architecture of knowledge systems. A screenshot, PDF, audio clip, document and text note can become searchable in a unified retrieval space.
For production, embeddings must be paired with metadata, access control, indexing freshness, chunk strategy and evaluation.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates File Search and Multimodal Embeddings into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Embedding model | Convert text, images, video, audio and PDFs into vectors where supported. |
| Metadata index | Store titles, owners, dates, pages, timestamps, confidentiality and application scope. |
| Vector search | Retrieve semantically related items across modalities. |
| Hybrid search | Combine vector similarity with keyword, filters and exact matching. |
| Grounded answer | Generate final responses with Gemini and surface the retrieved evidence. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create separate pipelines for text, PDF, image, audio and video metadata extraction.
- Use source IDs and page or timestamp references everywhere.
- Test retrieval quality with known-answer questions.
- Use hybrid search for code, table names, error codes and migration IDs.
- Re-embed documents after significant changes or model upgrades.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Vector similarity can retrieve plausible but wrong sources.
- Embeddings can be expensive at scale without ingestion budgets.
- Access control must be enforced before retrieval or before answer generation.
- Multimodal source references are harder to present clearly in the UI.
- Embedding model upgrades may change retrieval behavior.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Screenshot library | Search previous UI errors and console screenshots by semantic similarity. |
| PDF library | Search old database and mainframe documents. |
| Guide search | Link page terms to modal explanations and source guide sections. |
| Code search | Combine keyword and vector retrieval for Django codebase analysis. |
| Audit | Store retrieval traces with each generated answer. |
File Search and Multimodal Embeddings principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 13
Function Calling and Tool Contracts
Function Calling and Tool Contracts
Function calling lets Gemini bridge natural language and real systems by selecting functions and generating arguments. This is the foundation for assistants that can query databases, create tickets, read logs, update records or trigger workflows.
The engineering rule is simple: never let the model perform an action without a typed tool contract, validation, permissions and audit.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Function Calling and Tool Contracts into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Tool declaration | Name, description, parameters and schema exposed to Gemini. |
| Model decision | Gemini selects whether and how to call a function. |
| Validation layer | Application validates arguments, permissions, rate limits and business rules. |
| Execution layer | The backend executes the real API or database call. |
| Response layer | Tool results return to Gemini for final explanation or next step. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Start with read-only tools before write tools.
- Make tool names and descriptions precise and non-overlapping.
- Validate every argument server-side.
- Require confirmation for destructive, financial, customer-facing or production actions.
- Log tool call intent, arguments, result, user, timestamp and model ID.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Prompt injection can try to manipulate tool calls.
- Ambiguous tool descriptions cause wrong function selection.
- The model may generate syntactically valid but semantically dangerous arguments.
- A chain of small tool calls can create unexpected side effects.
- Audit trails become essential once agents touch production systems.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| MigrateSafe tools | Expose read-only commands for status, migration list, DDL preview and testbench output. |
| SRDF tools | Expose run status, group finalization state and transport batch summaries. |
| Admin tools | Create tickets or reports only after human confirmation. |
| Code tools | Allow repository reads first; require review before writes. |
| Security | Never expose raw shell or SQL execution without a locked abstraction. |
Function Calling and Tool Contracts principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 14
Built-in Tools: Search, URL Context, Maps, File Search, Code Execution
Built-in Tools: Search, URL Context, Maps, File Search, Code Execution
Gemini built-in tools can run inside a single API workflow depending on tool and model support. They can ground answers, inspect URLs, search files, run code or use Google services without building every integration from scratch.
Built-in tools are powerful but still require product-level decisions about data, logging, display, cost and user trust.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Built-in Tools: Search, URL Context, Maps, File Search, Code Execution into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Google Search | Fresh public web grounding and citation-style workflows. |
| URL Context | Analyze specific URLs supplied by the user or application. |
| File Search | Retrieve from user-provided or indexed file content where supported. |
| Code Execution | Let the model write and run code for calculation or analysis in a controlled environment. |
| Google Maps | Location-aware tool use for supported workflows. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Enable only the built-in tools required by the workflow.
- Tell users when an answer used external retrieval or code execution.
- Use code execution for calculations and data analysis, not hidden production changes.
- Combine built-in tools with custom functions only when the control model is clear.
- Add timeouts and fallback messages for tool failures.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Tool results can be incomplete or unavailable.
- Code execution is not a replacement for secured application logic.
- External URL content can contain prompt injection.
- Maps or location workflows may involve privacy-sensitive data.
- A tool-enabled model can feel more authoritative than it really is.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Research guides | Use Search and URL Context for official docs verification. |
| Calculations | Use code execution for token cost estimates and benchmark summaries. |
| File Q&A | Use File Search for uploaded technical packs. |
| Ops assistant | Keep production actions in custom tools with stricter gates. |
| UI | Show a clear badge when tools were used. |
Built-in Tools: Search, URL Context, Maps, File Search, Code Execution principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 15
Live API and Real-Time Voice / Video
Live API and Real-Time Voice / Video
The Live API enables low-latency interaction patterns where users speak, show, ask and receive responses in near real time. This changes UX from request/response chat to continuous multimodal collaboration.
Live systems require additional controls: session state, interruption handling, audio privacy, streaming errors, tool permissions and safety limits.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Live API and Real-Time Voice / Video into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Session | A live session handles streaming inputs, state, model configuration and output modality. |
| Input streams | Audio, video or multimodal streams with buffering and latency constraints. |
| Tool calls | Function calling can enrich live agents with external actions. |
| Output streams | Voice, text or multimodal response depending on product design. |
| Safety boundary | Mute, stop, confirmation, content policy, logging and retention choices. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Start with internal demos before customer-facing live agents.
- Define audio and video retention policy before collecting real sessions.
- Keep tool calls read-only until interruption and confirmation UX are robust.
- Measure latency as users experience it: capture, model, tool and playback.
- Add visual indicators for recording, processing and tool use.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Real-time agents can capture sensitive information accidentally.
- Latency spikes break trust faster than text chat delays.
- Voice responses can sound authoritative even when uncertain.
- Live tool calls can create rapid side effects if not gated.
- Accessibility, consent and recording policy must be explicit.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Support assistant | Voice walkthroughs for installing IDEO-Lab tools. |
| Ops cockpit | Live explanation of SRDF dashboard states without write access. |
| Training | Interactive voice/video lessons for Django migration concepts. |
| Debug sessions | Screen-share style analysis of admin and console errors. |
| Control | No production change without explicit typed confirmation. |
Live API and Real-Time Voice / Video principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 16
Structured Outputs and JSON Schemas
Structured Outputs and JSON Schemas
Structured outputs are essential for production AI. They turn free text into typed contracts that backend code can validate, store, diff, test and act on.
The safest design treats Gemini as a probabilistic generator whose output must still be parsed, validated and tested by deterministic code.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Structured Outputs and JSON Schemas into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Schema | Define fields, types, enums, required properties and nested structures. |
| Prompt | Explain task, source constraints and refusal conditions. |
| Generation config | Request JSON or schema-constrained output where supported. |
| Validator | Reject malformed, incomplete or policy-violating outputs. |
| Executor | Only after validation, use the output for UI, database, workflows or tools. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use schemas for extraction, routing, risk scoring, patch plans and agent actions.
- Prefer enums over free-form categories.
- Add confidence and evidence fields where decisions need review.
- Validate with application code and return repair prompts only when safe.
- Version schemas like API contracts.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Valid JSON can still be factually wrong.
- Overly complex schemas reduce reliability.
- Optional fields can hide missing evidence.
- Downstream systems may trust the output too much.
- Schema changes can break pipelines without migration strategy.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Migration reports | Return structured issue lists with app, migration, table, severity and action. |
| Guide cards | Generate title, tags, chips, summary and modal sections as JSON. |
| SRDF status | Emit phase status objects and incident summaries. |
| Glossary | Output category IDs, aliases, confidence and source snippets. |
| Release gate | Validate AI output before any code or DB operation. |
Structured Outputs and JSON Schemas principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 17
Context Caching and Cost Engineering
Context Caching and Cost Engineering
Context caching is a core FinOps feature for long-context Gemini workloads. It helps when many requests share the same large prefix: system instructions, codebase context, document bundle, video, audio or repeated knowledge pack.
A good caching design reduces cost and latency while respecting data retention, region and lifecycle constraints.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Context Caching and Cost Engineering into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Implicit caching | Automatic caching behavior where supported, useful for repeated prompt prefixes. |
| Explicit caching | Application-created cache resources for controlled reuse of large context. |
| TTL strategy | Set cache lifetime according to workflow duration and data sensitivity. |
| Cost model | Account for cache creation, storage where applicable and discounted reuse. |
| Security model | Understand region, expiration, logging and content retention choices. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Cache stable context: repository snapshot, long manual, video transcript or policy pack.
- Put shared content at the beginning of prompts to maximize cache hits.
- Use explicit cache names and metadata for workflow ownership.
- Do not cache secrets or regulated data unless policy allows it.
- Measure cache hit rate and savings per workflow.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Caching sensitive content can create retention risk.
- Cache misses can surprise budgets if prompts are not stable.
- Expired caches must be recreated; applications need fallback logic.
- Model or region constraints can affect cache availability.
- Caching does not fix poor prompt or retrieval design.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Patch sessions | Cache a Django project context while generating multiple patches. |
| MigrateSafe docs | Cache the error catalog and command reference during analysis. |
| SRDF runbooks | Cache stable architecture docs for repeated incident Q&A. |
| Guide factory | Cache source material when generating many modal chapters. |
| Billing | Add cache hit metrics to internal AI usage reports. |
Context Caching and Cost Engineering principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 18
Gemini Embeddings and Semantic Search
Gemini Embeddings and Semantic Search
Embeddings transform content into vectors for semantic retrieval. Gemini embeddings can power search, recommendations, deduplication, clustering, RAG and multimodal knowledge systems.
Production embedding systems require deterministic metadata, access control, re-indexing strategy and query evaluation.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini Embeddings and Semantic Search into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Content extraction | Parse HTML, PDFs, code, tickets, logs, images or media metadata. |
| Chunking | Split content by semantic structure, not arbitrary length only. |
| Embedding | Generate vectors with selected embedding model and version. |
| Index | Store vectors plus metadata in vector database or managed retrieval service. |
| Evaluation | Test recall, precision, freshness and answer grounding. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use hybrid retrieval for technical content with IDs, error codes and exact names.
- Store model version with every embedding.
- Re-index when extraction logic, chunking or model changes.
- Create a benchmark question set before optimizing ranking.
- Enforce permission filters before content reaches the model.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Semantic similarity can retrieve related but incorrect chunks.
- Long chunks reduce precision; tiny chunks lose context.
- Embedding drift affects comparisons across model versions.
- Access control mistakes can leak documents through search.
- RAG quality is limited by ingestion quality.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Tech glossary | Embed extracted guide sections and technical definitions. |
| Modal linking | Find related modals and keywords automatically. |
| Migration errors | Retrieve historical fixes by error signature. |
| Database docs | Search tuning guides by symptom and vendor. |
| Feedback loop | Log bad retrieval and improve chunking. |
Gemini Embeddings and Semantic Search principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 19
Vertex AI Agents, Agent Engine and Agent Builder
Vertex AI Agents, Agent Engine and Agent Builder
Agents combine models, tools, memory, instructions and orchestration. Vertex AI provides enterprise-oriented surfaces for developing, deploying and evaluating agents in a managed Google Cloud environment.
The serious agent question is not whether the model can plan. It is whether the agent can be limited, observed, tested, interrupted and held accountable.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Vertex AI Agents, Agent Engine and Agent Builder into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Agent definition | Instructions, model route, tool contracts, memory policy and response style. |
| Runtime | Deployment environment for running the agent and handling requests. |
| Tools | APIs, functions, retrieval, code execution or enterprise connectors. |
| Evaluation | Task completion, tool accuracy, safety, latency and regression metrics. |
| Governance | IAM, logs, owners, escalation, budget and lifecycle management. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Build one narrow agent before attempting a general enterprise agent.
- Start with read-only tools and explicit handoff to humans.
- Use task-based evals that measure completion, not just answer quality.
- Log each tool call and decision boundary.
- Define kill switches, rate limits and fallback responses.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Agents can loop, over-call tools or pursue the wrong goal.
- A tool-enabled agent can create business impact from a prompt injection.
- Memory and state can retain sensitive information unexpectedly.
- Evaluation must include adversarial and failure cases.
- Enterprise users may overtrust autonomous behavior.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Migration agent | Diagnose migrations, propose patches and request approval before code changes. |
| SRDF agent | Explain phase state and recommend operator actions without executing destructive operations. |
| Guide agent | Turn source packs into HTML guide drafts with review gates. |
| Support agent | Answer toolbox questions grounded in docs. |
| Governance | Every agent gets owner, scope, tools, evals and rollback. |
Vertex AI Agents, Agent Engine and Agent Builder principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 20
Deep Research and Research Agents
Deep Research and Research Agents
Research agents combine web retrieval, document understanding, synthesis, source tracking and report generation. Gemini can support multimodal research inputs including PDFs and images depending on API and product surface.
A reliable research workflow needs source hierarchy, freshness checks, citation display, conflict handling and explicit uncertainty.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Deep Research and Research Agents into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Question framing | Define topic, time range, source preference and output format. |
| Source gathering | Use web grounding, URL context, uploaded docs or enterprise search. |
| Evidence extraction | Collect claims, dates, metrics, quotes and source metadata. |
| Synthesis | Compare sources, detect conflicts and generate structured report. |
| Review | Human checks source quality and final recommendations. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Prefer official docs for product capabilities and pricing.
- Ask for a claim-source table before a final narrative.
- Separate facts from interpretation.
- Record research date and model used.
- Create update tasks for volatile guides.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Fresh public information may be wrong or incomplete.
- Research agents can over-cite weak sources.
- Long reports can blur source-specific claims.
- Competitive claims need caution and clear dates.
- Automated research still needs human review before publication.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| LLM landscape | Generate updated Claude, Mistral, Gemini and OpenAI comparison guides. |
| Django watch | Track Django, PostgreSQL and Python release changes. |
| Vendor docs | Produce source-grounded summaries for cloud AI pricing and model lifecycle. |
| LinkedIn posts | Turn research into accurate educational posts. |
| Audit | Keep source links in each generated guide footer. |
Deep Research and Research Agents principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 21
Vertex AI Model Garden and Open Models
Vertex AI Model Garden and Open Models
Model Garden is the discovery and deployment layer for Google, partner and open models in Vertex AI. It supports a broader model strategy than Gemini alone.
This matters when an enterprise wants model choice, open models, specialized models, cost comparison or private serving in a single governed Cloud environment.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Vertex AI Model Garden and Open Models into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Discovery | Find Google, partner and open models for text, vision, code, embeddings and other tasks. |
| Evaluation | Compare models before deployment with task-specific metrics. |
| Deployment | Deploy models through managed endpoints, MaaS or supported serving frameworks. |
| Optimization | Use quantization, serving choices, accelerator selection and throughput settings. |
| Governance | Control model access with IAM and organization policies. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use Model Garden when Gemini is not the only candidate.
- Compare models on real workloads, not only marketing benchmarks.
- Document license, owner, region, cost and endpoint for every deployed model.
- Use managed serving when platform reliability matters more than raw flexibility.
- Restrict model discovery and deployment rights in enterprise projects.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Open model licenses and acceptable use rules differ widely.
- Deploying many models creates operational sprawl.
- Self-managed endpoints require monitoring, scaling and security expertise.
- Benchmark wins may not translate to production workflows.
- Organization policy must prevent shadow model deployment.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Open model lab | Test Gemma and third-party models against MigrateSafe tasks. |
| Cost lab | Compare Gemini hosted cost with self-deployed open models. |
| Product catalog | Offer multiple AI backends behind one Django service contract. |
| Governance | Keep model registry with license and endpoint details. |
| Fallback | Use smaller open models for offline or privacy-sensitive tools. |
Vertex AI Model Garden and Open Models principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 22
OpenAI-Compatible Integration and Migration
OpenAI-Compatible Integration and Migration
Many codebases already use OpenAI-style chat abstractions. Google documentation includes patterns for calling Vertex AI models with OpenAI libraries, which can reduce migration friction.
Compatibility is useful, but a serious migration should not hide model differences in tools, schemas, streaming, safety, context and multimodal handling.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates OpenAI-Compatible Integration and Migration into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Adapter layer | Wrap provider-specific APIs behind one internal service. |
| Schema layer | Normalize messages, tools, JSON schema, files and streaming events. |
| Routing layer | Choose provider and model by workflow, cost, policy and availability. |
| Observability | Log provider, model ID, token usage, latency, errors and quality outcome. |
| Governance | Keep provider-specific restrictions and data rules visible. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Build a provider adapter instead of scattering SDK calls across Django views.
- Create contract tests for streaming, tool calls and structured output.
- Do not assume identical behavior for the same prompt across providers.
- Use feature flags to migrate workloads gradually.
- Keep rollback to the previous provider until evals pass.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- API compatibility does not mean behavioral equivalence.
- Tool schemas, safety settings and file handling may differ.
- A hidden provider switch can break pricing assumptions.
- Errors and rate limits need provider-specific handling.
- Legal and data processing terms differ by vendor.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Provider service | Create a Django AIProviderClient abstraction for OpenAI, Anthropic, Mistral and Gemini. |
| Budget routing | Route cheap tasks to cost-efficient providers and hard tasks to stronger models. |
| Patch safety | Keep provider-specific code generation evals. |
| Docs | Expose which provider generated each guide or patch. |
| Licensing | Record provider terms for each released tool. |
OpenAI-Compatible Integration and Migration principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 23
Gemini for Software Engineering
Gemini for Software Engineering
Gemini can be valuable for software engineering because it can combine long context, code reasoning, tool use and structured output. The highest-value workflows are not random code snippets, but repository-aware analysis and test-driven patch planning.
A safe coding assistant should never bypass tests, review, version control or production gates.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gemini for Software Engineering into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Repository context | Relevant files, dependency map, error logs, tests and constraints. |
| Task contract | Bug, feature, refactor, migration, security review or documentation. |
| Patch plan | Files changed, functions replaced, risks, tests and rollback. |
| Code generation | Generate complete functions or files with minimal ambiguity. |
| Validation | Run tests, linters, migrations and security checks. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Provide exact files and errors, not vague descriptions.
- Ask for complete function definitions when modifying existing Python code.
- Require a test plan and rollback note for every patch.
- Use structured output for file list, changes, risks and commands.
- Keep human review before merging generated code.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Generated code may compile but violate project architecture.
- Large context can include outdated or irrelevant files.
- The model may invent APIs if docs are missing.
- Security-sensitive code needs manual review.
- Tests must be real, not merely described.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Django patches | Use Gemini with full-function replacement rules and service-based patches. |
| MigrateSafe | Generate diagnosis and patch plans from testbench failures. |
| SRDF | Explain concurrent capture, finalization and batch constraints. |
| Docs | Generate install guides and code examples from source. |
| CI | Run generated patches through testbench and release gate. |
Gemini for Software Engineering principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 24
Batch, Webhooks and Asynchronous AI Workflows
Batch, Webhooks and Asynchronous AI Workflows
Not every AI task should be interactive. Bulk document processing, embeddings, classification, dataset cleanup and report generation often belong in asynchronous pipelines.
Batch and webhook-style patterns help reduce polling, isolate workloads and integrate AI with backend job systems.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Batch, Webhooks and Asynchronous AI Workflows into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Job creation | Create a batch task with inputs, model, config and output location. |
| Queueing | Separate user request from background processing. |
| Completion | Use webhook or long-running operation completion where supported. |
| Post-processing | Validate JSON, store outputs, update status and index content. |
| Monitoring | Track job count, failures, cost and retry behavior. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use batch for bulk pages, PDFs, logs, tickets and embedding refresh.
- Make every input item idempotent with a stable external ID.
- Store raw response, normalized output and validation status separately.
- Retry only safe failures and cap retries.
- Expose admin progress and final reports.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Bulk errors can be expensive and hard to inspect.
- One bad prompt can multiply mistakes across thousands of records.
- Webhook endpoints must be authenticated and idempotent.
- Large batches need cost estimates before launch.
- User-visible status should not depend on polling loops only.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide factory | Batch-generate card metadata for hundreds of HTML pages. |
| Glossary | Batch classify extracted terms and synonyms. |
| OCR pipeline | Batch process technical PDFs. |
| MigrateSafe | Analyze historical failure logs offline. |
| Admin | Show progress bars and downloadable synthesis reports. |
Batch, Webhooks and Asynchronous AI Workflows principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 25
Security, IAM, Data Residency and Governance
Security, IAM, Data Residency and Governance
Vertex AI brings Gemini into the Google Cloud governance model. This matters for enterprise workloads that need IAM, service accounts, project isolation, regions, quotas, logging, data residency and procurement controls.
The architecture must define who can call models, which data can be sent, where logs go, what is retained and how incidents are handled.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Security, IAM, Data Residency and Governance into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Identity | Users, service accounts, workload identity and least-privilege IAM roles. |
| Project boundary | Separate dev, staging and production Google Cloud projects. |
| Data boundary | Classify prompts, files, logs, caches and model outputs. |
| Network boundary | Private access, VPC Service Controls or approved egress patterns where required. |
| Audit boundary | Log model use, tool calls, data class, user and cost metadata. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create an AI use policy before enabling production workloads.
- Use separate service accounts for each application feature.
- Block sensitive data classes from non-approved surfaces.
- Review context caching and logging retention for confidential material.
- Add an incident process for leaked prompts, unsafe outputs or wrong actions.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Developers may test confidential data in uncontrolled AI Studio experiments.
- Service accounts with broad permissions can turn tool use into a security incident.
- Logs can accidentally store secrets or personal data.
- Region and residency requirements may be misunderstood.
- Governance that is too vague will not survive a real audit.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Provider policy | Define which data classes can use Gemini API vs Vertex AI. |
| Secrets | Never put API keys, DB passwords or license secrets into prompts. |
| Admin logs | Log every AI-generated patch, guide and operational recommendation. |
| RBAC | Restrict AI tools in Django admin by role. |
| Compliance | Keep a model registry and audit report for each productized utility. |
Security, IAM, Data Residency and Governance principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 26
Gen AI Evaluation and Release Gates
Gen AI Evaluation and Release Gates
The Gen AI evaluation service in Vertex AI helps teams measure model and agent behavior using task-specific metrics. Evaluation is the difference between a demo and a production capability.
A real evaluation program includes golden datasets, human review, model-based metrics, regression baselines, adversarial tests and cost/latency thresholds.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Gen AI Evaluation and Release Gates into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Dataset | Representative prompts, expected outputs, sources, edge cases and failure cases. |
| Metrics | Accuracy, groundedness, helpfulness, tool success, latency, cost and safety. |
| Comparison | Current prompt/model vs candidate prompt/model. |
| Agent eval | Task completion, step quality, tool calls and goal achievement. |
| Release gate | Go/no-go decision with evidence and rollback plan. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create eval sets from real production failures.
- Keep separate evals for extraction, RAG, code, agents and summarization.
- Run evals before model upgrade, prompt change or tool change.
- Track cost and latency alongside quality.
- Make the release decision visible in CI/CD or admin reports.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Generic benchmark scores may not predict your workflow quality.
- Small eval sets can overfit prompts.
- Model-based evaluation can inherit model bias and must be sampled by humans.
- Agents require task completion evaluation, not only final text scoring.
- Skipping regression tests makes upgrades dangerous.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| MigrateSafe evals | Use historical migration failures as golden cases. |
| Guide evals | Check structure, factuality, density, sources and no broken JS. |
| SRDF evals | Evaluate incident summaries against known run states. |
| Code evals | Run generated patches through testbench. |
| CI/CD | Expose GO/NO GO summaries for deployment pipelines. |
Gen AI Evaluation and Release Gates principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 27
Quotas, Rate Limits and Provisioned Throughput
Quotas, Rate Limits and Provisioned Throughput
Production Gemini applications need capacity planning. Rate limits regulate API usage; enterprise workloads may need quotas, workload shaping, regional planning or Provisioned Throughput depending on scale and requirements.
Capacity design is not only requests per minute. It includes token volume, media size, tool latency, batch workload, concurrency, model choice and fallback behavior.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Quotas, Rate Limits and Provisioned Throughput into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Quota view | Per-project, per-model and per-region limits depending on surface and account. |
| Traffic shaping | Queueing, concurrency limits, retry policy and backpressure. |
| Provisioned capacity | Reserve capacity for predictable enterprise workloads where available. |
| Fallbacks | Route to smaller models, cached answers or degraded responses when capacity is constrained. |
| Monitoring | Track RPM, TPM, latency, errors, queue depth and cost. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Estimate peak usage with tokens, not just request count.
- Apply per-feature rate limits in the application.
- Use exponential backoff for transient errors but cap retries.
- Separate interactive traffic from batch jobs.
- Define degraded mode before launch.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A single batch job can starve interactive users.
- Retries can amplify outages and costs.
- Media inputs can consume capacity unexpectedly.
- Provisioned capacity tied to model lifecycle needs migration planning.
- Quota issues can look like application bugs if not monitored.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Django middleware | Add per-user and per-feature AI rate limits. |
| Job queues | Separate guide generation from chat Q&A. |
| Admin dashboard | Show AI quota, latency and failure summaries. |
| Fallback | Use cached docs search if model call fails. |
| Pricing | Set product limits based on real token budgets. |
Quotas, Rate Limits and Provisioned Throughput principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 28
Token Economics and FinOps
Token Economics and FinOps
LLM cost is a product architecture issue. Gemini applications must control input tokens, output tokens, model selection, context reuse, media resolution, batch jobs, retries and tool loops.
The right metric is cost per completed business workflow, not cost per isolated prompt.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Token Economics and FinOps into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Routing | Choose model by task complexity, risk and cost ceiling. |
| Budgets | Per-user, per-tenant, per-feature and per-environment limits. |
| Caching | Reuse repeated context and common prefixes. |
| Compression | Summarize or retrieve only what is needed instead of dumping everything. |
| Reporting | Track model, tokens, media, tools, retries, latency and business outcome. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Put token estimation into the application before calling the model.
- Use cheap models for routing and extraction; escalate only when needed.
- Cap max output tokens and refuse runaway tasks.
- Cache stable long context.
- Create unit economics dashboards before selling AI features.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Users can accidentally send huge files or repeated prompts.
- Agents can create hidden loops and repeated tool calls.
- Retry storms can multiply cost.
- Using the strongest model by default kills margin.
- A feature that feels cheap in tests can become expensive at scale.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Credit packs | Design toolbox AI credits based on measured token workflows. |
| Guide generation | Estimate cost per guide and per modal. |
| MigrateSafe | Charge or limit deep analysis separately from basic scans. |
| Dashboard | Show cost by project, feature and provider. |
| Architecture | Fail closed when budget is exceeded. |
Token Economics and FinOps principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 29
Observability, Logs and AI Incident Response
Observability, Logs and AI Incident Response
AI observability must capture more than HTTP status. You need model ID, prompt version, schema version, token counts, latency, tool calls, retrieval sources, cache usage, refusal flags, user feedback and downstream actions.
Without observability, teams cannot explain cost spikes, quality regressions, unsafe tool calls or hallucination incidents.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Observability, Logs and AI Incident Response into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Request metadata | User, tenant, feature, model, prompt version and data class. |
| Quality metadata | Grounding sources, schema validation, confidence, refusal and human feedback. |
| Tool metadata | Tool name, arguments, result, duration and approval status. |
| Cost metadata | Input tokens, output tokens, cached tokens, media and retries. |
| Incident workflow | Detect, triage, disable, notify, fix, replay evals and document. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create a central AI request table or log stream.
- Redact secrets and avoid storing full sensitive prompts by default.
- Keep trace IDs from UI to model call to tool call.
- Add kill switches for each AI feature.
- Write incident postmortems for hallucination, leak, cost spike and bad action scenarios.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Full prompt logging can become a data leak.
- No tool trace means no accountability for agent actions.
- Feedback without source traces is hard to debug.
- Cost reports without feature labels are useless for product decisions.
- AI incidents need product, engineering, legal and support coordination.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| AI run table | Store provider, model, tokens, feature, status and artifacts. |
| Patch audit | Link every generated patch to prompt, files and test result. |
| Guide audit | Store source URLs and generation timestamp. |
| SRDF assistant | Log recommendations separately from executed operations. |
| Admin UI | Filter AI runs by app, cost, error and model. |
Observability, Logs and AI Incident Response principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 30
Prompt Design, System Instructions and Prompt Ops
Prompt Design, System Instructions and Prompt Ops
Prompt design for Gemini should be a disciplined software process. A prompt is not a magic paragraph; it is a contract between product intent, model behavior, tool permissions, data constraints and output validation.
Prompt Ops means versioning prompts, testing them, measuring regressions, and controlling rollout.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Prompt Design, System Instructions and Prompt Ops into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| System instruction | Role, boundaries, refusal rules, output style and tool policy. |
| Task prompt | User request plus context and acceptance criteria. |
| Few-shot examples | Representative examples where behavior must be stable. |
| Output contract | JSON schema, table, markdown structure or code constraints. |
| Evaluation | Golden prompts, adversarial tests, cost and latency thresholds. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Keep prompts modular: system, task, context, output contract and examples.
- Never hide business rules only in prompt text; validate them in code.
- Version prompts with changelog and owner.
- Run prompt changes through evals before production rollout.
- Use smaller prompts when context quality is already high.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Prompts become unmaintainable when they accumulate hacks.
- System instructions may conflict with developer or user instructions.
- Examples can bias outputs in unwanted ways.
- Prompt-only security is weak against injection.
- No versioning makes regressions impossible to trace.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide prompts | Create reusable templates for cards, modals, source panels and JS checks. |
| Patch prompts | Include full-function replacement rule and no French in code. |
| Migration prompts | Require exact file paths, commands, risks and validation. |
| SRDF prompts | Preserve pipeline constraints and phase semantics. |
| Prompt registry | Store prompt versions in Django models. |
Prompt Design, System Instructions and Prompt Ops principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 31
Vertex AI Media Models: Image, Video and Creative Workflows
Vertex AI Media Models: Image, Video and Creative Workflows
Vertex AI is not limited to text LLMs. Google Cloud includes generative AI capabilities for image and video workflows, and Gemini can participate as planner, evaluator, prompt generator or multimodal analyst.
A professional media workflow separates generation, review, rights, safety, branding and publication.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Vertex AI Media Models: Image, Video and Creative Workflows into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Prompt planning | Use Gemini to create and refine image or video prompts. |
| Generation | Use appropriate image or video generation model where available. |
| Review | Apply brand, safety, legal and quality review before publication. |
| Metadata | Store prompt, model, seed or generation settings where applicable. |
| Reuse | Package generated assets into guides, posts, training or product UI. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use media models for illustrations, explainers, thumbnails and learning assets.
- Keep brand rules and forbidden content explicit.
- Use human review before external publication.
- Store generation provenance.
- Avoid generating misleading technical diagrams without validation.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Generated visuals can be beautiful but technically wrong.
- Rights and attribution rules must be understood for commercial use.
- Deepfake, impersonation or misleading content risks require policy.
- Video generation can be expensive and slow.
- Brand consistency needs prompt and asset governance.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide visuals | Generate chapter illustrations for LLM and database guides. |
| LinkedIn posts | Create branded images for tooling announcements. |
| Training | Create explanatory visuals for migrations, SRDF and databases. |
| Review | Mark AI-generated visuals and validate technical diagrams. |
| Asset library | Store prompts and final image paths in Django. |
Vertex AI Media Models: Image, Video and Creative Workflows principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 32
Vertex AI as an Enterprise AI Control Plane
Vertex AI as an Enterprise AI Control Plane
Vertex AI is broader than Gemini. It is Google Cloud's AI platform for building, deploying and managing ML and generative AI workloads. For enterprise teams, it can act as the AI control plane.
A good control plane reduces chaos: common identity, projects, billing, model registry, endpoint management, evaluation and governance.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Vertex AI as an Enterprise AI Control Plane into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Project structure | Separate projects by environment, data class and product domain. |
| Model registry | Track model IDs, endpoints, owners, lifecycle and approvals. |
| Endpoint strategy | Hosted Gemini, deployed open models, partner models or custom models. |
| Evaluation strategy | Pre-release evals and post-release monitoring. |
| Governance strategy | IAM, policy, logs, cost center, incident response and audit. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Define a cloud project structure before scaling AI usage.
- Create a model and prompt registry.
- Centralize AI spend reporting.
- Use separate service accounts per product feature.
- Review deployed models quarterly for cost, usage and lifecycle.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- AI sprawl can recreate the worst parts of shadow IT.
- Too many unmanaged endpoints create security and cost risk.
- MLOps and GenAI Ops require different but connected practices.
- Ownership must be explicit for every model-backed feature.
- Enterprise controls can slow teams unless good templates exist.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| AI registry | Track every provider, model, prompt, tool and feature. |
| Django services | Centralize AI calls under service modules. |
| Admin cockpit | Display usage, eval status and cost per project. |
| Release process | Require evals and review before enabling new model IDs. |
| Product factory | Use Vertex AI patterns for reusable AI-powered tools. |
Vertex AI as an Enterprise AI Control Plane principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 33
Reference Enterprise Architecture
Reference Enterprise Architecture
A production Gemini / Vertex application is a layered architecture, not one API call. It has UI controls, backend service contracts, prompt registry, retrieval, model routing, tool gateways, structured outputs, evals, logs, budgets and governance.
This blueprint can be adapted to internal copilots, customer support, technical documentation, DevOps assistants and code analysis systems.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Reference Enterprise Architecture into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| UI layer | Chat, forms, file upload, source display, warnings and feedback. |
| Backend layer | Django service, provider adapter, auth, rate limits and request validation. |
| Knowledge layer | Vertex AI Search, embeddings, metadata and source permissions. |
| Model layer | Gemini routing, context caching, tools and structured outputs. |
| Control layer | Evals, logs, budgets, incident response and release gates. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Draw the data path before writing code.
- Define what the model may read, write and execute.
- Build retrieval and tools as separate services.
- Add evaluation before wide rollout.
- Expose source citations and confidence to users.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Skipping architecture creates an expensive chatbot, not a product.
- Tool use without permission design is dangerous.
- RAG without metadata and access control leaks or misleads.
- No budgets means the product can become unprofitable.
- No evals means every model upgrade is a gamble.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Service split | ai_core/services/provider_gemini.py, rag_service.py, eval_service.py, cost_service.py. |
| Admin pages | AI runs, prompt versions, model registry, eval reports and cost dashboards. |
| Artifacts | Generated HTML, PDF, XLSX and code patches stored with provenance. |
| Tools | Read-only diagnostics before write workflows. |
| CI/CD | Release gate checks before production enablement. |
Reference Enterprise Architecture principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 34
IDEO-Lab Gemini / Vertex Playbook
IDEO-Lab Gemini / Vertex Playbook
For IDEO-Lab, Gemini / Vertex can become an industrial AI layer across documentation, code generation, migration diagnostics, SRDF operations, search, glossary extraction and productized utilities.
The correct path is service-based integration: central AI provider service, model registry, prompt registry, evals, logs, budgets and admin visibility.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates IDEO-Lab Gemini / Vertex Playbook into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Django integration | One AI service layer, no scattered SDK calls in views. |
| Guide factory | Convert source packs into IDEO-Lab cards and modals. |
| MigrateSafe | Analyze migration failures, generate patch plans and export reports. |
| SRDF | Summarize runs, detect anomalies and explain orchestration status. |
| Knowledge layer | Vertex AI Search over guides, PDFs, runbooks and code docs. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Create an AIProvider abstraction with Gemini and Vertex routes.
- Store every generated artifact with source, prompt, model and timestamp.
- Start with read-only assistants before code-changing agents.
- Use evals from real historical failures.
- Add costs and token counts to Django admin.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Without a provider abstraction, switching models becomes painful.
- Generated code can break project conventions if not tested.
- Operational assistants must not execute production actions without approval.
- Guide generation can drift factually without source refresh.
- Cost must be visible before selling credits or subscriptions.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Phase 1 | Guide generation and technical summarization. |
| Phase 2 | RAG over IDEO-Lab docs and uploaded PDFs. |
| Phase 3 | MigrateSafe diagnostic assistant with structured reports. |
| Phase 4 | SRDF operations assistant with read-only tools. |
| Phase 5 | Productized AI utilities with license, quotas and billing. |
IDEO-Lab Gemini / Vertex Playbook principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 35
Production Checklist
Production Checklist
A Gemini / Vertex AI feature should not go live because the demo looked impressive. It should go live because data, model, retrieval, tools, output contract, evals, monitoring, security and cost are controlled.
This checklist is the practical release gate for production usage.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Production Checklist into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Data | Approved data classes, retention, redaction and logging. |
| Model | Exact model ID, fallback, lifecycle watch and cost profile. |
| Prompt | Versioned prompt, system instruction and output contract. |
| Tools | Allowlist, validation, approvals and audit. |
| Operations | Evals, monitoring, rate limits, budget, incident process and rollback. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Complete data classification before model integration.
- Pin model IDs and define lifecycle review.
- Create evals and pass regression thresholds.
- Validate structured outputs in application code.
- Enable monitoring, budget alerts and kill switch before launch.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A missing rollback plan turns a model issue into a product outage.
- No source citations makes grounded answers impossible to audit.
- No rate limits exposes the product to cost explosions.
- No model registry hides preview endpoint dependencies.
- No incident owner delays response when something goes wrong.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| GO | Evals pass, cost acceptable, logs enabled, owner assigned, rollback ready. |
| NO GO | Unclear data policy, missing tests, uncontrolled tools, no budget or no fallback. |
| Admin report | Generate a one-page release gate summary. |
| Artifacts | Attach prompt, model, schema and test evidence. |
| Review | Repeat after every model or prompt upgrade. |
Production Checklist principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 36
Future of Gemini / Vertex AI
Future of Gemini / Vertex AI
Gemini / Vertex AI is moving toward a world where models are not isolated chat endpoints. They are multimodal, tool-using, source-grounded, real-time and embedded into enterprise control planes.
The winners will not be teams that call the newest model first. The winners will be teams that build reliable AI systems: evaluated, governed, observable, cost-controlled and useful.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Future of Gemini / Vertex AI into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Model evolution | More capable reasoning, multimodal understanding and efficient variants. |
| Agent evolution | Better tool use, planning, memory, handoffs and enterprise orchestration. |
| Search evolution | Unified retrieval across text, images, video, audio and enterprise systems. |
| Interface evolution | Live voice, video, code, documents and workflow-native assistants. |
| Governance evolution | Model registry, evals, audit, policy and FinOps become mandatory. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Design for model replacement, not model permanence.
- Invest in evals and observability because they survive vendor changes.
- Keep provider adapters and tool contracts clean.
- Use AI for workflows with measurable business value.
- Treat AI as an engineering discipline, not a magic layer.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Hype cycles can cause premature adoption of unstable preview features.
- Vendor lock-in grows if internal abstractions are weak.
- Autonomous agents can outpace governance.
- Multimodal systems expand privacy and compliance exposure.
- Cost and reliability will decide whether AI features survive commercially.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| AI toolbox | Package specialized AI utilities with clear scope and pricing. |
| Private knowledge | Build a long-term IDEO-Lab search and reasoning layer. |
| DevOps AI | Make MigrateSafe and SRDF more autonomous but never uncontrolled. |
| Content engine | Industrialize dense guide generation with sources and checks. |
| Principle | Own the architecture even when the model is rented. |
Future of Gemini / Vertex AI principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 37
Grounding vs RAG vs Long Context
Grounding vs RAG vs Long Context
Many Gemini failures are architecture selection failures. Teams send everything in a giant prompt when they need retrieval, or build RAG when a bounded long-context package would be simpler.
This chapter gives the decision logic for evidence-based Gemini systems.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Grounding vs RAG vs Long Context into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Google Search grounding | Fresh public web facts and citation-style public research. |
| Vertex AI Search RAG | Enterprise corpus with metadata, permissions and source-backed answers. |
| File Search | User-provided or managed files for document-specific Q&A. |
| Long context | One bounded input bundle where full structure matters. |
| Context caching | Repeated long-context analysis over stable inputs. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Ask whether knowledge is public or private.
- Ask whether the corpus is static or changing.
- Ask whether permissions differ by user.
- Ask whether evidence needs page or line references.
- Measure cost and latency for each architecture.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Wrong retrieval architecture can create both poor quality and high cost.
- Public grounding cannot answer confidential internal questions.
- Long context can hide missing evidence.
- RAG without permission filters is dangerous.
- Caching sensitive context can create retention risk.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide updates | Use Search grounding for current public model facts. |
| Internal docs | Use Vertex AI Search over IDEO-Lab corpora. |
| Patch context | Use long context plus caching for one active codebase. |
| User files | Use file-specific Q&A for uploaded docs. |
| Admin rule | Expose which evidence mode was used. |
Grounding vs RAG vs Long Context principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 38
Data Pipeline for Gemini Applications
Data Pipeline for Gemini Applications
Gemini quality depends heavily on the data pipeline before the model call. Enterprise AI often fails because documents are messy, metadata is missing, access rules are unclear or indexes are stale.
A strong ingestion pipeline makes RAG, search, document extraction and analytics far more reliable.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Data Pipeline for Gemini Applications into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Capture | Collect files, URLs, tickets, logs, code, PDFs and media. |
| Normalize | Extract text, layout, metadata, language, owner and version. |
| Validate | Check file type, size, security, completeness and schema. |
| Index | Create chunks, embeddings, keyword index and source map. |
| Refresh | Re-ingest on change and archive stale sources. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Define metadata before ingestion.
- Store original source and normalized representation separately.
- Version extraction and chunking logic.
- Run retrieval evals after indexing changes.
- Attach retention and confidentiality to each object.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Bad metadata breaks filtering and permissions.
- Stale indexes create wrong answers.
- Unvalidated files can poison search quality.
- Large media ingestion can create surprise costs.
- No source map makes audits impossible.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Glossary scanner | Normalize HTML guides into modal-aware sections. |
| PDF ingestion | Extract pages, tables and figures into RAG-ready objects. |
| Code docs | Index Django modules with app and file metadata. |
| Run logs | Ingest SRDF and MigrateSafe logs into searchable incident records. |
| Retention | Archive old sources rather than deleting audit evidence. |
Data Pipeline for Gemini Applications principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 39
Human-in-the-Loop and Approval Design
Human-in-the-Loop and Approval Design
The more useful an AI system becomes, the closer it gets to real business impact. Human-in-the-loop design is how you keep speed without losing accountability.
Approvals should be reserved for meaningful risk, not every harmless answer. The goal is risk-based friction.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Human-in-the-Loop and Approval Design into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Risk classification | Classify tasks by read-only, advisory, write action, destructive action and regulated output. |
| Approval UX | Show proposed action, evidence, diff, cost and rollback. |
| Authority | Only approved roles can approve high-risk actions. |
| Audit | Store who approved what, when, with which model and evidence. |
| Escalation | Send uncertain or rejected actions to expert review. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Use approvals for write actions and production changes.
- Show diffs and source evidence before approval.
- Require role-based approval for destructive operations.
- Allow users to reject, edit or ask for more evidence.
- Track approval metrics and failure outcomes.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Too many approvals cause users to bypass the system.
- Too few approvals create operational risk.
- Approvers need enough context to make a real decision.
- Approval logs can include sensitive data.
- Rubber-stamping is not governance.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Patch approval | Show exact functions changed and test commands before accepting code. |
| Migration approval | Require DBA or senior developer review for DDL plans. |
| SRDF approval | Never execute failover-like actions without human confirmation. |
| Guide publication | Review source freshness and broken JS before publishing. |
| Admin roles | Map approval levels to Django groups. |
Human-in-the-Loop and Approval Design principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 40
Product UX for Limits, Errors and Degraded Mode
Product UX for Limits, Errors and Degraded Mode
AI product UX must handle limits gracefully. Users should understand when a model is thinking, retrieving, using tools, failing, falling back or refusing.
The best systems make uncertainty and limits visible without turning the UI into a technical dump.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Product UX for Limits, Errors and Degraded Mode into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Status layer | Thinking, retrieving, using tools, validating, complete or failed. |
| Fallback layer | Smaller model, cached answer, queued job or human review. |
| Error layer | User-friendly messages with internal trace ID. |
| Budget layer | Explain when a request is too large or over quota. |
| Feedback layer | Let users report wrong, unsupported or unsafe answers. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Show progress for long jobs.
- Use asynchronous jobs for heavy documents.
- Provide retry choices only when safe.
- Surface source and confidence indicators.
- Give users a trace ID for support.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- Raw API errors confuse users.
- Silent fallback can reduce quality without disclosure.
- Overly technical status messages create noise.
- Retry buttons can amplify costs.
- Users lose trust when answers disappear after long waits.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| Guide generation | Show queue status and final download link. |
| MigrateSafe | Return a clear NO GO with trace and next action. |
| SRDF | Separate recommendation from actual system state. |
| Token limits | Ask user to narrow context or run batch mode. |
| Support | Store trace ID in admin logs. |
Product UX for Limits, Errors and Degraded Mode principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate
Chapter 41
Procurement, Terms and Vendor Governance
Procurement, Terms and Vendor Governance
Enterprise AI is also procurement and legal infrastructure. Gemini API, AI Studio and Vertex AI can live under different account, billing and governance models.
Before production, teams should know which terms apply, who pays, who supports incidents and how model lifecycle notices are handled.
- Capability: choose the exact Gemini or Vertex route for the workflow.
- Evidence: prefer grounded answers, citations, source IDs and validation.
- Control: define model ID, tool scope, data class, region and cost limit.
- Operations: observe token use, latency, cache hits, failures and feedback.
- Release: evaluate before production and after every model, prompt or tool change.
Gemini gives the intelligence layer; Vertex AI gives the production control plane. The winning architecture combines both with retrieval, tools, schemas, evals and governance.
| Dimension | Production question | Control |
|---|---|---|
| Model | Which exact model and version? | Registry |
| Data | What can be sent or cached? | Policy |
| Evidence | Which sources support the answer? | Grounding |
| Tools | What can the agent execute? | Allowlist |
| Cost | What is the cost per workflow? | FinOps |
Architecture view
This section translates Procurement, Terms and Vendor Governance into implementation components and control boundaries.
| Component | Engineering meaning |
|---|---|
| Account boundary | Developer API key vs Google Cloud project and billing account. |
| Terms boundary | Applicable service terms, data processing terms and acceptable use. |
| Support boundary | Community, standard cloud support or enterprise account team. |
| Lifecycle boundary | Preview, GA, deprecation and retirement process. |
| Cost boundary | Budget owner, cost center, alerts and chargeback. |
Architecture rule
Never deploy a model call alone. Deploy a system: prompt, data policy, retrieval, tools, schemas, validation, logs, budgets, evals and rollback.
Internal AI service contract: owner: product_or_team surface: gemini_api_or_vertex_ai model_id: pinned_model_or_registry_key data_class: public_internal_confidential_regulated context_mode: direct_rag_search_cache_file output_contract: text_json_schema_markdown_code allowed_tools: read_only_or_approved_write validation: schema_tests_human_review operations: logs_cost_limits_eval_rollback
Operating playbook
Use this practical sequence to move from a demo to a repeatable Gemini / Vertex AI workflow.
- Keep a vendor governance record for each AI feature.
- Review terms before sending customer or regulated data.
- Use enterprise billing and support for critical workloads.
- Track release notes and lifecycle announcements.
- Define who can enable new models or tools.
| Phase | Action | Evidence |
|---|---|---|
| Prototype | Small workflow with safe data | Manual examples |
| Pilot | Limited users and logging | Failure log |
| Evaluation | Golden dataset and regression | Eval report |
| Production | Budgets, monitoring and fallback | Dashboard |
| Upgrade | Re-run evals before model change | Comparison |
Release gate checklist: 1. Workflow and owner defined. 2. Data class approved. 3. Model route selected and pinned. 4. Prompt and schema versioned. 5. Retrieval or grounding tested. 6. Tool scopes validated. 7. Evals passed. 8. Cost limit configured. 9. Logging enabled. 10. Rollback documented.
Risks and controls
Every Gemini / Vertex capability transfers new responsibility to the application owner. The right control depends on data sensitivity, tool autonomy and business impact.
- A prototype key can become a hidden production dependency.
- Preview features may not meet procurement requirements.
- Costs without owner become organizational conflict.
- Legal review after launch is too late.
- Vendor changes require internal communication.
| Risk | Control | Proof |
|---|---|---|
| Unsupported answer | Grounding and source display | Source trace |
| Bad tool call | Schema validation and approvals | Tool log |
| Cost spike | Budgets, caching and routing | FinOps report |
| Data leak | Data policy and redaction | Security review |
| Regression | Versioned evals | Eval diff |
Production rule: the more autonomy, context, media or tool access you give Gemini, the more you need deterministic validation, logs and approval boundaries.
IDEO-Lab application
Concrete ways this chapter can become useful inside IDEO-Lab, Django tooling, MigrateSafe, SRDF, technical guides and productized AI utilities.
| IDEO-Lab asset | Gemini / Vertex role |
|---|---|
| License center | Extend IDEO-Lab license_center concepts to AI provider governance. |
| Admin registry | Track model, terms, owner, data class and cost center. |
| Release notes | Schedule review of Google AI and Vertex AI updates. |
| Support | Define incident contact path for AI features. |
| Commercial products | Map AI cost to subscription or credit pricing. |
Procurement, Terms and Vendor Governance principle: build with Gemini where multimodal intelligence, long context, grounding or tool use creates measurable value, and use Vertex AI where enterprise control, evaluation, governance and scale matter.
IDEO-Lab integration pattern: Django view -> AI service facade -> provider adapter -> model registry -> prompt registry -> retrieval/tools -> validation -> artifact store -> admin logs -> eval report -> release gate