Google Gemma 2: Running and Fine-Tuning Open AI Models Locally with Ollama and Python
A hands-on guide to Google's open-weights Gemma 2 models (2B, 9B, 27B). Learn local inference with Ollama, 4-bit quantization, and LoRA fine-tuning with Hugging Face.
- 1.Google's Open Weights Revolution: Gemma 2
- 2.1. Running Gemma 2 Locally with Ollama in 60 Seconds
- 3.2. Calling Local Gemma 2 from Node.js or Python
- 4.3. Gemma 2 Model Sizes & Hardware Requirements
- 5.4. Fine-Tuning Gemma 2 with LoRA in Python (PEFT)
- 6.5. Summary Key Takeaways
- 7.6. Frequently Asked Questions (FAQ)
Google's Open Weights Revolution: Gemma 2#
While Gemini models power Google's cloud APIs, Google also releases state-of-the-art open-weights models under the Gemma family. Built from the same research, dataset filtering, and architectural technology used to create Gemini, Gemma 2 delivers class-leading reasoning and coding performance in compact model sizes: 2B, 9B, and 27B parameters.
Because Gemma 2 models are open-weights, you can run them completely offline on your own local laptop or private GPU servers with zero API costs, zero vendor lock-in, and 100% data privacy.
1. Running Gemma 2 Locally with Ollama in 60 Seconds#
[Ollama](https://ollama.com) is the easiest way to run open LLMs locally with automated hardware acceleration (Apple Silicon Metal & NVIDIA CUDA):
# 1. Pull and run Gemma 2 9B model (requires ~6GB RAM)
ollama run gemma2:9b
# 2. Or run the ultra-lightweight Gemma 2 2B model on low-end hardware
ollama run gemma2:2b2. Calling Local Gemma 2 from Node.js or Python#
Ollama exposes a standard OpenAI-compatible REST API at http://localhost:11434:
import OpenAI from "openai";
// Connect to local Ollama instance
const localAI = new OpenAI({
baseURL: "http://localhost:11434/v1",
apiKey: "ollama", // Required by SDK but unused locally
});
export async function summarizeTextLocally(text: string) {
const response = await localAI.chat.completions.create({
model: "gemma2:9b",
messages: [
{ role: "system", content: "You are a concise technical summarizer." },
{ role: "user", content: text },
],
temperature: 0.2,
});
return response.choices[0].message.content;
}3. Gemma 2 Model Sizes & Hardware Requirements#
| Model Size | Parameters | Minimum RAM (4-bit Quantized) | Recommended Hardware | Ideal Use Case |
|---|---|---|---|---|
| Gemma 2 2B | 2.6 Billion | 2.5 GB RAM | Any Laptop, Raspberry Pi 5, iPhone | On-device classification, edge devices |
| Gemma 2 9B | 9.2 Billion | 6.0 GB RAM | Apple M1/M2/M3 (16GB), NVIDIA RTX 3060 | Code generation, general reasoning, summarization |
| Gemma 2 27B | 27.2 Billion | 18.0 GB RAM | Apple M-Max (36GB+), NVIDIA RTX 4090 / A100 | Near GPT-4 level complex analytical reasoning |
4. Fine-Tuning Gemma 2 with LoRA in Python (PEFT)#
When you need Gemma 2 to follow specific enterprise formatting or domain terminology, use Low-Rank Adaptation (LoRA) to fine-tune model weights efficiently:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
model_id = "google/gemma-2-9b-it"
# 1. Load tokenizer and 4-bit quantized base model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
# 2. Configure Parameter-Efficient Fine-Tuning (LoRA)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
peft_model = get_peft_model(model, lora_config)
print("Trainable parameters:", peft_model.print_trainable_parameters())5. Summary Key Takeaways#
- Gemma 2 9B Punches Above Its Weight: In independent benchmarks, Gemma 2 9B matches or outperforms models twice its size.
- Total Privacy & Offline Execution: Sensitive medical, legal, or proprietary codebases can be processed entirely offline without sending a single byte over the internet.
- Open Commercial License: Google licenses Gemma under permissive terms that allow free commercial deployment in applications.
6. Frequently Asked Questions (FAQ)#
Q: Can Gemma 2 run on Apple Silicon Macs with unified memory? Yes, exceptionally well. Using Ollama or MLX (Apple's machine learning framework for silicon), Gemma 2 9B runs at over 35 tokens/second on an M2/M3 Pro Mac with 16GB of unified memory.
Q: What is the difference between Gemma Base and Gemma Instruct? **Gemma Base** models are pre-trained on raw internet text for continuation and completions. **Gemma Instruct (-it)** models have undergone Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to follow structured conversational prompts and markdown output instructions. For 99% of applications, always use the instruct version!
Published by
Vyuhantrix Team
Open Source AI & ML · Vyuhantrix
Vyuhantrix is an open technology learning platform based in Ahmedabad, India, publishing step-by-step programming tutorials, system design breakdowns, and free developer tools.
Keep Learning
Recommended Guides
Top 5 Programming Languages to Learn in 2026 for High-Impact Careers
Discover the most in-demand languages driving cloud infrastructure, AI development, web platforms, systems engineering, and enterprise backend systems.
Python Data Engineering: Building Production Data Pipelines in 2026
A practical guide to building production data pipelines with Python — covering Pandas, Polars, Apache Airflow for orchestration, dbt for transformations, data validation with Great Expectations, and deployment patterns.
Python vs. JavaScript: Detailed Comparison for Beginners & Career Starters
An honest, in-depth comparison of Python and JavaScript in 2026 — comparing syntax, web engineering, AI capabilities, job markets, and learning curves.