Camera & Vision

Open Genie can monitor RTSP, HTTP, and MJPEG camera streams with periodic AI-powered scene analysis. Configurable alert rules trigger notifications when specific conditions are detected.

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Camera 1   │     │   Camera 2   │     │   Camera N   │
│   (RTSP)     │     │   (HTTP)     │     │   (MJPEG)    │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       └──────────────────┬─────────────────────┘
                          ↓
                ┌─────────────────┐
                │  Camera Worker   │  Periodic capture loop
                │  (lib/camera/    │  Staggered intervals
                │   worker.ts)     │
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │  FFmpeg Capture  │  Extract single frame
                │  (capture.ts)    │  Save to storage
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │  Ollama Vision   │  Analyze image with
                │  (vision.ts)     │  custom prompt
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │  Alert Rules     │  Evaluate conditions
                │  (rules.ts)      │  against parsed data
                └────────┬────────┘
                    ┌────┴────┐
                    ↓         ↓
             ┌──────────┐ ┌──────────────┐
             │  Store    │ │ Notification  │
             │  Event    │ │ (if triggered)│
             └──────────┘ └──────────────┘

Camera Configuration

Cameras are stored in the cameraConfigs database table and managed via API.

{
  "name": "Backyard Camera",
  "url": "rtsp://192.168.1.100:554/stream",
  "type": "rtsp",
  "captureIntervalSec": 60,
  "visionPrompt": "Describe the scene. Note any people, animals, or vehicles. Is there any motion?",
  "alertRules": [
    {
      "id": "person-detect",
      "field": "people_count",
      "operator": "greater_than",
      "value": 0,
      "alertLevel": "warning",
      "message": "Person detected: {people_count} people in view",
      "cooldownMinutes": 15,
      "notifyDevices": "phones"
    }
  ],
  "enabled": true
}

Stream Types

Type	Description
rtsp	RTSP video stream (e.g., IP cameras)
http	HTTP video stream or snapshot URL
mjpeg	Motion JPEG stream

Camera Worker (lib/camera/worker.ts)

The worker runs a continuous loop that processes each enabled camera at its configured interval.

Features

• Staggered capture — Cameras are processed with staggered timing to avoid overwhelming the system
• Overlap prevention — Tracks per-camera processing state to prevent concurrent captures
• Failure tolerance — After 10 consecutive failures, a camera is temporarily disabled
• Alert cooldowns — Each rule has a per-camera cooldown to prevent notification spam

Lifecycle

// Start the worker (called at boot)
cameraWorker.start();

// Stop the worker (called at shutdown)
cameraWorker.stop();

Capture Process (lib/camera/capture.ts)

Uses FFmpeg to extract a single frame from the camera stream:

1. Spawn ffmpeg with the camera URL
2. Extract one frame as JPEG
3. Save to data/files/camera-captures/{cameraId}/{timestamp}.jpg
4. Return the file path and base64-encoded image data

Vision Analysis (lib/camera/vision.ts)

Sends the captured frame to Ollama's vision model:

1. Encode image as base64
2. Send to ollama.vision() with the camera's custom visionPrompt
3. Parse the JSON response from the model
4. Return structured analysis data

The vision prompt should instruct the model to respond in JSON format for reliable parsing.

Alert Rules (lib/camera/rules.ts)

Alert rules evaluate conditions against the parsed vision analysis.

Rule Structure

interface AlertRule {
  id: string;                    // Unique rule identifier
  field: string;                 // Dot-notation path (e.g., "motion.detected")
  operator: AlertOperator;       // Comparison operator
  value: string | number | boolean; // Expected value
  alertLevel: "info" | "warning" | "critical";
  message: string;               // Template with {field} interpolation
  cooldownMinutes: number;        // Minutes between repeat alerts
  notifyDevices: "all" | "phones" | "tablets" | string[];
}

Operators

Operator	Description	Example
equals	Exact match	field == value
not_equals	Not equal	field != value
contains	String contains	"car" in field
greater_than	Numeric comparison	field > value
less_than	Numeric comparison	field < value
is_truthy	Boolean check	field is truthy

Dot-Notation Fields

Rules support nested field access:

{
  "field": "motion.detected",
  "operator": "equals",
  "value": true
}

This accesses analysisResult.motion.detected.

Message Templates

Alert messages support {field} interpolation:

{
  "message": "Alert: {people_count} people detected in {location}"
}

Cleanup (lib/camera/cleanup.ts)

Periodic cleanup enforces retention policies:

• Delete captures older than 30 days (configurable)
• Enforce a maximum of 1000 records per camera
• Clean test captures older than 24 hours

REST API

Method	Path	Description
GET	/api/cameras	List all cameras
POST	/api/cameras	Create a camera
GET	/api/cameras/[id]	Get camera details
PUT	/api/cameras/[id]	Update camera config
DELETE	/api/cameras/[id]	Delete a camera
POST	/api/cameras/[id]/test	Test capture + analysis
GET	/api/cameras/[id]/events	List events (filterable by ?level=)
GET	/api/cameras/[id]/latest	Get latest capture

Key Files

File	Purpose
lib/camera/worker.ts	Capture loop orchestration
lib/camera/capture.ts	FFmpeg frame extraction
lib/camera/vision.ts	Ollama vision analysis
lib/camera/rules.ts	Alert rule evaluation
lib/camera/cleanup.ts	Retention policy enforcement