backfill Gitea heatmap from Git history

README.md

@@ -0,0 +1,217 @@
+# gitea-heatmap-backfill
+
+Retroactively populate the Gitea activity heatmap from existing Git history.
+
+When you migrate repositories into Gitea, past commits do not appear in the
+contribution heatmap because no `action` records exist for them. This script
+reads the commit history directly from the bare repositories on disk and
+inserts the missing records into the Gitea SQLite database.
+
+---
+
+## Tested on
+
+| Component | Version |
+|---|---|
+| Gitea | **1.26.2** |
+| Database | SQLite 3 |
+
+Other Gitea versions using SQLite may work but have not been tested.
+PostgreSQL and MySQL backends are not supported.
+
+---
+
+## How it works
+
+1. Discovers all bare repositories under the Gitea repository root
+2. Runs `git log --all` on each one to extract commit SHAs, author emails and dates
+3. Resolves each email against the `email_address` table, primary address **and** all registered alternative addresses are matched
+4. Inserts one `action` row per commit (op\_type = 5, PUSH\_EVENT) into the Gitea SQLite database
+5. Deduplicates via a SHA-256 content hash so the script is safe to run multiple times
+
+---
+
+## Requirements
+
+- Python 3.7+
+- `git` available on the machine running the script
+- Read/write access to the Gitea SQLite database file
+- Gitea using SQLite as its database backend
+
+---
+
+## Setup
+
+### 1. Clone the repository
+
+```bash
+git clone https://github.com/Maethik/gitea-heatmap-backfill.git
+cd gitea-heatmap-backfill
+```
+
+No external dependencies, only the Python standard library is used.
+
+### 2. Register all your commit emails in Gitea
+
+The script matches commits by email. Open **Settings → Email Addresses** in
+your Gitea profile and add every email address you have ever used in your
+commits (work addresses, old personal addresses, GitHub no-reply addresses,
+etc.).
+
+You can check which emails appear in your history:
+
+```bash
+# Replace the path with your actual repository root
+for repo in /path/to/gitea/repositories/owner/*.git; do
+  git -C "$repo" log --all --format="%ae"
+done | sort | uniq
+```
+
+### 3. Configure paths
+
+The script is configured via two environment variables or by editing the
+constants at the top of the file.
+
+| Variable | Default | Description |
+|---|---|---|
+| `DB_PATH` | `/etc/komodo/stacks/gitea/data/gitea/gitea.db` | Absolute path to `gitea.db` on the host |
+| `GITEA_ROOT` | `/etc/komodo/stacks/gitea/data/git/repositories` | Absolute path to the Gitea repository root on the host |
+
+To find the correct paths for your setup:
+
+```bash
+# Find the repository root declared in app.ini
+docker exec gitea cat /data/gitea/conf/app.ini | grep -E "^ROOT\s*="
+
+# Find the host path mounted into the container
+docker inspect gitea | grep -A 10 '"Mounts"'
+```
+
+---
+
+## Usage
+
+> **The script runs on the Docker host, not inside the container.**
+> Python is not available inside the minimal Gitea image.
+
+```bash
+# Using environment variables
+DB_PATH=/path/to/gitea.db \
+GITEA_ROOT=/path/to/repositories \
+python3 gitea_backfill.py
+```
+
+```bash
+# Or edit the constants directly at the top of the script, then run
+python3 gitea_backfill.py
+```
+
+### Fix git safe.directory errors
+
+If git refuses to read the repositories due to ownership mismatch, run
+this once before executing the script:
+
+```bash
+git config --global --add safe.directory '*'
+```
+
+### Expected output
+
+```
+Gitea Heatmap Backfill
+============================================
+[1] Creating backup → /path/to/gitea.backup-before-backfill.db
+    [✓] Backup created.
+
+[✓] 5 email(s) loaded from email_address table
+[✓] 8 repo(s) loaded from repository table
+[✓] 8 bare repo(s) found on disk
+
+[2] Existing PUSH_EVENT rows: 0
+
+[→] owner/my-project (id=3)
+    847 commits found.
+    [500 inserted so far]
+[→] owner/another-repo (id=4)
+    312 commits found.
+...
+────────────────────────────────────────────
+  Backfill complete
+────────────────────────────────────────────
+  Inserted  : 1159
+  Skipped   : 0       (already present or bad date)
+  No user   : 4       (email not registered in Gitea)
+  No repo   : 0       (repo not found in database)
+────────────────────────────────────────────
+
+Restart Gitea to refresh the heatmap:
+  docker restart gitea
+```
+
+### After the script completes
+
+Restart Gitea so the heatmap is recomputed from the new records:
+
+```bash
+docker restart gitea
+```
+
+---
+
+## Troubleshooting
+
+**`No user` count is high**
+
+Some commit emails are not registered in Gitea. Run the email audit command
+from the Setup section above, then add the missing addresses in your Gitea
+profile settings.
+
+**`No repo` count is non-zero**
+
+The repository name on disk does not match the `owner_name/name` stored in
+the database. Check for casing differences:
+
+```bash
+# Names in the database
+docker exec gitea sqlite3 /data/gitea/gitea.db \
+  "SELECT owner_name, name FROM repository;"
+
+# Names on disk
+ls /path/to/gitea/repositories/owner/
+```
+
+**`git log` fails with `dubious ownership`**
+
+```bash
+git config --global --add safe.directory '*'
+```
+
+**Running the script a second time**
+
+Safe. Already-inserted rows are detected via their content hash and skipped.
+
+---
+
+## Rollback
+
+A backup of the database is created automatically before any writes at:
+
+```
+/path/to/gitea.backup-before-backfill.db
+```
+
+To restore it:
+
+```bash
+docker stop gitea
+cp /path/to/gitea.backup-before-backfill.db /path/to/gitea.db
+docker start gitea
+```
+
+---
+
+## Limitations
+
+- SQLite only — not compatible with Gitea instances using PostgreSQL or MySQL
+- Commit emails must be registered in Gitea; unmatched emails are skipped
+- Wiki repositories (`*.wiki.git`) are intentionally ignored