gitea-heatmap-backfill/README.md

# gitea-heatmap-backfill

Retroactively populate the Gitea activity heatmap from existing Git history.

When you migrate repositories into Gitea, past commits do not appear in the
contribution heatmap because no `action` records exist for them. This script
reads the commit history directly from the bare repositories on disk and
inserts the missing records into the Gitea SQLite database.

---

## Tested on

| Component | Version |
|---|---|
| Gitea | **1.26.2** |
| Database | SQLite 3 |

Other Gitea versions using SQLite may work but have not been tested.
PostgreSQL and MySQL backends are not supported.

---

## How it works

1. Discovers all bare repositories under the Gitea repository root
2. Runs `git log --all` on each one to extract commit SHAs, author emails and dates
3. Resolves each email against the `email_address` table, primary address **and** all registered alternative addresses are matched
4. Inserts one `action` row per commit (op\_type = 5, PUSH\_EVENT) into the Gitea SQLite database
5. Deduplicates via a SHA-256 content hash so the script is safe to run multiple times

---

## Requirements

- Python 3.7+
- `git` available on the machine running the script
- Read/write access to the Gitea SQLite database file
- Gitea using SQLite as its database backend

---

## Setup

### 1. Clone the repository

```bash
git clone https://github.com/Maethik/gitea-heatmap-backfill.git
cd gitea-heatmap-backfill
```

No external dependencies, only the Python standard library is used.

### 2. Register all your commit emails in Gitea

The script matches commits by email. Open **Settings → Email Addresses** in
your Gitea profile and add every email address you have ever used in your
commits (work addresses, old personal addresses, GitHub no-reply addresses,
etc.).

You can check which emails appear in your history:

```bash
# Replace the path with your actual repository root
for repo in /path/to/gitea/repositories/owner/*.git; do
  git -C "$repo" log --all --format="%ae"
done | sort | uniq
```

### 3. Configure paths

The script is configured via two environment variables or by editing the
constants at the top of the file.

| Variable | Default | Description |
|---|---|---|
| `DB_PATH` | `/etc/komodo/stacks/gitea/data/gitea/gitea.db` | Absolute path to `gitea.db` on the host |
| `GITEA_ROOT` | `/etc/komodo/stacks/gitea/data/git/repositories` | Absolute path to the Gitea repository root on the host |

To find the correct paths for your setup:

```bash
# Find the repository root declared in app.ini
docker exec gitea cat /data/gitea/conf/app.ini | grep -E "^ROOT\s*="

# Find the host path mounted into the container
docker inspect gitea | grep -A 10 '"Mounts"'
```

---

## Usage

> **The script runs on the Docker host, not inside the container.**
> Python is not available inside the minimal Gitea image.

```bash
# Using environment variables
DB_PATH=/path/to/gitea.db \
GITEA_ROOT=/path/to/repositories \
python3 gitea_backfill.py
```

```bash
# Or edit the constants directly at the top of the script, then run
python3 gitea_backfill.py
```

### Fix git safe.directory errors

If git refuses to read the repositories due to ownership mismatch, run
this once before executing the script:

```bash
git config --global --add safe.directory '*'
```

### Expected output

```
Gitea Heatmap Backfill
============================================
[1] Creating backup → /path/to/gitea.backup-before-backfill.db
    [✓] Backup created.

[✓] 5 email(s) loaded from email_address table
[✓] 8 repo(s) loaded from repository table
[✓] 8 bare repo(s) found on disk

[2] Existing PUSH_EVENT rows: 0

[→] owner/my-project (id=3)
    847 commits found.
    [500 inserted so far]
[→] owner/another-repo (id=4)
    312 commits found.
...
────────────────────────────────────────────
  Backfill complete
────────────────────────────────────────────
  Inserted  : 1159
  Skipped   : 0       (already present or bad date)
  No user   : 4       (email not registered in Gitea)
  No repo   : 0       (repo not found in database)
────────────────────────────────────────────

Restart Gitea to refresh the heatmap:
  docker restart gitea
```

### After the script completes

Restart Gitea so the heatmap is recomputed from the new records:

```bash
docker restart gitea
```

---

## Troubleshooting

**`No user` count is high**

Some commit emails are not registered in Gitea. Run the email audit command
from the Setup section above, then add the missing addresses in your Gitea
profile settings.

**`No repo` count is non-zero**

The repository name on disk does not match the `owner_name/name` stored in
the database. Check for casing differences:

```bash
# Names in the database
docker exec gitea sqlite3 /data/gitea/gitea.db \
  "SELECT owner_name, name FROM repository;"

# Names on disk
ls /path/to/gitea/repositories/owner/
```

**`git log` fails with `dubious ownership`**

```bash
git config --global --add safe.directory '*'
```

**Running the script a second time**

Safe. Already-inserted rows are detected via their content hash and skipped.

---

## Rollback

A backup of the database is created automatically before any writes at:

```
/path/to/gitea.backup-before-backfill.db
```

To restore it:

```bash
docker stop gitea
cp /path/to/gitea.backup-before-backfill.db /path/to/gitea.db
docker start gitea
```

---

## Limitations

- SQLite only — not compatible with Gitea instances using PostgreSQL or MySQL
- Commit emails must be registered in Gitea; unmatched emails are skipped
- Wiki repositories (`*.wiki.git`) are intentionally ignored