Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/.auxesia.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title: objgit
subtitle: A git server that stores repositories in Tigris
4 changes: 4 additions & 0 deletions docs/architecture/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Architecture"
---

# Architecture

`objgitd` is one binary. It stores git repositories as objects in a Tigris
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/auth.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "The auth seam (`internal/auth`)"
---

# The auth seam (`internal/auth`)

Every transport authorizes through one interface. A real authentication and
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/hooks.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Push hooks (`hooks.go`)"
---

# Push hooks (`hooks.go`)

When `-allow-hooks` is set, a successful `receive-pack` runs the repository
Expand Down
32 changes: 18 additions & 14 deletions docs/architecture/lfs.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Git LFS (`internal/lfs`)"
---

# Git LFS (`internal/lfs`)

`objgitd` answers the Git LFS batch API with presigned Tigris URLs. The client
Expand All @@ -16,12 +20,12 @@ route, so every LFS path answers 404.

## Layout in the bucket

| Key | Contents |
| --- | --- |
| `lfs/blobs/<oid>` | The object bytes, stored once for the whole bucket |
| `<orgID>/<repo>/lfs/staging/<oid>` | An upload in flight, before it is verified |
| Key | Contents |
| ---------------------------------- | ----------------------------------------------------- |
| `lfs/blobs/<oid>` | The object bytes, stored once for the whole bucket |
| `<orgID>/<repo>/lfs/staging/<oid>` | An upload in flight, before it is verified |
| `<orgID>/<repo>/lfs/objects/<oid>` | A zero-byte marker: this repository holds that object |
| `<orgID>/<repo>/lfs/locks.json` | Every lock held in this repository |
| `<orgID>/<repo>/lfs/locks.json` | Every lock held in this repository |

`<oid>` is the SHA-256 of the object, in lowercase hexadecimal.

Expand Down Expand Up @@ -92,7 +96,7 @@ the query string. `X-Amz-Checksum-Sha256` is one of those, and a hoisted
checksum is a checksum nobody sends and nothing enforces. `noHoistPresigner`
turns hoisting off for uploads. `TestPresignPutSignsTheChecksumHeader` pins it.

**At verify time**, `Verify` heads the *staged* object with the checksum mode
**At verify time**, `Verify` heads the _staged_ object with the checksum mode
enabled and compares both the length and the digest. This is the only check that
runs on the daemon's own credentials. It is also the only check on size, because
`Content-Length` is not signed on a presigned PUT and cannot be enforced there.
Expand All @@ -113,13 +117,13 @@ client sends that header back. A download URL has to stand on its own, so only

## Batch outcomes

| Operation | Marker | Response for that object |
| --- | --- | --- |
| `download` | present | `actions.download`, a presigned GET |
| `download` | absent | `error`, code 404 |
| `upload` | present, size matches | no `actions` key, so the client skips it |
| `upload` | present, size differs | `error`, code 422 |
| `upload` | absent | `actions.upload` and `actions.verify` |
| Operation | Marker | Response for that object |
| ---------- | --------------------- | ---------------------------------------- |
| `download` | present | `actions.download`, a presigned GET |
| `download` | absent | `error`, code 404 |
| `upload` | present, size matches | no `actions` key, so the client skips it |
| `upload` | present, size differs | `error`, code 422 |
| `upload` | absent | `actions.upload` and `actions.verify` |

Every object carries `authenticated: true`. Without it git-lfs runs credential
discovery against the storage host and can attach an `Authorization` header,
Expand Down Expand Up @@ -205,7 +209,7 @@ trusting a change to `presign.go` or `store.go`.

- Nothing deletes a blob. The marker gives a collector its reference count, but
no collector exists.
- `-lfs-max-size` caps the size a client *declares*, not the size it uploads.
- `-lfs-max-size` caps the size a client _declares_, not the size it uploads.
Content-Length is not signed on a presigned PUT, so the real limit at upload
time is what the bucket accepts. `Verify` catches the difference, but a client
that never verifies leaves the staged object behind. Staged objects under
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/metrics.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Metrics (`internal/metrics`)"
---

# Metrics (`internal/metrics`)

`internal/metrics` defines every Prometheus vector with `promauto`, against
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/s3fs.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "`internal/s3fs` — a billy.Filesystem on Tigris"
---

# `internal/s3fs` — a billy.Filesystem on Tigris

This package is vendored from the s3fs of Austin Poor. It is adapted to
Expand Down
8 changes: 6 additions & 2 deletions docs/architecture/snapshots.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "erofs snapshots (`internal/snapshot`)"
---

# erofs snapshots (`internal/snapshot`)

After a push, `objgitd` can build one EROFS image of the tree at the tip of
Expand All @@ -14,8 +18,8 @@ The design is in

## Object layout

| Key | Contents |
| --------------------------------- | -------------------------------------------------------------------- |
| Key | Contents |
| --------------------------------- | --------------------------------------------------------------------- |
| `snapshots/erofs/v2/<tree>.erofs` | One EROFS image of the git tree `<tree>`, with LFS pointers resolved. |

The key uses the tree hash, not the commit hash. Many commits can point at one
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/tigris-storer.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "`internal/storage/tigris` — a go-git `storage.Storer` on one Tigris bucket"
---

# `internal/storage/tigris` — a go-git `storage.Storer` on one Tigris bucket

This package implements `storage.Storer` for Tigris. It speaks
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/transports.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Transports"
---

# Transports

objgitd speaks three git transports. All three answer the protocol natively
Expand Down
84 changes: 44 additions & 40 deletions docs/plans/bin-cue-benchmark.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Benchmark objgit before and after the bin/cue format change"
---

# Benchmark objgit before and after the bin/cue format change

Status: Implemented as `cmd/formatbench`. See
Expand Down Expand Up @@ -28,10 +32,10 @@ plus `packs/<id>.cue` containers through `internal/storage/tigris`.

## What "before" and "after" mean

| Column | Ref | What it runs |
| --- | --- | --- |
| before | `v1.0.2` | `internal/s3fs` + go-git dotgit. One bucket. Listing cache and pack cache on. |
| after | `main` (`ce5a7ca`, v1.5.0) | `internal/storage/tigris`. bin/cue containers, zstd, cue v3. |
| Column | Ref | What it runs |
| ------ | -------------------------- | ----------------------------------------------------------------------------- |
| before | `v1.0.2` | `internal/s3fs` + go-git dotgit. One bucket. Listing cache and pack cache on. |
| after | `main` (`ce5a7ca`, v1.5.0) | `internal/storage/tigris`. bin/cue containers, zstd, cue v3. |

`v1.0.2` is the last release where repositories live in the single `-bucket`
flag, which is also how HEAD works. That keeps the setup identical on both
Expand All @@ -55,10 +59,10 @@ One thing about this pair goes in the post as a caveat, not as a hidden fact:

Mirror-clone each repository once. Never write to the mirror.

| Name | URL | Shape |
| --- | --- | --- |
| objgit | https://github.com/tigrisdata/objgit | Small. Mostly text. |
| x | https://github.com/Xe/x | Medium. Large object count. |
| Name | URL | Shape |
| ----------- | ----------------------------------------- | ----------------------------------- |
| objgit | https://github.com/tigrisdata/objgit | Small. Mostly text. |
| x | https://github.com/Xe/x | Medium. Large object count. |
| tigris-blog | https://github.com/tigrisdata/tigris-blog | Blob-heavy. Images do not compress. |

Confirm the `tigris-blog` clone URL before the first run. Record the resolved
Expand All @@ -77,14 +81,14 @@ at `cmd/objgitd/main.go:81`. HEAD feeds it from `Storer.observe`

Per cell, that is per build and per repository and per workload:

| Field | Source |
| --- | --- |
| Wall seconds | Driver timer around the `git` subprocess. |
| S3 requests by verb | Delta of `objgit_s3_requests_total{operation}`. |
| S3 time | Delta of `objgit_s3_request_duration_seconds_sum`. |
| Keys written | `ListObjectsV2` over the repository prefix after the push. |
| Bytes stored | Sum of the sizes from that same listing. |
| Bytes on the wire | Clone only. The `Receiving objects` line from git stderr. |
| Field | Source |
| ------------------- | ---------------------------------------------------------- |
| Wall seconds | Driver timer around the `git` subprocess. |
| S3 requests by verb | Delta of `objgit_s3_requests_total{operation}`. |
| S3 time | Delta of `objgit_s3_request_duration_seconds_sum`. |
| Keys written | `ListObjectsV2` over the repository prefix after the push. |
| Bytes stored | Sum of the sizes from that same listing. |
| Bytes on the wire | Clone only. The `Receiving objects` line from git stderr. |

Both eras key a repository under its `orgID/name` path. `v1.0.2` gets there
through the go-git loader (`cmd/objgitd/git_protocol.go:61`). HEAD gets there
Expand Down Expand Up @@ -132,7 +136,7 @@ Reuse these patterns from membench rather than inventing new ones:
2. Make a fresh empty directory for `-pack-cache-dir`.
3. Start the daemon and wait for it to answer. Flags for both builds:
`-bucket $BUCKET -http-bind 127.0.0.1:8080 -metrics-bind 127.0.0.1:9090
-ssh-bind= -allow-push -pack-cache-dir <tmp>`. Leave every other flag at
-ssh-bind= -allow-push -pack-cache-dir <tmp>`. Leave every other flag at
its default, so each build runs the way it shipped.
4. Scrape `/metrics` and record the counters.
5. Run `git push --mirror http://127.0.0.1:8080/bench/<repo>.git` from the
Expand Down Expand Up @@ -197,32 +201,32 @@ Two files in the run directory:
Push:

| repo | before wall | after wall | before PUTs | after PUTs | before keys | after keys | before bytes | after bytes |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ---- | ----------- | ---------- | ----------- | ---------- | ----------- | ---------- | ------------ | ----------- |

Clone:

| repo | before wall | after wall | before GETs | after GETs | wire bytes |
| --- | --- | --- | --- | --- | --- |
| ---- | ----------- | ---------- | ----------- | ---------- | ---------- |

End the section with this line, as written:

> Oh yeah, this was with my corporate laptop on wifi.

## Files

| Path | Change |
| --- | --- |
| `cmd/formatbench/main.go` | New. The driver: setup, cell loop, interleaving, timeouts. |
| `cmd/formatbench/metrics.go` | New. Scrape and diff `objgit_s3_requests_total` by label. |
| `cmd/formatbench/bucket.go` | New. List a prefix, sum key count and bytes. |
| `cmd/formatbench/git.go` | New. Timed git subprocesses and the wire-byte parser. |
| `cmd/formatbench/verify.go` | New. The correctness gate. |
| `cmd/formatbench/report.go` | New. `results.json` and `results.md`. |
| `cmd/formatbench/uuid.go` | New. Collision-free repository names. |
| `docs/plans/bin-cue-benchmark.md` | New. This plan, committed where `AGENTS.md` says plans live. |
| `docs/usage/format-benchmark.md` | New. How to run it. Follow `docs/usage/memory-benchmark.md` in shape. |
| `AGENTS.md` | One row in the "Where the code lives" table, next to `cmd/membench/`. |
| `var/bin-cue-git.md` | Replace the `TODO(Xe)` at line 66 with the finished tables. |
| Path | Change |
| --------------------------------- | --------------------------------------------------------------------- |
| `cmd/formatbench/main.go` | New. The driver: setup, cell loop, interleaving, timeouts. |
| `cmd/formatbench/metrics.go` | New. Scrape and diff `objgit_s3_requests_total` by label. |
| `cmd/formatbench/bucket.go` | New. List a prefix, sum key count and bytes. |
| `cmd/formatbench/git.go` | New. Timed git subprocesses and the wire-byte parser. |
| `cmd/formatbench/verify.go` | New. The correctness gate. |
| `cmd/formatbench/report.go` | New. `results.json` and `results.md`. |
| `cmd/formatbench/uuid.go` | New. Collision-free repository names. |
| `docs/plans/bin-cue-benchmark.md` | New. This plan, committed where `AGENTS.md` says plans live. |
| `docs/usage/format-benchmark.md` | New. How to run it. Follow `docs/usage/memory-benchmark.md` in shape. |
| `AGENTS.md` | One row in the "Where the code lives" table, next to `cmd/membench/`. |
| `var/bin-cue-git.md` | Replace the `TODO(Xe)` at line 66 with the finished tables. |

## First results

Expand All @@ -233,17 +237,17 @@ old layout pays on a big one.

Push:

| Build | Wall | S3 requests | PUT | GET | LIST | Keys | Bucket bytes |
| --- | --: | --: | --: | --: | --: | --: | --: |
| before | 12.6s | 231 | 46 | 47 | 138 | 42 | 829.27 KiB |
| after | 2.1s | 19 | 5 | 11 | 3 | 4 | 902.98 KiB |
| Build | Wall | S3 requests | PUT | GET | LIST | Keys | Bucket bytes |
| ------ | ----: | ----------: | --: | --: | ---: | ---: | -----------: |
| before | 12.6s | 231 | 46 | 47 | 138 | 42 | 829.27 KiB |
| after | 2.1s | 19 | 5 | 11 | 3 | 4 | 902.98 KiB |

Clone:

| Build | Wall | S3 requests | GET | LIST | Wire bytes |
| --- | --: | --: | --: | --: | --: |
| before | 14.3s | 323 | 51 | 272 | 763.63 KiB |
| after | 13.4s | 53 | 49 | 4 | 767.96 KiB |
| Build | Wall | S3 requests | GET | LIST | Wire bytes |
| ------ | ----: | ----------: | --: | ---: | ---------: |
| before | 14.3s | 323 | 51 | 272 | 763.63 KiB |
| after | 13.4s | 53 | 49 | 4 | 767.96 KiB |

Two things to read from this before the full run:

Expand Down
4 changes: 4 additions & 0 deletions docs/plans/bound-concurrent-pushes.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Plan: bound the number of concurrent pushes"
---

# Plan: bound the number of concurrent pushes

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/erofs-lfs-resolution.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Resolve LFS pointers in EROFS snapshots"
---

# Resolve LFS pointers in EROFS snapshots

## Goal
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/git-default-branck-problems.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Fix: clone leaves `warning: remote HEAD refers to nonexistent ref, unable to checkout`"
---

# Fix: clone leaves `warning: remote HEAD refers to nonexistent ref, unable to checkout`

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/git-hooks-kefka.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Plan: post-receive hooks for objgitd (sandboxed via kefka)"
---

# Plan: post-receive hooks for objgitd (sandboxed via kefka)

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/git-hooks-stream.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Stream push-hook output to the client live"
---

# Stream push-hook output to the client live

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/git-http-protocol.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Plan: Git Smart HTTP protocol for objgitd"
---

# Plan: Git Smart HTTP protocol for objgitd

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/git-keep-pushed-packs-whole.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Keep pushed packs whole on git:// and SSH"
---

# Keep pushed packs whole on git:// and SSH

## Context
Expand Down
4 changes: 4 additions & 0 deletions docs/plans/kefka-uutils.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: "Update kefka and add uutils to the hook shell"
---

# Update kefka and add uutils to the hook shell

## Context
Expand Down
Loading
Loading