Diagnosing and Safely Recovering Git Object Corruption in Cloud Mac CI

When a pipeline suddenly reports bad object, missing blob, or pack checksum mismatch during checkout, rerunning it may restore the build—or cause more jobs to fail at the same time. These symptoms usually do not indicate an application code error. Instead, they point to inconsistencies in the workspace, object cache, or Git packfiles. The correct response is not to clear the cache immediately, but to stop further writes, preserve the evidence, and determine whether the corruption is limited to one repository or affects a shared layer.

Distinguish Network Failures from Object Corruption First

Network failures usually occur during fetch operations. Their logs commonly show interrupted connections, unexpected remote disconnects, or incomplete transfers. Object corruption more often appears while checking out files, merging, reading history, or creating archives, and the error typically includes a specific object hash. Record the failed command, exit code, commit ID, workspace path, and cache generation. Do not retain only the final few dozen lines of the pipeline log.

Start with read-only checks:

git status --porcelain=v2
git rev-parse --show-toplevel
git rev-parse HEAD
git fsck --full --strict --no-dangling

If git fsck reports missing blob, a file object reachable from a reference is missing. A missing tree error will often prevent directory checkout, while invalid sha1 pointer indicates that a reference points to an unreadable object. Dangling objects alone do not prove that the repository is corrupt; they may have been left behind by a rebase, compaction, or superseded commit.

The goal of diagnosis is to determine the scope of the corruption, not merely to make the current command pass temporarily. Any operation that rewrites the object database should be postponed until the evidence has been preserved.

Freeze the Workspace and Preserve Evidence

First, pause runners, scheduled fetches, and cache update jobs that write to the same directory. Do not run git gc, git prune, repacking operations, or recursive deletion in the original directory. These actions can change the object layout and make the initial failure impossible to reproduce.

At a minimum, preserve the complete job log, .git/HEAD, .git/config, .git/packed-refs, .git/refs, .git/logs, the current status, and all uncommitted changes. The source workspace may contain untracked files, so review the file list before archiving it to avoid collecting credentials or large build artifacts.

incident="$HOME/git-incidents/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$incident"
git status --porcelain=v2 > "$incident/status.txt"
git show-ref --head > "$incident/refs.txt"
git reflog show --all --date=iso > "$incident/reflog.txt"
git diff --binary > "$incident/worktree.patch"
git diff --cached --binary > "$incident/index.patch"
git fsck --full --strict > "$incident/fsck.txt" 2>&1 || true

If CI can recreate the workspace at any time, the safest approach is to rename and isolate the entire directory instead of attempting an in-place repair. Mark the isolated directory as read-only and record the job and cache source associated with it.

Locate Corruption in Loose Objects, Packfiles, and Shared Caches

Git objects may exist as loose files or be stored in a .pack file. When an error provides an object hash, first query its type:

object="0123456789abcdef0123456789abcdef01234567"
git cat-file -t "$object"
git cat-file -s "$object"

If the command fails, check whether .git/objects/${object:0:2}/${object:2} exists. If the file exists but cannot be read, its contents are usually truncated, fail validation, or were incompletely copied at the storage layer. Do not copy a file with the same name from another unverified source into the evidence directory.

For packfiles, verify each index individually:

for index in .git/objects/pack/*.idx; do
  git verify-pack -v "$index" >/dev/null || printf '%s\n' "$index"
done

If the repository uses alternates, also inspect .git/objects/info/alternates. When multiple jobs share the same writable object directory, a single interrupted write can affect every workspace that references it. In that case, recloning one repository does not remove the root cause. You must stop publishing the affected cache generation.

Check Whether the Corruption Is Reproducible

Copy the isolated directory before reproducing the checks. Record the remaining filesystem capacity, how the process terminated, and the number of jobs running during the same period. If the same object fails every time, investigate the object source and cache first. If the failing object changes, inspect disk pressure, concurrent writes, and workspace cleanup logic.

Recover with a Clean Clone Instead of In-Place Surgery

For CI workspaces without manual changes, the safe recovery path is usually to create a completely new directory, fetch the target commit in full, validate it, and then atomically replace the old directory. Do not allow the new clone to reuse an object cache that has not yet been verified.

root="$HOME/ci-workspaces"
next="$root/project.next"
active="$root/project"
failed="$root/project.failed"

rm -rf "$next"
git clone --no-local "$REPOSITORY_PATH" "$next"
git -C "$next" checkout --detach "$EXPECTED_COMMIT"
git -C "$next" fsck --full --strict
test "$(git -C "$next" rev-parse HEAD)" = "$EXPECTED_COMMIT"
mv "$active" "$failed"
mv "$next" "$active"

Before replacement, also run the project's minimum acceptance checks, such as parsing the project, listing build entry points, or executing a small set of fast tests. If the isolated repository contains unpushed commits, do not assume that recloning will recover them. Use the preserved references and reflog to attempt recovery from a copy. If the objects are genuinely missing, they can only be restored from a trusted remote, another complete clone, or the original work products.

Integrate Object Validation into the Cache Publication Process

A shared Git cache should not be writable by every job at the same time. A more reliable model is “build, validate, publish”: jobs construct a new cache in a temporary directory, run git fsck after fetching completes, and rename the directory as a new read-only generation only after validation succeeds. Running jobs remain pinned to their own generation rather than following updates made while they are in progress.

Routine checks can be reduced to four rules: validate objects before publishing a cache; keep job workspaces separate from the shared baseline; preserve object hashes and cache generations when failures occur; and have cleanup scripts delete only complete generations that are no longer referenced. This keeps the impact of an incomplete write caused by a failed download or terminated process confined to a temporary directory that has not yet been published.

Finally, turn the recovery procedure into an executable script and validate it in a test repository that contains no production data. A reliable drill must demonstrate at least that a corrupted cache is no longer distributed, the target commit can be rebuilt from a clean source, uncommitted changes have an independent preservation path, and the replacement process never allows two jobs to write to the same workspace simultaneously.

Frequently asked questions