Reproducible Git Submodule Checkouts on Cloud Mac CI

An iOS project may clone successfully on a development machine yet stall on submodules in cloud Mac CI: the main repository checks out, but private components fail with permission errors; nested dependencies remain on old commits; or a shallow clone cannot find the required object. The problem is usually not Xcode. It is that the checkout process does not clearly distinguish between the main repository’s current branch and the submodule commits recorded by the main repository.

Define the reproducibility boundary first

For each Git Submodule, the main repository stores a path, a remote URL, and an exact commit. The goal of a reproducible build is for the same main repository commit to always resolve to the same submodule objects—not to track the latest state of each submodule branch on every run.

First, record a baseline in the development environment:

git submodule status --recursive
git config --file .gitmodules --get-regexp 'submodule\..*\.path'
git config --file .gitmodules --get-regexp 'submodule\..*\.url'

A - at the beginning of a status line means the submodule has not been initialized. A + means the working tree commit does not match the commit recorded by the main repository. U indicates a merge conflict. None of these states should be treated as normal before the project enters CI.

Do not run git submodule update --remote in a build job. It follows the configured branch as it moves, causing the same main repository commit to resolve to different dependencies at different times.

Relative URLs are useful when repositories move within the same hosting boundary, but they are resolved against the main repository’s remote URL. If a job temporarily rewrites origin, run git submodule sync --recursive before initialization and print the sanitized effective URLs to verify how they were resolved.

Establish a single checkout entry point

Do not mix clone --recurse-submodules, manual pulls from inside submodule directories, and recovery commands embedded in build scripts. Instead, have CI fetch the main repository first, then use one entry point to process every submodule level:

set -euo pipefail

git submodule sync --recursive
git submodule update --init --recursive --jobs 4

expected_file="$(mktemp)"
actual_file="$(mktemp)"
trap 'rm -f "$expected_file" "$actual_file"' EXIT

git ls-tree -r HEAD |
  awk '$2 == "commit" { print $3, $4 }' |
  sort > "$expected_file"

git submodule status --recursive |
  sed -E 's/^[ +\-U]([0-9a-f]+) ([^ ]+).*/\1 \2/' |
  sort > "$actual_file"

diff -u "$expected_file" "$actual_file"

Here, --jobs 4 only limits concurrent network requests; it is not a fixed performance recommendation. Network conditions, remote rate limits, and the number of submodules vary by node, so start with lower concurrency and adjust it based on the failure rate.

The script should also run git diff --submodule=log --exit-code before the build. If an installation step silently modifies a submodule working tree, the job should fail immediately instead of archiving a build produced from unknown source code.

Use shallow clones carefully

Shallow clones reduce data transfer, but they are also a common cause of the “the commit exists on the remote, but CI cannot find it” problem. A submodule object recorded by the main repository may predate the current shallow boundary, or it may only be reachable through the history of a deleted branch.

If shallow clones must be retained, try this first:

git submodule update --init --recursive --depth 50

If it fails, do not switch directly to --remote. Enter the path that reported the error, run git fetch --deepen=100, and then verify the object with git cat-file -e <commit>^{commit}. If the object is still unavailable, investigate remote object retention or the commit recorded by the main repository instead of adding more random retries.

Isolate credentials for private submodules

Submodules may have different authorization boundaries. The safest approach is to create a temporary Git configuration for each job, pass it only for the lifetime of the process, and leave the node’s global configuration unchanged.

git_config="$(mktemp)"
chmod 600 "$git_config"
trap 'rm -f "$git_config"' EXIT

git config --file "$git_config" credential.helper ""
git config --file "$git_config" protocol.version 2

GIT_CONFIG_GLOBAL="$git_config" \
git submodule update --init --recursive

Authentication values should come from protected CI variables or short-lived files. They must not be written to .gitmodules, remote URLs, command arguments, or build logs. At a minimum, check the following after the job:

git config --global --list
git remote -v
git submodule foreach --recursive 'git remote -v'

Sanitize the output before inspecting it. If credentials were embedded in a URL, deleting temporary files is not enough. Restore the remote URL as well, and inspect shell history, diagnostic bundles, and cache directories.

Turn failures into actionable evidence

A submodule failure should not be reduced to a single “exit code 128” message. Record the main repository commit, failing path, expected object, actual object, resolved hostname, checkout depth, and failure stage—but never usernames, tokens, or full authenticated URLs.

Use a consistent troubleshooting sequence:

1. Run git ls-tree HEAD <path> to confirm the object expected by the main repository.
2. Check .gitmodules and the local Git configuration to confirm the final URL.
3. Use git cat-file -e to determine whether the object is already available locally.
4. Verify remote access with read-only credentials.
5. Check the shallow-clone boundary and the state of nested submodules.
6. Clean the working tree and rerun the same checkout script.

Finally, archive git submodule status --recursive as build metadata. Its output is small, but it identifies exactly which dependency commits were used for that build. Jobs on SDKMac nodes should follow the same principle: pin the source inputs before starting Xcode or any other toolchain, keeping checkout failures clearly separated from compilation failures.

Frequently asked questions