-
Notifications
You must be signed in to change notification settings - Fork 1
Manifests
Manifests describe a game build to be distributed.
TODO(james7132): Document.
Manifests are fetched via the GetLatestManifest gRPC endpoint. A call will attempt to update the local client with the latest manifest with one of of the three following responses:
- Empty: the client is up to date with the latest
- A diff between the client's local manifest and the latest manifest (format TBD)
- The full manifest for the latest build.
The build id and CRC64 checksum is included in all of the responses to validate the resultant ManifestProto. If the checksum does not match, The client should retry by repeating the same request, but without specifying the local_build_id parameter (effectively always requesting the full manifest).
Repeated failures, either for network issues or issues creating a valid manifest, should employ exponential backoff. Consistently sending repeated requests may result in the client being rate limited.
Clients should save a local copy of the latest manifest for every game/branch combination downloaded to reduce the amount of bandwidth consumed by both client and server in future requests.
TODO(james7132): Document.
TODO(james7132): Document.
A local game build can be verified using the following operation primitves:
- Fetch the remote manifest to validate against.
- Generate a manifest from the local game files.
- Diff the local manifest against the remote manifest.
If the resultant diff is empty, the local installation matches the remote.
Go clients should generally use Manifest.ToProto/ManifestBuilder.FromProto to convert manifests to and from it's wire format. For other clients, this section documents how to read and parse the protobuf into a more usable format.
Manifest's protobuffer definition are separated into three main components: The block list, the item hierarchy, and the metadata. The format is built to pack the maximal amount of information about the build in as small of a binary package as possible. Because of this format, it may be difficult to directly parse
A list of block metadata. Each element contains the size and hash of each block.
The hash uniquely identifies the block, is used to construct how to locate the block within remote storage, and can be used to independently verify if a copy of the block is valid. Block hashes should be unique within the list. In the rare situation that two different blocks within the same build share the same SHA-512 hash, return an error, and report the collision to the appropriate experts immediately. If blocks that share the same hash are the same, include it in the list only once. The hash must not be encoded and is provided as raw bytes.
The size provided is the uncompressed size of the block. It is used to calculate the size of files, layout files on disk before downloading the block. If the field is not set, the max_block_size value for the manifest is to be used instead.
This is an ordered list, and the index at which a block is situated matters. For maximum efficiency, blocks for a file should have the corresponding metadata stored contiguously within the list, in order they are seen in the original file. Each block is referenced by index elsewhere in the manifest.
Files and directories are stored in a recursive protobuffer definition. Each directory is stored as a map of strings to ManifestItemProtos. The string is the name of the child item within the directory
An item is a subdirectory are denoted by the presence of a map of strings to children. Like the parent directory, this also is a mapping of name to child item.
Files are denoted by the presence of a BlockRange list. Each block range defines a start ID and a size of the range. These are unsigned integers pointing to indices. Block ranges are listed sequentially in the order they are used within Clients should validate that the range is correct and stays within bounds, and appropriately return errors when the block ranges are invalid. Example of how to translate block ranges to a list of block IDs:
Proto:
range {
start_id: 521
size: 5
}
range {
start_id: 15
size: 3
}
Block IDs (Indices into the manifest block list):
[521, 522, 523, 524, 525, 15, 16, 17]
This submessage is always included and contains various metadata regarding the entire build.
For more up to date and complete information, see the source code ManifestProto.