-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework of parsing the header of a replay #143
Merged
Merged
Changes from 13 commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
1403423
Separate out the reading and interpreting of the file
Garanas 01f0f6e
Undo lfs of fafreplay
Sheikah45 01f3bc2
Fix replays that were damaged by LFS
Garanas 20155f0
Make the memory-based functions private
Garanas a5ee027
Clean up of the code
Garanas c7f75b4
Improve documentation of unknown values
Garanas bccff78
Document the header of a replay
Garanas a8e7303
First working end-to-end example of the new replay parsing
Garanas 43c0572
Restructure the project, process feedback by Sheikah in #142
Garanas 8558a85
Extend tests and fix a few bugs
Garanas a5467f0
Discover the byte that is set when queueing orders
Garanas d0e9566
Introduce the first semantics
Garanas dbb816b
Add interpretation of chat messages
Garanas 7d4ef66
Introduce additional semantics
Garanas 0f0aca2
Use atomics
Garanas cce2ee3
Rework the enums related to game options
Garanas a18552a
Remove excessive whitespace
Garanas 2caafd3
Add documentation that it still requires to be implemented
Garanas 07bee25
Fix typo in name
Garanas 020955d
Rename 'Utils' to 'LoadUtils'
Garanas 4348c64
Use a better describing exception
Garanas 49ddff9
Undo formatting changes
Garanas 76615a8
Remove the tokenizer of the header
Garanas 706ad77
Extend pattern matching of modern Java
Garanas bdee610
DO THAT SCREAMING SNAKE CASE THING
Garanas 1b9492f
Process feedback
Garanas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
10 changes: 10 additions & 0 deletions
10
faf-commons-data/src/main/java/com/faforever/commons/replay/ReplayContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| package com.faforever.commons.replay; | ||
|
|
||
| import com.faforever.commons.replay.body.Event; | ||
| import com.faforever.commons.replay.header.ReplayHeader; | ||
| import com.faforever.commons.replay.semantics.records.TrackedEvent; | ||
|
|
||
| import java.util.List; | ||
|
|
||
| public record ReplayContainer(ReplayMetadata metadata, ReplayHeader header, List<TrackedEvent> trackedEvents) { | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
125 changes: 125 additions & 0 deletions
125
faf-commons-data/src/main/java/com/faforever/commons/replay/ReplayLoader.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| package com.faforever.commons.replay; | ||
|
|
||
| import com.faforever.commons.replay.body.Event; | ||
| import com.faforever.commons.replay.body.ReplayBodyParser; | ||
| import com.faforever.commons.replay.body.ReplayBodyToken; | ||
| import com.faforever.commons.replay.body.ReplayBodyTokenizer; | ||
| import com.faforever.commons.replay.header.*; | ||
| import com.faforever.commons.replay.semantics.Semantics; | ||
| import com.faforever.commons.replay.semantics.records.TrackedEvent; | ||
| import com.fasterxml.jackson.databind.DeserializationFeature; | ||
| import com.google.common.io.BaseEncoding; | ||
| import com.google.common.io.LittleEndianDataInputStream; | ||
| import org.apache.commons.compress.compressors.CompressorException; | ||
| import org.apache.commons.compress.compressors.CompressorInputStream; | ||
| import org.apache.commons.compress.compressors.CompressorStreamFactory; | ||
| import org.apache.commons.compress.utils.IOUtils; | ||
| import org.jetbrains.annotations.Contract; | ||
| import org.jetbrains.annotations.NotNull; | ||
|
|
||
| import com.fasterxml.jackson.databind.ObjectMapper; | ||
| import java.io.ByteArrayInputStream; | ||
| import java.io.ByteArrayOutputStream; | ||
| import java.io.EOFException; | ||
| import java.io.IOException; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.nio.file.Files; | ||
| import java.nio.file.Path; | ||
| import java.util.Arrays; | ||
| import java.util.List; | ||
| import java.util.Objects; | ||
|
|
||
| public class ReplayLoader { | ||
|
|
||
| @Contract(pure = true) | ||
| private static ReplayHeader loadSCFAReplayHeader(LittleEndianDataInputStream stream) throws IOException{ | ||
| ReplayHeaderToken headerToken = ReplayHeaderTokenizer.tokenize(stream); | ||
| return ReplayHeaderParser.parseHeader(headerToken); | ||
| } | ||
|
|
||
| @Contract(pure = true) | ||
| private static @NotNull List<TrackedEvent> loadSCFAReplayBody(List<Source> sources, LittleEndianDataInputStream stream) throws IOException{ | ||
| List<ReplayBodyToken> bodyTokens = ReplayBodyTokenizer.tokenize(stream); | ||
| List<Event> bodyEvents = ReplayBodyParser.parseTokens(bodyTokens); | ||
| return Semantics.registerEvents(sources, bodyEvents); | ||
| } | ||
|
|
||
| @Contract(pure = true) | ||
| private static ReplayContainer loadSCFAReplayFromMemory(ReplayMetadata metadata, byte[] scfaReplayBytes) throws IOException { | ||
| try (LittleEndianDataInputStream stream = new LittleEndianDataInputStream((new ByteArrayInputStream(scfaReplayBytes)))) { | ||
| ReplayHeader replayHeader = loadSCFAReplayHeader(stream); | ||
| List<TrackedEvent> replayBody = loadSCFAReplayBody(replayHeader.sources(), stream); | ||
|
|
||
| if(stream.available() > 0) { | ||
| throw new EOFException(); | ||
| } | ||
|
|
||
| return new ReplayContainer(metadata, replayHeader, replayBody); | ||
| } | ||
| } | ||
|
|
||
| public static ReplayContainer loadSCFAReplayFromDisk(Path scfaReplayFile) throws IOException, IllegalArgumentException { | ||
| if (!scfaReplayFile.toString().toLowerCase().endsWith("scfareplay")) { | ||
| throw new IllegalArgumentException ("Unknown file format: " + scfaReplayFile.getFileName()); | ||
| } | ||
|
|
||
| byte[] bytes = Files.readAllBytes(scfaReplayFile); | ||
| return loadSCFAReplayFromMemory(null, bytes); | ||
| } | ||
|
|
||
| @Contract(pure = true) | ||
| private static ReplayContainer loadFAFReplayFromMemory(byte[] fafReplayBytes) throws IOException, CompressorException { | ||
| int separator = findSeparatorIndex(fafReplayBytes); | ||
| byte[] metadataBytes = Arrays.copyOfRange(fafReplayBytes, 0, separator); | ||
| String metadataString = new String(metadataBytes, StandardCharsets.UTF_8); | ||
|
|
||
| ObjectMapper parsedMetadata = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); | ||
| ReplayMetadata replayMetadata = parsedMetadata.readValue(metadataString, ReplayMetadata.class); | ||
|
|
||
| byte[] compressedReplayBytes = Arrays.copyOfRange(fafReplayBytes, separator + 1, fafReplayBytes.length); | ||
| byte[] scfaReplayBytes = decompress(compressedReplayBytes, replayMetadata); | ||
|
|
||
| return loadSCFAReplayFromMemory(replayMetadata, scfaReplayBytes); | ||
| } | ||
|
|
||
| public static ReplayContainer loadFAFReplayFromDisk(Path fafReplayFile) throws IOException, CompressorException, IllegalArgumentException { | ||
| if (!fafReplayFile.toString().toLowerCase().endsWith("fafreplay")) { | ||
| throw new IllegalArgumentException ("Unknown file format: " + fafReplayFile.getFileName()); | ||
| } | ||
|
|
||
| byte[] fafReplayBytes = Files.readAllBytes(fafReplayFile); | ||
| return loadFAFReplayFromMemory(fafReplayBytes); | ||
| } | ||
|
|
||
| private static int findSeparatorIndex(byte[] replayData) { | ||
| int headerEnd; | ||
| for (headerEnd = 0; headerEnd < replayData.length; headerEnd++) { | ||
| if (replayData[headerEnd] == '\n') { | ||
| return headerEnd; | ||
| } | ||
| } | ||
| throw new IllegalArgumentException("Missing separator between replay header and body"); | ||
| } | ||
|
|
||
| private static byte[] decompress(byte[] data, @NotNull ReplayMetadata metadata) throws IOException, CompressorException { | ||
| CompressionType compressionType = Objects.requireNonNullElse(metadata.getCompression(), CompressionType.QTCOMPRESS); | ||
|
|
||
| switch (compressionType) { | ||
| case QTCOMPRESS: { | ||
| return QtCompress.qUncompress(BaseEncoding.base64().decode(new String(data))); | ||
| } | ||
| case ZSTD: { | ||
| ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(data); | ||
| CompressorInputStream compressorInputStream = new CompressorStreamFactory() | ||
| .createCompressorInputStream(arrayInputStream); | ||
|
|
||
| ByteArrayOutputStream out = new ByteArrayOutputStream(); | ||
| IOUtils.copy(compressorInputStream, out); | ||
| return out.toByteArray(); | ||
| } | ||
| case UNKNOWN: | ||
| default: | ||
| throw new IOException("Unknown replay format in replay file"); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
leave this wrapped as it is more readable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's the formatter!! 😛
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Processed in: