Avoid crashes when running commands on huge files

FIX: JsonTools will now refuse to run plugin commands on files longer than int.MaxValue rather than crashing as soon as it tries to get the length. FIX: close progress bar for grepper form when we quit because of too much text to parse Also fix occasional error due to progress bar CHANGE: reduce the limit on how much text the grepper form can parse, because the old limits weren't strict enough to consistently prevent out-of-memory errors.
molsonkiko · Aug 1, 2024 · 8b4f53a · 8b4f53a
1 parent 4f07750
commit 8b4f53a
Show file tree

Hide file tree

Showing 13 changed files with 114 additions and 75 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -73,6 +73,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 1. If there would be an `OutOfMemoryException` due to running out of memory while formatting JSON (a likely occurrence when using the `grepper form`), that error is caught and reported with a message box, rather than potentially causing Notepad++ to crash.
 2. Ensure that hitting the down key does nothing when the last row of the [error form](/docs/README.md#error-form-and-status-bar) is selected.
 3. Fix bug with [random json from schema](/docs/README.md#generating-random-json-from-a-schema) when parsing a schema that has the `enum` key but not the `type` key.
+4. Make it so JsonTools simply does nothing rather than causing Notepad++ to crash when attempting to run plugin commands on files with more than 2147483647 bytes.
 
 ## [8.0.0] - 2024-06-29
 

diff --git a/JsonToolsNppPlugin/Forms/GrepperForm.cs b/JsonToolsNppPlugin/Forms/GrepperForm.cs
@@ -277,14 +277,14 @@ private void ReportJsonParsingProgress(int lengthParsedSoFar, int __)
                 lock (progressReportLock)
                 {
                     progressLabel.Text = Regex.Replace(progressLabel.Text, @"^\d+(?:\.\d+)?", _ => (lengthParsedSoFar / 1e6).ToString("F3", JNode.DOT_DECIMAL_SEP));
-                    progressBar.Value = lengthParsedSoFar;
+                    progressBar.Value = lengthParsedSoFar > progressBar.Maximum ? progressBar.Maximum : lengthParsedSoFar;
                 }
             }
             else
             {
                 // don't need to use the lock when reading files, because that is single-threaded
                 progressLabel.Text = Regex.Replace(progressLabel.Text, @"^\d+", _ => lengthParsedSoFar.ToString());
-                progressBar.Value = lengthParsedSoFar;
+                progressBar.Value = lengthParsedSoFar > progressBar.Maximum ? progressBar.Maximum : lengthParsedSoFar;
                 progressBarForm.Refresh();
             }
         }

diff --git a/JsonToolsNppPlugin/Forms/TreeViewer.cs b/JsonToolsNppPlugin/Forms/TreeViewer.cs
@@ -1273,7 +1273,7 @@ public static bool LengthOfStringInRegexMode(JNode[] nodes, char delim, char quo
 
         public void SelectTreeNodeJson(TreeNode node)
         {
-            if (Main.activeFname != fname)
+            if (Main.activeFname != fname || !Npp.TryGetLengthAsInt(out int len, false))
                 return;
             bool isRegex = GetDocumentType() == DocumentType.REGEX;
             (int selectionStart, int selectionEnd) = ParentSelectionStartEnd(node);
@@ -1288,7 +1288,7 @@ public void SelectTreeNodeJson(TreeNode node)
                     nodeEndPos = nodeStartPos + utf8Lengths[0];
                 }
                 else
-                    nodeEndPos = Main.EndOfJNodeAtPos(nodeStartPos, selectionEnd < 0 ? Npp.editor.GetLength() : selectionEnd);
+                    nodeEndPos = Main.EndOfJNodeAtPos(nodeStartPos, selectionEnd < 0 ? len : selectionEnd);
             }
             if (!isRegex && nodeStartPos == nodeEndPos) // empty selections are fine in regex mode
                 Translator.ShowTranslatedMessageBox(

diff --git a/JsonToolsNppPlugin/JSONTools/JsonGrepper.cs b/JsonToolsNppPlugin/JSONTools/JsonGrepper.cs
@@ -12,11 +12,15 @@ namespace JSON_Tools.JSON_Tools
 {
     public class TooMuchTextToParseException : Exception
     {
-        public static readonly int MAX_COMBINED_LENGTH_TEXT_TO_PARSE = IntPtr.Size == 4 ? int.MaxValue / 10 : int.MaxValue / 5;
+        /// <summary>
+        /// the limits are significantly lower than you might expect,
+        /// because the combined memory of the tree view and the Notepad++ buffer and the JNodes could exhaust all the memory that the OS allocated to Notepad++.
+        /// </summary>
+        public static readonly int MAX_COMBINED_LENGTH_TEXT_TO_PARSE = IntPtr.Size == 4 ? int.MaxValue / 20 : int.MaxValue / 8;
 
         public int lengthOfTextToParse;
 
-        private TooMuchTextToParseException(int lengthOfTextToParse)
+        public TooMuchTextToParseException(int lengthOfTextToParse)
         {
             this.lengthOfTextToParse = lengthOfTextToParse;
         }
@@ -25,19 +29,6 @@ public override string ToString()
         {
             return $"The total length of text ({lengthOfTextToParse}) to be parsed exceeded the maximum length ({MAX_COMBINED_LENGTH_TEXT_TO_PARSE})";
         }
-
-        /// <summary>
-        /// throw an exception of this type if lengthOfTextToParse is greater than <see cref="MAX_COMBINED_LENGTH_TEXT_TO_PARSE"/>,
-        /// based on the expectation that the computer would run out of memory while attempting to parse everything.<br></br>
-        /// Does nothing if an exception is not thrown.
-        /// </summary>
-        /// <param name="lengthOfTextToParse"></param>
-        /// <exception cref="TooMuchTextToParseException"></exception>
-        public static void ThrowIfTooMuchText(int lengthOfTextToParse)
-        {
-            if (lengthOfTextToParse > MAX_COMBINED_LENGTH_TEXT_TO_PARSE)
-                throw new TooMuchTextToParseException(lengthOfTextToParse);
-        }
     }
 
 	/// <summary>
@@ -146,7 +137,7 @@ private void ReadJsonFiles(string rootDir, bool recursive, params string[] searc
             catch { return; }
             totalLengthToParse = 0;
             SearchOption searchOption = recursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly;
-            FileInfo[] allFiles = dirInfo.GetFiles("*.*", searchOption);
+            FileInfo[] allFiles = dirInfo.GetFiles("*", searchOption);
             long totalLengthToRead = allFiles.Sum(x => x.Length);
             int nFiles = allFiles.Length;
             bool shouldReportProgress = reportProgress
@@ -170,7 +161,13 @@ private void ReadJsonFiles(string rootDir, bool recursive, params string[] searc
                     {
                         string text = fp.ReadToEnd();
                         totalLengthToParse += text.Length;
-                        TooMuchTextToParseException.ThrowIfTooMuchText(totalLengthToParse);
+                        if (totalLengthToParse > TooMuchTextToParseException.MAX_COMBINED_LENGTH_TEXT_TO_PARSE)
+                        {
+                            // throw an exception if there's too much text,
+                            // based on the expectation that the computer would run out of memory while attempting to parse everything.
+                            progressReportTeardown?.Invoke();
+                            throw new TooMuchTextToParseException(totalLengthToParse);
+                        }
                         fnameStrings[fname] = text;
                     }
                 }

diff --git a/JsonToolsNppPlugin/Main.cs b/JsonToolsNppPlugin/Main.cs
@@ -511,6 +511,8 @@ public static void OpenUrlInWebBrowser(string url)
         /// <returns></returns>
         public static (ParserState parserState, JNode node, bool usesSelections, DocumentType DocumentType) TryParseJson(DocumentType documentType = DocumentType.JSON, bool wasAutotriggered = false, bool preferPreviousDocumentType = false, bool isRecursion = false, bool ignoreSelections = false)
         {
+            if (!Npp.TryGetLengthAsInt(out int len, !wasAutotriggered))
+                return (ParserState.FATAL, new JNode(), false, DocumentType.NONE);
             JsonParser jsonParser = JsonParserFromSettings();
             string fname = Npp.notepad.GetCurrentFilePath();
             List<(int start, int end)> selRanges = SelectionManager.GetSelectedRanges();
@@ -524,7 +526,6 @@ public static (ParserState parserState, JNode node, bool usesSelections, Documen
                 // for some reason we don't want to use selections no matter what (probably because it was an automatic TryParseJson triggered by the timer)
                 || ignoreSelections;
             bool stopUsingSelections = false;
-            int len = Npp.editor.GetLength();
             double sizeThreshold = settings.max_file_size_MB_slow_actions * 1e6;
             bool hasOldSelections = false;
             DocumentType previouslyChosenDocType = DocumentType.NONE;
@@ -934,6 +935,8 @@ public static void CompressJson()
         public static Dictionary<string, (string newKey, JNode child)> ReformatFileWithJson(JNode json, Func<JNode, string> formatter, bool usesSelections)
         {
             var keyChanges = new Dictionary<string, (string newKey, JNode child)>();
+            if (!Npp.TryGetLengthAsInt(out int len))
+                return keyChanges;
             JsonFileInfo info;
             Npp.editor.BeginUndoAction();
             if (usesSelections)
@@ -943,7 +946,7 @@ public static void CompressJson()
                 pluginIsEditing = true;
                 int previouslySelectedIndicator = Npp.editor.GetIndicatorCurrent();
                 var keyvalues = obj.children.ToArray();
-                ClearPreviouslyRememberedSelections(Npp.editor.GetLength(), keyvalues.Length);
+                ClearPreviouslyRememberedSelections(len, keyvalues.Length);
                 int currentIndicator = selectionRememberingIndicator1;
                 Array.Sort(keyvalues, (kv1, kv2) => SelectionManager.StartEndCompareByStart(kv1.Key, kv2.Key));
                 foreach (KeyValuePair<string, JNode> kv in keyvalues)
@@ -1444,7 +1447,8 @@ private static string PathToPosition(KeyStyle style, char separator, int pos = -
         /// </summary>
         public static void SelectEveryValidJson()
         {
-            int utf8Len = Npp.editor.GetLength();
+            if (!Npp.TryGetLengthAsInt(out int utf8Len))
+                return;
             var selections = SelectionManager.GetSelectedRanges();
             if (SelectionManager.NoTextSelected(selections))
             {
@@ -1964,6 +1968,8 @@ public static int EndOfJNodeAtPos(int startUtf8Pos, int end)
         /// </summary>
         public static void SelectAllChildren(IEnumerable<int> positions, bool isJsonLines)
         {
+            if (!Npp.TryGetLengthAsInt(out int len))
+                return;
             int[] sortedPositions = positions.Distinct().ToArray();
             if (sortedPositions.Length == 0)
                 return;
@@ -1979,7 +1985,7 @@ public static void SelectAllChildren(IEnumerable<int> positions, bool isJsonLine
                     "Can't select all children", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
                 return;
             }
-            string slice = Npp.GetSlice(minPos, Npp.editor.GetLength());
+            string slice = Npp.GetSlice(minPos, len);
             var parser = new JsonParser(LoggerLevel.JSON5);
             int utf8ExtraBytes = 0;
             int positionsIdx = 0;
@@ -2054,7 +2060,8 @@ private static void DelayedParseAfterEditing(object s)
             DateTime now = DateTime.UtcNow;
             if (!settings.auto_validate
                 || !bufferFinishedOpening
-                || Npp.editor.GetLength() > settings.max_file_size_MB_slow_actions * 1e6 // current file too big
+                || !Npp.TryGetLengthAsInt(out int len, false)
+                || len > settings.max_file_size_MB_slow_actions * 1e6 // current file too big
                 || lastEditedTime == DateTime.MaxValue // set when we don't want to edit it
                 || lastEditedTime.AddMilliseconds(millisecondsAfterLastEditToParse) > now)
                 return;

diff --git a/JsonToolsNppPlugin/PluginInfrastructure/IScintillaGateway.cs b/JsonToolsNppPlugin/PluginInfrastructure/IScintillaGateway.cs
@@ -55,7 +55,7 @@ public interface IScintillaGateway
         void ClearDocumentStyle();
 
         /// <summary>Returns the number of bytes in the document. (Scintilla feature 2006)</summary>
-        int GetLength();
+        long GetLength();
 
         /// <summary>Returns the character byte at the position. (Scintilla feature 2007)</summary>
         int GetCharAt(int pos);

diff --git a/JsonToolsNppPlugin/PluginInfrastructure/ScintillaGateway.cs b/JsonToolsNppPlugin/PluginInfrastructure/ScintillaGateway.cs
@@ -164,10 +164,9 @@ public void ClearDocumentStyle()
             Win32.SendMessage(scintilla, SciMsg.SCI_CLEARDOCUMENTSTYLE, (IntPtr) Unused, (IntPtr) Unused);
         }
 
-        /// <summary>Returns the number of bytes in the document. (Scintilla feature 2006)</summary>
-        public int GetLength()
+        public long GetLength()
         {
-            return (int)Win32.SendMessage(scintilla, SciMsg.SCI_GETLENGTH, (IntPtr) Unused, (IntPtr) Unused);
+            return Win32.SendMessage(scintilla, SciMsg.SCI_GETLENGTH, (IntPtr)Unused, (IntPtr)Unused).ToInt64();
         }
 
         /// <summary>Returns the character byte at the position. (Scintilla feature 2007)</summary>

diff --git a/JsonToolsNppPlugin/Properties/AssemblyInfo.cs b/JsonToolsNppPlugin/Properties/AssemblyInfo.cs
@@ -28,5 +28,5 @@
 //      Build Number
 //      Revision
 //
-[assembly: AssemblyVersion("8.0.0.16")]
-[assembly: AssemblyFileVersion("8.0.0.16")]
+[assembly: AssemblyVersion("8.0.0.17")]
+[assembly: AssemblyFileVersion("8.0.0.17")]
diff --git a/JsonToolsNppPlugin/Utils/Npp.cs b/JsonToolsNppPlugin/Utils/Npp.cs
@@ -84,6 +84,33 @@ public static void SetLangIni(bool fatalError, bool usesSelections)
             }
         }
 
+        private static bool stopShowingFileTooLongNotifications = false;
+
+        /// <summary>
+        /// if <see cref="IScintillaGateway.GetLength"/> returns a number greater than <see cref="int.MaxValue"/>, return false and set len to -1.<br></br>
+        /// Otherwise, return true and set len to the length of the document.<br></br>
+        /// If showMessageOnFail, show a message box warning the user that the command could not be executed.
+        /// </summary>
+        public static bool TryGetLengthAsInt(out int len, bool showMessageOnFail = true)
+        {
+            long result = editor.GetLength();
+            if (result > int.MaxValue)
+            {
+                len = -1;
+                if (!stopShowingFileTooLongNotifications && showMessageOnFail)
+                {
+                    stopShowingFileTooLongNotifications = Translator.ShowTranslatedMessageBox(
+                        "JsonTools cannot perform this plugin command on a file with more than 2147483647 bytes.\r\nDo you want to stop showing notifications when a file is too long?",
+                        "File too long for JsonTools",
+                        MessageBoxButtons.YesNo, MessageBoxIcon.Warning) == DialogResult.Yes;
+
+                }
+                return false;
+            }
+            len = (int)result;
+            return true;
+        }
+
         public static DocumentType DocumentTypeFromFileExtension(string fileExtension)
         {
             switch (fileExtension)
@@ -208,7 +235,9 @@ public static void CreateConfigSubDirectoryIfNotExists()
         /// </summary>
         public static void RemoveTrailingSOH()
         {
-            int lastPos = editor.GetLength() - 1;
+            if (!TryGetLengthAsInt(out int lastPos, false))
+                return;
+            lastPos--;
             int lastChar = editor.GetCharAt(lastPos);
             if (lastChar == 0x01)
             {

diff --git a/docs/README.md b/docs/README.md
@@ -57,7 +57,7 @@ We can open up the JSON tree viewer in the main menu by navigating Plugins -> Js
 You can click on the nodes in that tree to see the children. When you select a node, the caret will snap to the line of the node you've selected. *New in [version 5](/CHANGELOG.md#500---2023-05-26): snaps to position instead.*
 
 __NOTES__
-1. __*JsonTools only works with UTF-8 encoded JSON.*__
+1. __*JsonTools only works with UTF-8 encoded JSON.*__ This should only be a problem if you have a JSON file in an encoding that Notepad++ cannot correctly interpret.
 2. If you submit a RemesPath query that is anything other than the default `@`, the JSON tree may no longer send the caret to the correct position.
 3. If you [edit your JSON](/docs/RemesPath.md#editing-with-assignment-expressions) with RemesPath queries *and then undo your change with `Ctrl+Z` or similar, the undo action will not undo the changes to the JSON*. To re-sync the JSON with the document, you will have to close and then re-open the tree view.
     - As of version 3.5.0, you can use the `Refresh` button to refresh the tree with the most recently parsed JSON, rather than closing and re-opening the tree.
@@ -69,6 +69,8 @@ __NOTES__
     - `Ctrl+Down` while in the tree selects the last direct child of the currently selected node. *Added in [v6.0](/CHANGELOG.md#600---2023-12-13).*
     - `Escape` takes focus from the tree view back to the editor.
 5. Beginning in [v4.4.0](/CHANGELOG.md#440---2022-11-23), you can have multiple tree views open.
+6. Beginning in [v8.1](/CHANGELOG.md#810---unreleased-yyyy-mm-dd), JsonTools will refuse to perform plugin commands on files with more than 2147483647 bytes. This is the same size restriction that 32-bit Notepad++ has for opening files.
+    - Prior to that release, JsonTools would cause Notepad++ to __crash__ if you attempted to execute plugin commands on files that large.
 
 If a node has a `+` or `-` sign next to it, you can click on that button to expand the children of the node, as shown here.
 
@@ -254,6 +256,8 @@ Prior to [v6.1](/CHANGELOG.md#610---2023-12-28), this automatic validation force
 
 Beginning in [v7.0](/CHANGELOG.md#700---2024-02-09), this automatic validation will only ever attempt to parse the entire document, not [a selection](#working-with-selections), and automatic validation is always disabled in selection-based mode. Prior to v7.0, automatic validation could change the user's selections unexpectedly.
 
+__WARNING:__ If this setting is turned on, versions of JsonTools older than [v8.1](/CHANGELOG.md#810---unreleased-yyyy-mm-dd) will cause Notepad++ to crash if you attempt to open *any* file with more than 2147483647 bytes.
+
 ## Path to current position ##
 
 *Added in version v5.0.0*