Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update dependency ClosedXML to 0.104.2 #80

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update dependency ClosedXML to 0.104.2 #80

wants to merge 1 commit into from

Conversation

renovate[bot]
Copy link
Contributor

@renovate renovate bot commented Mar 16, 2023 •
edited
Loading

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
ClosedXML 0.97.0 -> 0.104.2 age adoption passing confidence

Release Notes

ClosedXML/ClosedXML (ClosedXML)

v0.104.2

Compare Source

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.104.1...0.104.2

v0.104.1

Compare Source

Release notes from 0.102.1 to the 0.104.1.

Summary of breaking changes is available at docs.closedxml.io:

OpenXML SDK

OpenXML SDK has released version 3. The 0.104.0 uses it as a dependency.

XLParser replaced with ClosedParser

The XLParser has been replaced with ClosedParser. The key benefits are

  • performance - ~2μs/formula, it's likely formulas will be parseable on the demand, necessary for construction of dependency tree
  • A1/R1C1 parsing parity - both modes can be parsed with no problems
  • AST oriented - it's likely a construction of AST in memory won't even be necessary, just use AST factory to evaluate formula directly

There is also a visualizer to display AST in a browser at https://parser.closedxml.io

image

Formula Calculation

In previous version, formulas used to be calculated recursively. Each formula checked it's supporting cells for other formulas and if there were some, they were recursively evaluated. There was some logic to decrease number of evaluations. That works for a very simple cases, but isn't very good for various non-happy paths (i.e. cells weren't calculated when they should be).

This version has replaced it with a standard

  • dependency tree for checking which formulas are dirty and need to be recalculated
  • calculation chain that manages dependencies and order of formulas during calculation

For more info, see docs, the Microsoft has a page about principles Excel Recalculation
and there is one with API at docs.closedxml.io.

image

Structured references

New parser also allows a basic evaluation of structured references. Format of structured reference must use official grammar, not Excel friendly names (e.g. Pastry[@​Name] is user-friendly name for Pastry[[#This Row],[Name]]). It's now possible to

using var wb = new XLWorkbook();
var ws = wb.AddWorksheet();
ws.Cell("A1").InsertTable(new Pastry[]
{
    new("Cake", 14),
    new("Waffle", 3),
}, "Pastry");

ws.Cell("D1").FormulaA1 = "SUM(Pastry[Price])";
ws.Cell("D3").FormulaA1 = "\"Pastry \" & Pastry[[#This Row],[Name]]";
wb.RecalculateAllFormulas();

Console.WriteLine($"Expected: {17}, Actual: {ws.Cell("D1").Value}");
Console.WriteLine($"Expected: \"Pastry Waffle\", Actual: {ws.Cell("D3").Value}");

Expected: 17, Actual: 17
Expected: "Pastry Waffle", Actual: Pastry Waffle

Renaming sheet updates formulas

When a sheet is renamed, a formula referencing the sheet is also updated. This is a part of long term effort to fix effects of structural changes of a workbook. It will be a long road (e.g. sheet still delete doesn't swicth to #REF!),** but is one of basic features that should be working acorss the board.

using var wb = new XLWorkbook();
var sheet = wb.AddWorksheet();
var anotherSheet = wb.AddWorksheet("Another");
sheet.Cell("A1").FormulaA1 = "Another!B4";
anotherSheet.Name = "Changed";
Console.WriteLine(sheet.Cell("A1").FormulaA1);

Changed!B4

Workbook structure

Internal structure has been cleaned up and optimized.

The dirty tracking has been moved out of cells to formulas and thus memory taken up by a single cell value is now only 16 bytes instead of 24 (?) bytes in 0.102. Of course there are some other structures around that take up memory as well, but the single cell value is now 16 bytes (I hoped for 8, but not feasible with double, DateTime and TimeSpan as possible cell values - all take up 8 bytes... not enough bits).

The same string in different instances is now not duplicated, but only one instance is used. As seen on following test, it can lead to significant decrease in memory consumption. 250k rows with 10 text rows (same string, different instance): 117 MiB om 0.103 vs 325 MiB in 0.102.1.

InsertData performance

Insert 250k rows of 10 columns of text and 5 columns of numbers (gist).

Description Rows Columns Time/Memory to insert data Save workbook Total time/memory
0.103.0-beta 250 000 15 1.619 sec / 117 MiB 6.343 sec 477 MiB
0.102.1 250 000 15 7.160 sec / 325 MiB 6.676 sec 692 MiB

Loading of cells is now done through streaming

Basically workbooks with a large amount of cells should see ~15%-20% speedup (as long as there are mainly values, not styles or OLAP metadata....).

Reading the 250k from previous chapter:

Description Rows Columns Time to load data Used memory
0.103.0-beta 250 000 15 15.648 sec 236 MiB
0.102.1 250 000 15 20.460 sec 329 MiB

Of course, this includes all stuff from 0.103.0-beta. Version 0.103 never got a non-beta release.

Pivot tables

The internal structure of pivot tables, along with most other features, has been completely overhauled. This update should significantly reduce crashes when loading and saving workbooks containing pivot tables.

The main issue with the previous internal structure was that it didn't align with the structure used by OOXML. This was problematic because we need to support all valid files. As a result, we have to handle a wide range of inputs and correctly convert them to our internal structure, which is rather hard. A more clear 1:1 mapping with OOXML is much simpler and more reliable.

AutoFilter

The Autofilter feature has been revamped, which includes some API changes. Its behavior is now more closely aligned with how Excel operates. The XML documentation provides detailed explanations, and there is a dedicated documentation page. Several bugs have also been fixed.

For more details, refer to the Autofilter section of the migration guide.

Source link

Although ClosedXML still doesn't have source package (Fody static weaving causes pdb mismatch and nuget will refuse symbol package), there is a source link info in the package.

SourceLink basically takes a repository and a commit from the package and retrieves source from directly from forge (in this case GitHub).

CommonCrawl dataset

When workbook is a valid one, ClosedXML shouldn't throw on load. That is a rather high priority (more than saving or manipulation). Unfortunately, that is hard to find such areas that cause most problems.

One of activities that was going in a background is trying to use excel files around the internet (found by CommonCrawl) to evaluate how bad it is. There aren't results yet, but it is something that is going on.

What's Changed

Technical debt

Performance improvements

Features

Bugfixes

Documentation

Breaking changes

AutoFilter

Formulas

Functions

Dependencies

Fixes

Pivot tables

New Contributors

Full Changelog: ClosedXML/ClosedXML@0.104.0-preview2...0.104.0-rc1

v0.104.0

Compare Source

v0.102.3

Compare Source

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.102.2...0.102.3

v0.102.2

Compare Source

Add a warning about allowed ranges of DocumentFormat.OpenXML see issue #​2220 and PR #​2246.

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.102.1...0.102.2

v0.102.1: - SixLabors.Fonts dependency update

Compare Source

SixLabors.Fonts has released version 1.0.0 and some NET Framework projects suddently have errors due to NuGet behavior.

If a project is consuming ClosedXML through package.config instead of PackageReference style projects, the NuGet will resolve version 1.0.0 instead of declared beta19 dependency. SixLabors.Fonts has API changes and thus it will start to throw MissingMethodExceptions.

The issue should only affect net framework projects, not dotnet core that use PackageReference style by default.

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.102.0...0.102.1

v0.102.0

Compare Source

Breaking changes

Please read migration guide from 0.101 to 0.102. The key ones changes are:

  • IXLCell is now a proxy to a sparse array and a new proxy is created each time it is requested by user code. Object.ReferenceEquals(ws.Cell("A1"), ws.Cell("A1")) now evaluates to false (used to be true)
  • IXLWorksheet AddWorksheet(DataTable dataTable) and IXLWorksheet AddWorksheet(DataTable dataTable, string sheetName) now use different name for the created table.
Significant changes
Replaced cell storage engine (#​1969)

The cells in a workbook used to be stored in a Dictionary<int, Dictionary<XLCell>>. That has several significant drawbacks and the storage has been replaced with a sparse arrays of individual slices (basically a sparse array containing a specific part of a cell).

slices

Key benefits (in long term):

  • Decrease in memory consumption - sparse arrays don't use values that aren't use, from esoteric OLAP cube id to styles.
  • Bulk operations - there was pretty much no way to optimize bulk operations. Clear - go through each cell. With sparse arrays, ClosedXML can operate on pices of an array (e.g. preallocate for large inserts, clear an area).
  • Most operations use only data from a slice (e.g. setting a formula doesn't have to check all cells, only those that contain formula). Due to nature of sparse array of a slice, I only need to go through cells that contains actually useful content.
  • Easier insertion/deletion of rows. XLCell originally contain an address and when a row was inserted, I had to go though all cells and fix address (+ 20 other things).

Other than memory, it's a potential for the future. Replacing a storage engine is not simple and pretty much everything uses XLCell adapter.

An example of different for 500k rows of value only cells (gist). About 200MB vs 900MB.
image
image

Embedded fonts (#​2106)

Default graphic engine of ClosedXML now contains an embedded font. That should be a quality of life improvement for users on Linux and other non-Windows environment who encountered

Unable to find font font name or fallback font fallback font name. Install missing fonts or specify a different fallback font through ‘LoadOptions.DefaultGraphicEngine = new DefaultGraphicEngine(“Fallback font name”)’..

The embedded font is an absolute bare bones Carlito font (though with a different name to avoid collision with the real one).

See doc for workflow of font selection: https://closedxml.readthedocs.io/en/latest/features/graphic-engine.html#fallback-and-embedded-font

Array formulas

A basic support for array formulas has been added. You can create array formula through IXLRangeBase object.
csharp ws.Range("B2:D3").FormulaArrayA1 = "B1:D1*POWER(1+A2:A3, 2)";
image

For more info, see doc: https://closedxml.readthedocs.io/en/latest/features/formulas.html#array-formulas

Refactored pivot source (#​1238)

The ancient PR from 2019 has been finally fixed and merged. It doesn't really add very useful features (multiple pivot tables don't have to have individual data store files in a xlsx file), but it is a big step to represent pivot cache data in a workbook. That is pretty much required to do anything useful with pivot tables.

The biggest visible improvement is that there is at least some very basic documentation about pivot tables https://closedxml.readthedocs.io/en/latest/features/pivot-tables.html

Bugfix: Normal style is not required in a workbook (#​2102)

Some other OOXML producers (e.g. NPOI) don't add Normal style into a stylesheet and in 0.101 a feature had a missing null check leading to NullReferenceException.

Misc info & next release

Not really something that is useful at this moment, but there has been significant work done on a replacement of a XLParser (https://github.com/ClosedXML/ClosedXML.Parser).

I hope to be done with the Phase 1 of my maintainership - architecture and dependencies. Most of the "OMG, this must be done before anything else" has been done. I hope to finish the rest (split writers of individual files from 5000+LOC file, use better parser and use calculation chain for formula evaluation) in the next one.

What's Changed
Performance improvements
Features
Bugfixes
Documentation
Quality of life
New Contributors

Full Changelog: ClosedXML/ClosedXML@0.101.0...0.102.0

v0.101.0

Compare Source

Mostly speed improvements, but some cool and nifty improvements, see changelog below.

Other than some enums being changed from int to byte, and a new method on IXLGraphicEngine interface there should be no breaking changes in the release (see https://closedxml.readthedocs.io/en/latest/migrations/migrate-to-0.101.html).

Worksheet saving (#​1838)

Saving of cells in a worksheet has been significantly improved. As an example, a file of 100k rows and 44 columns (see gist) took 38 seconds to save in 0.100.3, but only 17 seconds in 0.101-rc .

Speedup-0 101-rc
Speedup-0 100 3

Adjust to content (#​1991)

Adjust to content has been refactored and it now calculates size of text by counting glyphs. Originally, we asked SixLabors.Fonts library to calculate it, but it is really slow (it has to deal with all possible typographic options, like having rtl and ltr text on same line).

It is faster to just use glyph size, plus I am pretty sure Excel also calculates width/height of a cell that way (likely legacy from earlier 90s version).

Time to run AdjustToContent using a sample of 1000 rows.

Run# 0.96.0 0.100.3 0.101-rc
1 168 ms 1203 ms 1674 ms
2 4 ms 568 ms 31 ms
3 4 ms 551 ms 25 ms
4 4 ms 540 ms 26 ms
5 3 ms 544 ms 25 ms
6 3 ms 550 ms 26 ms
7 4 ms 556 ms 46 ms
8 3 ms 556 ms 28 ms
9 4 ms 541 ms 23 ms
10 3 ms 535 ms 23 ms
Changelog
Performance improvements
Features
Cleanup updates
Bugfixes
Documentation
New Contributors

Full Changelog: ClosedXML/ClosedXML@0.100.3...0.101.0

v0.100.3: Bulk insert of numbers

Compare Source

Fix a regression where some types of numbers were inserted as text by InsertData/InsertTable API.

// Only int and double were inserted as numbers, now all types are
cell.InsertData(new object[] { (sbyte)1, (byte)2, (short)3, (ushort)4, (uint)6, (long)7, (ulong)8, 15f, 17m });
What's Changed

Full Changelog: ClosedXML/ClosedXML@0.100.2...0.100.3

v0.100.2: Nullable conversion

Compare Source

Nullable conversions

Second fix for https://github.com/ClosedXML/ClosedXML/releases/tag/0.100.0.

Added conversion for nullable numbers/DateTime/TimeSpan to XLCellValue. Null values will be converted to Blank.Value in the XLCellValue. There is no value in having user code littered by cell.Value = nullableNumber ?? Blank.Value.

Null strings assigned to XLCellValue will also be converted to Blank.Value (unlike exception from 0.100).

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.100.1...0.100.2

v0.100.1: Implicit conversion

Compare Source

This is a quick improvement for https://github.com/ClosedXML/ClosedXML/releases/tag/0.100.0.
It adds an implicit conversion for decimal numbers to XLCellValue, so users who generate workbooks with decimal number don't have to add explicit casting from decimal to double everywhere.

What's Changed

Full Changelog: ClosedXML/ClosedXML@0.100.0...0.100.1

v0.100.0: 0.100: Clean Break

Compare Source

Clean Break

These are release notes for a version 0.100. We skipped a few version since the last release (0.97), because 0.100 should denote a major change at the very heart of ClosedXML. Not as clean break as I hoped, but close enough.

The list of all things that were changed from 0.97 to 0.100 is at the migration guide at the https://closedxml.readthedocs.io/en/latest/migrations/migrate-to-0.100.html

This is more like list of you should upgrade despite breaking changes :)

Memory consumption during big was decreased

Memory consumption during saving of large data workbooks was significantly improved. Originally, ClosedXML workbook representation was converted to DocumentFomrat.OpenXML DOM representation and the DOM was then saved. Instead of creating whole DOM, sheet data (=cell values) are now directly streamed to the output file and aren't included in the DOM.

To demonstrate difference, see the before and after memory consumption of a report that generated 30 000 rows, 45 columns. Memory consumption has decreased from 2.08 GiB 🡆 0.8 GiB.

Save cells and strings through DOM: 2.08 GiB
1874-dtoMemory-save-30k-text-DOM

Save cell and strings through streaming: 0.8 GiB
1874-dtoMemory-save-30k-text-streaming

The purple area are bytes of uncompressed package zip stream.

Cell value is now strongly typed

IXLCell.Value and `IXL

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate renovate bot added the dependencies This PR updates or changes a dependency file label Mar 16, 2023
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.100.3 chore(deps): update dependency closedxml to v0.101.0 Apr 17, 2023
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.101.0 chore(deps): update dependency closedxml to v0.102.0 Jun 24, 2023
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.102.0 chore(deps): update dependency closedxml to v0.102.1 Aug 19, 2023
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.102.1 chore(deps): update dependency closedxml to v0.102.2 Jan 5, 2024
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.102.2 chore(deps): update dependency closedxml to v0.102.3 Jul 18, 2024
@renovate renovate bot changed the title chore(deps): update dependency closedxml to v0.102.3 Update dependency ClosedXML to v0.102.3 Aug 10, 2024
@renovate renovate bot changed the title Update dependency ClosedXML to v0.102.3 Update dependency ClosedXML to 0.102.3 Aug 28, 2024
@renovate renovate bot changed the title Update dependency ClosedXML to 0.102.3 Update dependency ClosedXML to 0.104.1 Sep 30, 2024
@renovate renovate bot changed the title Update dependency ClosedXML to 0.104.1 Update dependency ClosedXML to 0.104.2 Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dependencies This PR updates or changes a dependency file
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
0 participants