Package Versioning Policy versus Semantic Versioning

[tfausak]

Many people on Reddit took issue with me recommending SemVer over the PVP.

I totally disagree about SemVer v. PVP Versioning.

https://www.reddit.com/r/haskell/comments/5iok3l/haskell_package_checklist/

I am obviously in favor of SemVer. I wanted some empirical data about Haskell package versioning. I wrote a script that downloads Hackage's package index and looks at the version numbers.

Found 10642 packages.
Number of version components => number of packages
3 => 4255
4 => 4638
Number of (version components, major versions) => number of packages
(3,1) => 3747
(4,1) => 4357

https://gist.github.com/tfausak/98726c8255c5bc5c940e0f4939978a43

I assume that 3 version components means SemVer and 4 version components means the PVP. That's not quite accurate since SemVer requires exactly 3 components but the PVP requires at least 3.

A package version number SHOULD have the form A.B.C, and MAY optionally have any number of additional components

http://pvp.haskell.org/

Anyway, the PVP is the most popular versioning policy on Hackage. But SemVer is a very close second.

Of the 10,642 #Haskell packages on Hackage, 4,638 use the PVP and 4,255 use SemVer.

https://twitter.com/taylorfausak/status/811578904209227776

Most of the packages on Hackage never change their most significant version number. Only 9% have.

Of the 10,642 #Haskell packages on Hackage, only 921 have ever changed their most significant version number.

https://twitter.com/taylorfausak/status/811579159671668736

6% of PVP packages have ever changed their most significant version number. That's a lot less than the overall rate.

Of the 4,638 #Haskell packages on Hackage that use the PVP, only 281 have ever changed their most significant version number.

https://twitter.com/taylorfausak/status/811579605278670849

12% of SemVer packages have ever changed their most significant version number. That's a lot more than the overall rate.

Of the 4,255 #Haskell packages on Hackage that use SemVer, 508 have changed their most significant version number.

https://twitter.com/taylorfausak/status/811580634045026304

Those numbers make SemVer look less stable than the PVP, but it isn't a fair comparison. I need to update the script to look at the first two components of packages that use the PVP and compare that to the first component of packages that use SemVer.

Even as they are now, the numbers support a point I want to make: The PVP gives unnecessary freedom that most packages don't even use.

[tfausak]

Another thing I want to address is which major version packages are using. bytestring is on 0, aeson is on 1, base is on 4, and HTTP is on 4000.

[alanz]

There is another side to the preferences between SemVer and PVP. The hackage/cabal solver is explicitly designed to work well with the PVP, meaning your are more likely to get a viable build plan if more packages use PVP.

This is especially important if your use case does not fit squarely into a stack snapshot, if e.g. you are required to use anything outside of the snapshot.

[tfausak]

I updated the Gist to include information about the major version numbers.

Current major version => number of packages
0 => 8585
1 => 1411
2 => 270
3 => 130
(Number of version components, current major version) => number of packages
(3,0) => 3199
(3,1) => 700
(3,2) => 130
(3,3) => 59
(4,0) => 4037
(4,1) => 415
(4,2) => 91
(4,3) => 47

Out of 10,643 packages, 8,585 are on major version 0. That's 81%. For the 4,639 PVP packages, 4,037 or 87% are on 0. For the 4,255 SemVer packages, 3,199 or 75% are on 0.

[tfausak]

The hackage/cabal solver is explicitly designed to work well with the PVP, meaning your are more likely to get a viable build plan if more packages use PVP.

I've heard that elsewhere too. Can you help me understand how the solver works well with the PVP? I am not familiar with how Cabal's solver works.

[ezyang]

It doesn't really have anything to do with the solver; it's just that people are encouraged to write their version bounds in the form of >= x.y && < x'.y'. The fact that two components are given means that it is following PVP, not semver.

Maybe a good test for this is to find out, out of all upper bounds in build-depends, which ones specify two components, and which ones specify only one. Ones that specify two components are probably not following semver, because semver says that changes to 'y' are not breaking. (Although, now that I think about it, I feel like in Ruby I often see foo ~= 2.3 where the minver is fixed. So... they're not actually following semver?!)

[alanz]

I am not an expert, but I understand the core of it is that if the first two version numbers stay the same, then the API stays the same. So any dependency having the same first 2 digits should be usable.

Unfortunately this aspect is not fully automatic at present, so there us a need to manually specify version bounds on each dependency to constrain it to the set promised by adherence to PVP.

There is currently a discussion process around making this more automatic at haskell/ecosystem-proposals#1

[tfausak]

I see.

If you were really following the PVP, wouldn't your bounds need to be >= a.b.c && < a'.b'? You need at least some functionality provided by the minor version c and you know that you'll keep working until the major version changes to a'.b'.

Checking upper bounds would be a good way to do. I have two problems with it, both of which are minor: (1) Parsing Cabal files is going to be slower than checking path segments in a tar file; and (2) I use --pvp-bounds=both when releasing packages even though I use SemVer. I could see which bounds other packages put on some package, but that assumes the bounds are accurate.

[gbaz]

Most of the packages on Hackage never change their most significant version number. Only 9% have.

I would also bet most packages on hackage have only had relatively few releases. You'll have a long tail of packages that only see a few releases compared to those that make it to "long lived, stable".

As discussed elsewhere, looking at version number components as you do doesn't tell you anything about semver vs. pvp -- many packages using pvp only use three components. So these numbers are not very much use at all. If Haskell is indeed nearly entirely standardized on pvp (and I would contend that it is), then moving to a slightly different system seems like it would induce an enormous amount of pain and confusion for little-to-no material benefit.

[ezyang]

Yes. In practice, people often write >= a.b because they only use features that were available in a.b.0, but it would be strictly more accurate to always put the minor version in. But it causes errors so you see people often adding that minor version number in to fix things. Lower bounds don't get exercised as much because the depsolvers prefer later versions.

[tfausak]

I updated the Gist to include the number of releases as well.

Number of releases => number of packages
1 => 2168
2 => 1716
3 => 1359
4 => 945
5 => 688
6 => 532
7 => 448
8 => 341
9 => 295

80% of packages have fewer than 10 releases.

[tfausak]

After reading replies here, on Reddit, and on Twitter, it has become apparent that I must be very careful in how I interpret this data. Saying "3 version parts means SemVer" and "4 version parts means PVP" is inaccurate. It's true that SemVer requires exactly 3 version parts, but the PVP only specifies at least 3 version parts. Also some packages that do follow the PVP don't always use the same number of parts; look at lens for an example.

At this point, the things I can say about the data are pretty boring:

• Most packages have 3 or 4 version parts. It's about an even split.
• Most packages have never changed their major version number.
• If the package has 4 version parts, it's less likely to change the most significant part (compared to 3 parts).
• Most packages are on major version 0.
• If the package has 4 version parts, it's more likely to be on major version 0 (compared to 3 parts).
• Most packages have fewer than 10 releases.

[rwbarton]

Of the 10,642 #Haskell packages on Hackage, only 921 have ever changed their most significant version number.

Without consulting any statistics, what proportion of packages following SemVer would you expect to have never made any incompatible API changes?

(Maybe it would be best to ignore packages with fewer than, say, three released versions.)

[tfausak]

I've already linked all the interesting tweets. Here's the Reddit thread about them: https://www.reddit.com/r/haskell/comments/5jlgxe/of_the_10642_haskell_packages_on_hackage_4638_use/

Since this post deals with version numbers, invariably someone will bring up this recent Rich Hickey video: https://www.youtube.com/watch?v=oyLBGkS5ICk

what proportion of packages following SemVer would you expect to have never made any incompatible API changes?

I would expect a very small number of SemVer packages to be on a major version above 1. Pulling a number out of a hat, I'd say I expect... 2% to have ever made a breaking change after reaching version 1.0.0.

[tfausak]

It's true that SemVer requires exactly 3 version parts, but the PVP only specifies at least 3 version parts.

In fact, the PVP only recommends at least 3 parts. This has some interesting consequences:

• It is valid per the PVP to have a version 1.
• It is also valid per the PVP to signal breaking changes by going from version 1 to 1.0. That's because 1 is less than 1.0 according to Data.Version.
  • Hackage recently added a check to prevent users from uploading package versions like this. See Add normisedPackageIdExists predicate, and use it in upload haskell/hackage-server#518 and Min version cut versions haskell/cabal#3515 (comment).
  • This can also wreak havoc with version bounds since >= A.B and >= A.B.0 don't mean the same thing. This affects == A.B.* constructs too since they mean >= A.B && < A.(B+1). See cryptonite 0.21 and memory 0.14 commercialhaskell/stackage#2143 (comment).
• It's valid to use a different number of parts for each release. This is basically what the lens package does. By convention the missing components are assumed to be 0, but that's not exactly true.

SemVer avoids these shenanigans by requiring exactly 3 parts.