Pageable<T> KeyVault Certificate GetPropertiesOfCertificates does not honor the includePending optional parameter after the first page

[ahsonkhan]

The call to fetch the first page sets the appropriate query parameters based on the input parameter value:

azure-sdk-for-net/sdk/keyvault/Azure.Security.KeyVault.Certificates/src/CertificateClient.cs

Lines 942 to 947 in caede08

	public virtual Pageable<CertificateProperties> GetPropertiesOfCertificates(bool includePending = default, CancellationToken cancellationToken = default)
	{
	Uri firstPageUri = _pipeline.CreateFirstPageUri(CertificatesPath, ("includePending", includePending.ToString(CultureInfo.InvariantCulture).ToLowerInvariant()));
	return PageResponseEnumerator.CreateEnumerable(nextLink => _pipeline.GetPage(firstPageUri, nextLink, () => new CertificateProperties(), "Azure.Security.KeyVault.Keys.KeyClient.GetPropertiesOfCertificates", cancellationToken));
	}

But subsequent pages do not have that value set. That means, if the includePending param is set to true, it will not return all the certificates (including pending ones), if the pending certificate happens to be listed in a page other than the first.

Here's the swagger (not sure if this requires some fix to the swagger):
https://github.com/Azure/azure-rest-api-specs/blob/4a4acecea9901c29e19ba50f2d4cf65b20115b69/specification/keyvault/data-plane/Microsoft.KeyVault/stable/7.5/certificates.json#L30-L83

Sample repro:

using Azure.Identity;
using Azure.Security.KeyVault.Certificates;

// Pre-req: Create 25 certificates first so a page is full (either through the portal or programmatically) 

// TODO: Set to your own KeyVault URL
string keyVaultUrl = "https://<keyvault-name>.vault.azure.net/";

CertificateClient client = new CertificateClient(new Uri(keyVaultUrl), new DefaultAzureCredential());

// Case 1: Certificate gets created on the first page, works as expected.
string certNameFirstPage = $"aaa-{Guid.NewGuid()}";
CertificateOperation certOp1 = client.StartCreateCertificate(certNameFirstPage, CertificatePolicy.Default);

FetchCertificates();

// Wait about ~30 seconds for the certificate to be created before moving to case 2.
while (!certOp1.HasCompleted)
{
    certOp1.UpdateStatus();
    Thread.Sleep(TimeSpan.FromSeconds(1));
}

// Case 2: Certificate gets created on any other subsequent page, doesn't work as expected.
string certNameLastPage = $"zzz-{Guid.NewGuid()}";
CertificateOperation certOp2 = client.StartCreateCertificate(certNameLastPage, CertificatePolicy.Default);

FetchCertificates();

void FetchCertificates()
{
    int countFalse = 0;
    Console.WriteLine("Certificates in the key vault (includePending = false):");
    foreach (CertificateProperties cert in client.GetPropertiesOfCertificates())
    {
        Console.WriteLine($"{cert.Name}");
        countFalse++;
    }

    int countTrue = 0;
    Console.WriteLine("Certificates in the key vault (includePending = true):");
    foreach (CertificateProperties cert in client.GetPropertiesOfCertificates(true))
    {
        Console.WriteLine($"{cert.Name}");
        countTrue++;
    }

    // Expected countFalse < countTrue in both cases, since there's a certificate pending.
    // Case 1: In the case where the certificate gets created on the first page:
    //  -> countFalse < countTrue
    // Case 2: But, in the case where the certificate gets created on any other subsequent page:
    //  -> countFalse = countTrue
    Console.WriteLine($"countFalse = {countFalse} vs countTrue = {countTrue}");
}

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Azure.Identity" Version="1.13.1" />
    <PackageReference Include="Azure.Security.KeyVault.Certificates" Version="4.7.0" />
  </ItemGroup>

</Project>

dotnet --version
8.0.404

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64

The issue is pervasive across all the Pageable<T> methods that follow this pattern within the KeyVault SDKs, but GetPropertiesOfCertificates and GetDeletedCertificates seem to be the only ones that have optional parameters which are settable by the SDK methods (unlike maxResults) and hence have an actual behavioral bug here.

It's possible that some other service SDKs have similar concerns here, but newer SDKs Pageable<T> pattern, such as GetRoleAssignments in Azure.Security.KeyVault.Administration sets the filter parameter appropriately to both the first and subsequent pages:

azure-sdk-for-net/sdk/keyvault/Azure.Security.KeyVault.Administration/src/KeyVaultAccessControlClient.cs

Lines 342 to 370 in caede08

	return PageableHelpers.CreateEnumerable(_ =>
	{
	using DiagnosticScope scope = _diagnostics.CreateScope($"{nameof(KeyVaultAccessControlClient)}.{nameof(GetRoleAssignments)}");
	scope.Start();
	try
	{
	var response = _assignmentsRestClient.ListForScope(vaultBaseUrl: VaultUri.AbsoluteUri, scope: roleScope.ToString(), cancellationToken: cancellationToken);
	return Page.FromValues(response.Value.Value, response.Value.NextLink, response.GetRawResponse());
	}
	catch (Exception ex)
	{
	scope.Failed(ex);
	throw;
	}
	}, (nextLink, _) =>
	{
	using DiagnosticScope scope = _diagnostics.CreateScope($"{nameof(KeyVaultAccessControlClient)}.{nameof(GetRoleAssignments)}");
	scope.Start();
	try
	{
	var response = _assignmentsRestClient.ListForScopeNextPage(nextLink: nextLink, vaultBaseUrl: VaultUri.AbsoluteUri, scope: roleScope.ToString(), cancellationToken: cancellationToken);
	return Page.FromValues(response.Value.Value, response.Value.NextLink, response.GetRawResponse());
	}
	catch (Exception ex)
	{
	scope.Failed(ex);
	throw;
	}
	});

Related issues in other languages:
#47202
Azure/azure-sdk-for-cpp#6235
Azure/azure-sdk-for-python#38589
Azure/azure-sdk-for-go#23772
Azure/azure-sdk-for-js#31803
Azure/azure-sdk-for-java#42988