Add test to verify query response compression (#6849)

[siddarth2810]

What this PR does:
Adds an integration test which checks the Content-Encoding header. This ensures that query response compression works correctly

Which issue(s) this PR fixes:
Fixes #6849

Checklist

• Tests updated
• [NA] Documentation added
• [NA] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Overall I think this is on the right direction. Thanks for the PR!

[yeya24]

We don't need store gateway in this test so let's remove it.

[siddarth2810]

I see, got it.

[yeya24]

The current tests are good. But I'd like to see we have test cases covering queries against query frontend as well. Not just querier

[siddarth2810]

Oh alright, I'l add a new endpoint with query frontend and update the tests. Thanks a lot for the review !

[siddarth2810]

Hey @yeya24 , after tweaking around with query frontend for a while, I realized that testing both query frontend and querier cannot be done with same instance of cortex. So I divided the code into two isolated functions.

Is this what you had in mind ?

1. TestQuerierDirectResponseCompression - tests compression directly using querier endpoint
2. TestQueryFrontendResponseCompression - tests compression via query frontend endpoint

The query frontend test is failing because the response is not being compressed

Error: Not equal:
expected: "gzip"
actual  : ""
Test: TestQueryFrontendResponseCompression/Compressed

func TestQueryFrontendResponseCompression(t *testing.T) {
	s, err := e2e.NewScenario(networkName)
	require.NoError(t, err)
	defer s.Close()

	// Start dependencies.
	consul := e2edb.NewConsul()
	minio := e2edb.NewMinio(9000, bucketName)
	require.NoError(t, s.StartAndWaitReady(consul, minio))

	flags := mergeFlags(BlocksStorageFlags(), map[string]string{
		"-api.response-compression-enabled": "true",
	})

	// Start the query-frontend.
	queryFrontend := e2ecortex.NewQueryFrontend("query-frontend", flags, "")
	require.NoError(t, s.Start(queryFrontend))

	distributor := e2ecortex.NewDistributor("distributor", e2ecortex.RingStoreConsul, consul.NetworkHTTPEndpoint(), flags, "")
	ingester := e2ecortex.NewIngester("ingester", e2ecortex.RingStoreConsul, consul.NetworkHTTPEndpoint(), flags, "")
	require.NoError(t, s.StartAndWaitReady(distributor, ingester))

	// Wait until both the distributor updated the ring.
	require.NoError(t, distributor.WaitSumMetrics(e2e.Equals(512), "cortex_ring_tokens_total"))

	querier := e2ecortex.NewQuerier("querierWithFrontend", e2ecortex.RingStoreConsul, consul.NetworkHTTPEndpoint(), mergeFlags(flags, map[string]string{
		"-querier.frontend-address": queryFrontend.NetworkGRPCEndpoint(),
	}), "")

	require.NoError(t, s.StartAndWaitReady(querier))
	require.NoError(t, s.WaitReady(queryFrontend))

	now := time.Now()
	series, _ := generateSeries("series_1", now)

	c, err := e2ecortex.NewClient(distributor.HTTPEndpoint(), queryFrontend.HTTPEndpoint(), "", "", "user-1")
	require.NoError(t, err)

	res, err := c.Push(series)
	require.NoError(t, err)
	require.Equal(t, 200, res.StatusCode)

	require.NoError(t, querier.WaitSumMetrics(e2e.Equals(512), "cortex_ring_tokens_total"))

	endpoint := fmt.Sprintf("http://%s/api/prom/api/v1/query?query=series_1", queryFrontend.HTTPEndpoint())

	runQueryResponseTest(t, endpoint) //func to run existing compressed and uncompressed logic
}

Been stuck on this for a while, Is this expected or am I totally mistaken ? Would be really really thankful for any inputs, thanks !!