Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cached get() results to limit calls to get() #346

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

amandaharth
Copy link

Summary

There may be reasons this is the way it is (I am still wrapping my head around the Puppet Resource API) - but I am wondering if we can get rid of the names.nil? checks in the rsapi_provider_get function.

Please feel free to educate me if there is something I've misunderstood or not considered.

In my testing, with simple_get_filter not implemented and operating in a Puppet Agent and Master Server environment, removing the names.nil? checks stopped get(context) from being called again for every resource which had pending changes. All resources of the given type were collected from the system with one get() call during the agent run and cached, with subsequent calls to rsapi_provider_get using the cached information. To me this seems like it would be the desired behaviour.

Additional Context

The behaviour I observed before removing the names.nil? checks:

  1. The cached data is returned if the cache has been marked as complete and names is nil. E.g. after the initial get(context) has been done, when retrieving current state for resources which are defined in the catalogue but have no pending changes.
  2. Therefore, if names is not nil or if the cache hasn't been marked complete, information about all resources of the given type is fetched by calling the provider's get(context) function.
  3. The fetched information for all resources of the given type is added to the cache.
  4. The cache is marked complete if names is nil and simple_get_feature is not implemented.
  5. Subsequent calls to rsapi_provider_get (e.g. to retrieve the current state of a resource which has pending changes) pass in a value for 'names', and therefore names.nil? is false, and the cache in point 1 above isn't returned or used. get() is therefore called again per resource with pending changes, retrieving all resource information of that type again each time.

Removing the names.nil? checks allows the cache to be populated with information about all of the resources of the given type with one get() call, mark the cache as complete, and allow the cache to be used in subsequent calls to rsapi_provider_get for each resource, whether they have pending changes or not.

Simple_get_filter behaviour shouldn't be affected by this change, as when simple_get_filter is implemented the cache would never be marked as complete or returned and my_provider.get(context, names) would still be called every time.

Testing and observations

Before the change:

  • Defined 60+ resources of the same type in the catalogue
  • Manually made changes to 5 of those resources, to trigger Puppet to revert changes on those 5 resources
  • Observed 5 calls to get(context); once per resource with pending changes
  • Observed current state for the other defined (but unchanged) resources was retrieved from cache

After the change:

  • Defined 60+ resources of the same type in the catalogue
  • Manually made changes to 5 of those resources, to trigger Puppet to revert changes on those 5 resources
  • Observed 1 call to get(context)
  • Current state for all defined resources (whether unchanged or with changes pending) was retrieved from cache

Related Issues (if any)

Related to get() call optimisation and caching.

Checklist

  • 🟢 Spec tests.
  • 🟢 Acceptance tests.
  • Manually verified.

@amandaharth amandaharth requested review from a team as code owners April 12, 2024 06:58
@CLAassistant
Copy link

CLAassistant commented Apr 12, 2024

CLA assistant check
All committers have signed the CLA.

@joshcooper
Copy link
Contributor

Could you rebase your PR instead of pulling (and creating merge commits in your PR)

My understanding... there may be cases I haven't considered.

In my testing, with simple_get_filter not implemented, removing the
names.nil? checks stopped get(context) being called again for every resource
which had pending changes. All resources of the given type were
collected from the system with one get() call and cached. Subsequent
calls to rsapi_provider_get returned the cached information.

When simple_get_filter is not implemented:
1. The cached data is returned, if the cache has been marked as complete and names is nil.
2. Therefore, if names is NOT nil or if the cache hasn't been marked complete, information about all resources of the given type is fetched by calling the provider's get() function.
3. The fetched information for all relevant resources is added to the cache.
4. The cache is marked complete if names is nil and simple_get_feature is not implemented.
5. Subsequent calls to rsapi_provider_get (e.g. to retrieve the current state of a resource which has pending changes) pass in a value for 'names', and therefore names.nil? is false, and the cache in point 1 above isn't returned or used. get() is therefore called again per resource to retrieve all resource information again.

Removing the names.nil? checks on lines 255 and 268 allows the cache to be populated with information about all of the resources of the given type with one get() call, mark the cache as complete, and therefore allow the cache to be used in subsequent calls to rsapi_provider_get for each resource.

Simple_get_filter behaviour wouldn't change, as when simple_get_filter is implemented the cache would never be marked as complete or returned: my_provider.get(context, names) would still be called every time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: TODO
Development

Successfully merging this pull request may close these issues.

4 participants