Feature/federated search #888
Draft
+16,102
−38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
* update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <[email protected]>
* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
* add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
* move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <[email protected]>
* move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <[email protected]>
* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <[email protected]>
justincorrigible
force-pushed
the
develop
branch
3 times, most recently
from
6b33e90
to
7f882d1
Compare
* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]>
* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <[email protected]>
* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <[email protected]>
* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Ciaran Schutte <[email protected]>
* use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <[email protected]>
* add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
* improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <[email protected]>
* remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <[email protected]>
* remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.