Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: GQL responses #2872

Merged
merged 18 commits into from
Jul 30, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 1 addition & 9 deletions internal/db/collection_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -259,15 +259,7 @@ func (c *collection) makeSelectionPlan(
txn,
)

return planner.MakePlan(&request.Request{
Queries: []*request.OperationDefinition{
{
Selections: []request.Selection{
slct,
},
},
},
})
return planner.MakeSelectionPlan(slct)
}

func (c *collection) makeSelectLocal(filter immutable.Option[request.Filter]) (*request.Select, error) {
Expand Down
6 changes: 3 additions & 3 deletions internal/db/request.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,10 @@ func (db *db) execRequest(ctx context.Context, request string) *client.RequestRe
results, err := planner.RunRequest(ctx, parsedRequest)
if err != nil {
res.GQL.Errors = []error{err}
return res
}

res.GQL.Data = results
if len(results) > 0 {
res.GQL.Data = results[0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question/suggestion: Why is results still an array? Can we return just a map[string]any now instead?

Same question/suggestion for RunSelection.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I've updated both to return maps.

}
return res
}

Expand Down
2 changes: 1 addition & 1 deletion internal/db/subscriptions.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func (db *db) handleSubscription(ctx context.Context, r *request.Request) (<-cha
p := planner.New(ctx, identity, db.acp, db, txn)
s := subRequest.ToSelect(evt.DocID, evt.Cid.String())

result, err := p.RunSubscriptionRequest(ctx, s)
result, err := p.RunSelection(ctx, s)
if err == nil && len(result) == 0 {
txn.Discard(ctx)
continue // Don't send anything back to the client if the request yields an empty dataset.
Expand Down
101 changes: 97 additions & 4 deletions internal/planner/mapper/mapper.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,75 @@ const (
CommitSelection
)

// ToSelect converts the given [parser.Select] into a [Select].
// ToOperation converts the given [request.OperationDefinition] into an [Operation].
//
// In the process of doing so it will construct the document map required to access the data
// yielded by the [Operation].
func ToOperation(
ctx context.Context,
store client.Store,
operationRequest *request.OperationDefinition,
) (*Operation, error) {
operation := &Operation{
DocumentMapping: core.NewDocumentMapping(),
}

for i, s := range operationRequest.Selections {
switch t := s.(type) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The default case is not handled, it might be better to return an error instead of quietly ignoring it - I believe it could only happen if we mess up in the future, but it could save some head-scratching even if it means a line or two of dead code.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a default case with an error.

case *request.CommitSelect:
s, err := toCommitSelect(ctx, store, t, i)
if err != nil {
return nil, err
}
renderKey := core.RenderKey{Index: i}
if t.Alias.HasValue() {
renderKey.Key = t.Alias.Value()
} else {
renderKey.Key = t.Name
}
operation.CommitSelects = append(operation.CommitSelects, s)
operation.DocumentMapping.Add(i, s.Name)
operation.DocumentMapping.SetChildAt(i, s.DocumentMapping)
operation.DocumentMapping.RenderKeys = append(operation.DocumentMapping.RenderKeys, renderKey)

case *request.Select:
s, err := toSelect(ctx, store, ObjectSelection, i, t, "")
if err != nil {
return nil, err
}
renderKey := core.RenderKey{Index: i}
if t.Alias.HasValue() {
renderKey.Key = t.Alias.Value()
} else {
renderKey.Key = t.Name
}
operation.Selects = append(operation.Selects, s)
operation.DocumentMapping.Add(i, s.Name)
operation.DocumentMapping.SetChildAt(i, s.DocumentMapping)
operation.DocumentMapping.RenderKeys = append(operation.DocumentMapping.RenderKeys, renderKey)

case *request.ObjectMutation:
m, err := toMutation(ctx, store, t, i)
if err != nil {
return nil, err
}
renderKey := core.RenderKey{Index: i}
if t.Alias.HasValue() {
renderKey.Key = t.Alias.Value()
} else {
renderKey.Key = t.Name
}
operation.Mutations = append(operation.Mutations, m)
operation.DocumentMapping.Add(i, m.Name)
operation.DocumentMapping.SetChildAt(i, m.DocumentMapping)
operation.DocumentMapping.RenderKeys = append(operation.DocumentMapping.RenderKeys, renderKey)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: the bodies of all 3 cases are very similar. Is there a way to extract common functionality?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved some of this to a function and reduced most of the duplicate lines

}
}

return operation, nil
}

// ToSelect converts the given [request.Select] into a [Select].
//
// In the process of doing so it will construct the document map required to access the data
// yielded by the [Select].
Expand Down Expand Up @@ -1142,7 +1210,20 @@ func ToCommitSelect(
store client.Store,
selectRequest *request.CommitSelect,
) (*CommitSelect, error) {
underlyingSelect, err := ToSelect(ctx, store, CommitSelection, selectRequest.ToSelect())
return toCommitSelect(ctx, store, selectRequest, 0)
}

// toCommitSelect converts the given [request.CommitSelect] into a [CommitSelect].
//
// In the process of doing so it will construct the document map required to access the data
// yielded by the [Select] embedded in the [CommitSelect].
func toCommitSelect(
ctx context.Context,
store client.Store,
selectRequest *request.CommitSelect,
thisIndex int,
) (*CommitSelect, error) {
underlyingSelect, err := toSelect(ctx, store, CommitSelection, thisIndex, selectRequest.ToSelect(), "")
if err != nil {
return nil, err
}
Expand All @@ -1160,11 +1241,23 @@ func ToCommitSelect(
// In the process of doing so it will construct the document map required to access the data
// yielded by the [Select] embedded in the [Mutation].
func ToMutation(ctx context.Context, store client.Store, mutationRequest *request.ObjectMutation) (*Mutation, error) {
underlyingSelect, err := ToSelect(ctx, store, ObjectSelection, mutationRequest.ToSelect())
return toMutation(ctx, store, mutationRequest, 0)
}

// toMutation converts the given [request.Mutation] into a [Mutation].
//
// In the process of doing so it will construct the document map required to access the data
// yielded by the [Select] embedded in the [Mutation].
func toMutation(
ctx context.Context,
store client.Store,
mutationRequest *request.ObjectMutation,
thisIndex int,
) (*Mutation, error) {
underlyingSelect, err := toSelect(ctx, store, ObjectSelection, thisIndex, mutationRequest.ToSelect(), "")
if err != nil {
return nil, err
}

return &Mutation{
Select: *underlyingSelect,
Type: MutationType(mutationRequest.Type),
Expand Down
31 changes: 31 additions & 0 deletions internal/planner/mapper/operation.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// Copyright 2024 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package mapper

import "github.com/sourcenetwork/defradb/internal/core"

// Operation represents an operation such as query or mutation.
//
// It wraps child Selects belonging to this operation.
type Operation struct {
// The document mapping for this select, describing how items yielded
// for this select can be accessed and rendered.
*core.DocumentMapping

// Selects is the list of selections in the operation.
Selects []*Select

// Mutations is the list of mutations in the operation.
Mutations []*Mutation

// CommitSelects is the list of commit selections in the operation.
CommitSelects []*CommitSelect
}
178 changes: 178 additions & 0 deletions internal/planner/operation.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
// Copyright 2024 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package planner

import (
"github.com/sourcenetwork/defradb/client/request"
"github.com/sourcenetwork/defradb/internal/core"
"github.com/sourcenetwork/defradb/internal/planner/mapper"
)

const operationNodeKind string = "operationNode"

// operationNode is the top level node for operations with
// one or more child selections, such as queries or mutations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: I understand what you mean by operation but wonder if there is a better name for this node (I don't have any good ideas atm), the reason I say this is because every node is essentially an "instruction/operation step" so it seems a bit vague.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the name could be improved. It should be fairly easy to rename if we find a better name.

type operationNode struct {
documentIterator
docMapper

children []planNode
isDone bool
}

func (n *operationNode) Spans(spans core.Spans) {
for _, child := range n.children {
child.Spans(spans)
}
}

func (n *operationNode) Kind() string {
return operationNodeKind
}

func (n *operationNode) Init() error {
n.isDone = false
for _, child := range n.children {
err := child.Init()
if err != nil {
return err
}
}
return nil
}

func (n *operationNode) Start() error {
for _, child := range n.children {
err := child.Start()
if err != nil {
return err
}
}
return nil
}

func (n *operationNode) Close() error {
for _, child := range n.children {
err := child.Close()
if err != nil {
return err
}
}
return nil
}

func (n *operationNode) Source() planNode {
return nil
}

func (p *operationNode) Children() []planNode {
return p.children
}

func (n *operationNode) Explain(explainType request.ExplainType) (map[string]any, error) {
switch explainType {
case request.SimpleExplain:
return map[string]any{}, nil

case request.ExecuteExplain:
return map[string]any{}, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: is it supposed to be empty?

Did you add explain tests that assert this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super familiar with the planner package. Is this supposed to return the operations that this node executes? It does appear in the existing explain tests correctly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you can omit the Explain method entirely, since its an optional interface. I don't know if there is any context that this planNode can add to the existing plan, so simply not including it in the explain system is an option.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @jsimnz pointed out, making a node explainable is optional. Moreover, MultiNode types are not / don't need to be explainable but we use them to wrap the children under them so the explain graph builder I believe handles that case. But when/if you make operationNode unexplainable don't forget to leave the added line in the tests/integration/explain.go file because the debug explain graph should still need it to assert it properly in the testing framework.

An example of this is parallelNode has no Explain method, yet still shows in the explain graph to wrap the children under it.


default:
return nil, ErrUnknownExplainRequestType
}
}

func (n *operationNode) Next() (bool, error) {
if n.isDone {
return false, nil
}

n.currentValue = n.documentMapping.NewDoc()
for i, child := range n.children {
switch child.(type) {
case *topLevelNode:
hasChild, err := child.Next()
if err != nil {
return false, err
}
if !hasChild {
return false, ErrMissingChildValue
}
n.currentValue = child.Value()

default:
var docs []core.Doc
for {
hasChild, err := child.Next()
if err != nil {
return false, err
}
if !hasChild {
break
}
docs = append(docs, child.Value())
}
n.currentValue.Fields[i] = docs
}
}

n.isDone = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: do you really need this flag? It's being set only in Init(). Next() is not supposed to be called anyway before Init() and it's not multi-threaded.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried removing it, but the return value was always incorrect. I believe we need it so that the operation is able to run at least once.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick/todo: The planNode interface and it's consuption dictates that Init can be called multiple times, and that it's state should be reset by the first call (before doing it's thing again). I can't think why this node would have an instance Init'ed multiple times, but it is kind of incorrect atm, and just last week we had a production issue caused by this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed now. I'm assuming you were referencing the currentValue not being reset.

return true, nil
}

// Operation creates a new operationNode using the given Selects.
func (p *Planner) Operation(operation *mapper.Operation) (*operationNode, error) {
var children []planNode

for _, s := range operation.Selects {
if _, isAgg := request.Aggregates[s.Name]; isAgg {
// If this Select is an aggregate, then it must be a top-level
// aggregate and we need to resolve it within the context of a
// top-level node.
child, err := p.Top(s)
if err != nil {
return nil, err
}
children = append(children, child)
} else {
child, err := p.Select(s)
if err != nil {
return nil, err
}
children = append(children, child)
}
}

for _, m := range operation.Mutations {
child, err := p.newObjectMutationPlan(m)
if err != nil {
return nil, err
}
children = append(children, child)
}

for _, s := range operation.CommitSelects {
child, err := p.CommitSelect(s)
if err != nil {
return nil, err
}
children = append(children, child)
}

if len(children) == 0 {
return nil, ErrOperationDefinitionMissingSelection
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Is this spec-compliant? A selection set must always be provided?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see anything in the spec about empty selections. I've removed this error since it should be fine to return empty results.

}

return &operationNode{
docMapper: docMapper{operation.DocumentMapping},
children: children,
}, nil
}
2 changes: 2 additions & 0 deletions internal/planner/operations.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ var (
_ planNode = (*valuesNode)(nil)
_ planNode = (*viewNode)(nil)
_ planNode = (*lensNode)(nil)
_ planNode = (*operationNode)(nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: MultiNode implements/extends planNode, so this line is not required.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


_ MultiNode = (*parallelNode)(nil)
_ MultiNode = (*topLevelNode)(nil)
_ MultiNode = (*operationNode)(nil)
)

// type joinNode struct {
Expand Down
Loading