Skip to content

Commit 74e3aa7

Browse files
perf(GraphQL): Generate GraphQL query response by optimized JSON encoding (GraphQL-730) (#7371)
* perf(GraphQL): JSON Part-1: JSON is generated by Dgraph for simple queries (GRAPHQL-877) (#7230) This PR moves the GraphQL JSON generation for simple queries inside Dgraph. Previously, the GraphQL layer would get a JSON response from Dgraph in the form expected for DQL. That JSON response had to go through the following pipeline: Unmarshal Run GraphQL completion Marshal After this pipeline was executed, then the output JSON would be in the form as expected by GraphQL. Now, the response from Dgraph would already be in the form expected by GraphQL. So, we no more need to run the above pipeline. Hence, saving valuable time and memory. * perf(GraphQL): JSON Part-2: JSON is generated by Dgraph for simple mutations (GRAPHQL-877) (#7237) This PR is a continuation of #7230. It makes standard GraphQL mutations work with the JSON changes. Each mutation now returns the GraphQL JSON required for that mutation as a byte array from the MutationResolver itself, so no further processing of the JSON is required. Delete mutations now query the `queryField` explicitly before executing the mutation, to make it work with the JSON changes. * perf(GraphQL): JSON Part-3: Geo queries (GRAPHQL-879) (#7238) Continued from: * #7230 * #7237 This PR makes Geo queries/mutations work with the new JSON changes. * perf(GraphQL): JSON Part-4: Aggregate queries at root and child levels (GRAPHQL-936) (#7240) Continued from: * #7230 * #7237 * #7238 This PR makes aggregate queries at root and child levels work with the new JSON changes. It removes the default behaviour of rewriting the count field to DQL query everytime (introduced in #7119). Now, rewritten DQL will have count field only if it was requested in GraphQL. * review comments from merged PRs * perf(GraphQL): JSON Part-5: Change query rewriting for scalar/object fields asked multiple times at same level (GRAPHQL-938) (#7283) Previously, for a GraphQL query like this: ``` query { queryThing { i1: id i2: id name n: name n1: name } } ``` GraphQL layer used to generate following DQL query: ``` query { queryThing(func: type(Thing)) { id : uid name : Thing.name } } ``` This can’t work with the new way of JSON encoding for GraphQL because it expects that everything asked for in GraphQL query is present in DQL response. So, Now, the GraphQL layer generates the following DQL to make it work: ``` query { queryThing(func: type(Thing)) { Thing.i1 : uid Thing.i2: uid Thing.name : Thing.name Thing.n : Thing.name Thing.n1 : Thing.name } } ``` Basically, the `DgraphAlias()` is now generated using the `TypeCondition.GraphqlAlias` format. That works even with fragments as they have their specific type conditions. Here are related thoughts: ``` query { queryHuman { name n: name } # DQL: use field.name as DgAlias() # queryHuman(func: type(Human)) { # name: Human.name # name: Human.name # } # => just using field.name doesn't work => use field.alias # queryHuman(func: type(Human)) { # name: Human.name # n: Human.name # } # but there are interfaces and fragments queryCharacter { ... on Human { n: name } ... on Droid { n: bio } } # DQL: use field.alias as DgAlias() # queryCharacter(func: type(Character)) { # n: Character.name # n: Character.bio # } # => just using field.alias doesn't work => use field.name + field.alias # queryCharacter(func: type(Character)) { # name.n: Character.name # bio.n: Character.bio # } # but the implementing types may have fields with same name not inherited from the interface queryThing { ... on ThingOne { c: color } ... on ThingTwo { c: color } } # DQL: use field.name + field.alias as DgAlias() # queryThing(func: type(Thing)) { # color.c: ThingOne.color # color.c: ThingTwo.color # } # => even that doesn't work => use Typename + field.name + field.alias # queryThing(func: type(Thing)) { # ThingOne.color.c: ThingOne.color # ThingTwo.color.c: ThingTwo.color # } # nice! but then different frags explicitly ask for an inherited field with same name & alias queryCharacter { ... on Human { n: name } ... on Droid { n: name } } # DQL: use Typename + field.name + field.alias as DgAlias() # queryCharacter(func: type(Character)) { # Character.name.n: Character.name # Character.name.n: Character.name # } # => doesn't work => use typeCond + Typename + field.name + field.alias # queryCharacter(func: type(Character)) { # Human.Character.name.n: Character.name # Droid.Character.name.n: Character.name # } # but wait, wouldn't just typeCond + field.alias work? # queryCharacter(func: type(Character)) { # Human.n: Character.name # Droid.n: Character.name # } # yeah! that works even for all the above cases. # # OR # # just appending the count when encountering duplicate alias at the same level also works. # But, we need to maintain the count map every time we need DgAlias(). # Also, this requires query pre-processing and the generated aliases are non-deterministic. # # So, we are going with the typeCond + field.alias approach. That will work with @Custom(dql:...) too. } ``` * perf(GraphQL): JSON Part-6: Admin Server and custom HTTP query/mutation are now resolved separately (GRAPHQL-937) (#7317) This PR: * makes the admin server on `/admin` and `@custom(http: {...})` Query/Mutation work through a common pipeline separate from the queries/mutations resolved through Dgraph * adds scalar coercion to queries resolved through Dgraph * and, removes completion as a step from resolvers, as it is not needed anymore. * perf(GraphQL): JSON Part-7: JSON is generated by Dgraph for custom HTTP fields (GRAPHQL-878) (#7361) This PR moves the resolution and GraphQL JSON generation for `@custom(http: {...})` fields inside Dgraph. It also fixes following existing bugs with `@custom` field resolution: * https://discuss.dgraph.io/t/slash-graphql-lambda-bug/12233 (GRAPHQL-967) * race condition in custom logic with repeated fields in implementing types (GRAPHQL-860) * GRAPHQL-639 * perf(GraphQL): JSON part-8: Cleanup and comments (GRAPHQL-947) (#7376) This is a cleanup PR for the JSON changes. It is mostly about nit-picks, making some old tests work, skipping some tests to be ported later to e2e and making the CI green overall. It also fixes a [Discuss Issue](https://discuss.dgraph.io/t/custom-field-resolvers-are-not-always-called/12489) (GRAPHQL-989). * fix merge * review comments * fix tests * fix more tests
1 parent 5bf00f5 commit 74e3aa7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

71 files changed

+4840
-3708
lines changed

dgraph/cmd/alpha/run_test.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ func processToFastJSON(q string) string {
8080
log.Fatal(err)
8181
}
8282

83-
buf, err := query.ToJson(&l, qr.Subgraphs)
83+
buf, err := query.ToJson(context.Background(), &l, qr.Subgraphs, nil)
8484
if err != nil {
8585
log.Fatal(err)
8686
}

edgraph/access_ee.go

+38-26
Original file line numberDiff line numberDiff line change
@@ -288,12 +288,15 @@ func authorizeUser(ctx context.Context, userid string, password string) (
288288
"$userid": userid,
289289
"$password": password,
290290
}
291-
queryRequest := api.Request{
292-
Query: queryUser,
293-
Vars: queryVars,
291+
req := &Request{
292+
req: &api.Request{
293+
Query: queryUser,
294+
Vars: queryVars,
295+
},
296+
doAuth: NoAuthorize,
294297
}
295298

296-
queryResp, err := (&Server{}).doQuery(ctx, &queryRequest, NoAuthorize)
299+
queryResp, err := (&Server{}).doQuery(ctx, req)
297300
if err != nil {
298301
glog.Errorf("Error while query user with id %s: %v", userid, err)
299302
return nil, err
@@ -326,13 +329,16 @@ func RefreshAcls(closer *z.Closer) {
326329
maxRefreshTs = refreshTs
327330

328331
glog.V(3).Infof("Refreshing ACLs")
329-
queryRequest := api.Request{
330-
Query: queryAcls,
331-
ReadOnly: true,
332-
StartTs: refreshTs,
332+
req := &Request{
333+
req: &api.Request{
334+
Query: queryAcls,
335+
ReadOnly: true,
336+
StartTs: refreshTs,
337+
},
338+
doAuth: NoAuthorize,
333339
}
334340

335-
queryResp, err := (&Server{}).doQuery(closer.Ctx(), &queryRequest, NoAuthorize)
341+
queryResp, err := (&Server{}).doQuery(closer.Ctx(), req)
336342
if err != nil {
337343
return errors.Errorf("unable to retrieve acls: %v", err)
338344
}
@@ -406,18 +412,21 @@ func ResetAcl(closer *z.Closer) {
406412
}
407413
`, x.GuardiansId)
408414
groupNQuads := acl.CreateGroupNQuads(x.GuardiansId)
409-
req := &api.Request{
410-
CommitNow: true,
411-
Query: query,
412-
Mutations: []*api.Mutation{
413-
{
414-
Set: groupNQuads,
415-
Cond: "@if(eq(len(guid), 0))",
415+
req := &Request{
416+
req: &api.Request{
417+
CommitNow: true,
418+
Query: query,
419+
Mutations: []*api.Mutation{
420+
{
421+
Set: groupNQuads,
422+
Cond: "@if(eq(len(guid), 0))",
423+
},
416424
},
417425
},
426+
doAuth: NoAuthorize,
418427
}
419428

420-
resp, err := (&Server{}).doQuery(ctx, req, NoAuthorize)
429+
resp, err := (&Server{}).doQuery(ctx, req)
421430

422431
// Structs to parse guardians group uid from query response
423432
type groupNode struct {
@@ -474,19 +483,22 @@ func ResetAcl(closer *z.Closer) {
474483
Predicate: "dgraph.user.group",
475484
ObjectId: "uid(guid)",
476485
})
477-
req := &api.Request{
478-
CommitNow: true,
479-
Query: query,
480-
Mutations: []*api.Mutation{
481-
{
482-
Set: userNQuads,
483-
// Assuming that if groot exists, it is in guardian group
484-
Cond: "@if(eq(len(grootid), 0) and gt(len(guid), 0))",
486+
req := &Request{
487+
req: &api.Request{
488+
CommitNow: true,
489+
Query: query,
490+
Mutations: []*api.Mutation{
491+
{
492+
Set: userNQuads,
493+
// Assuming that if groot exists, it is in guardian group
494+
Cond: "@if(eq(len(grootid), 0) and gt(len(guid), 0))",
495+
},
485496
},
486497
},
498+
doAuth: NoAuthorize,
487499
}
488500

489-
resp, err := (&Server{}).doQuery(ctx, req, NoAuthorize)
501+
resp, err := (&Server{}).doQuery(ctx, req)
490502
if err != nil {
491503
return errors.Wrapf(err, "while upserting user with id %s", x.GrootId)
492504
}

edgraph/graphql.go

+95-76
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,11 @@ import (
2323
"encoding/hex"
2424
"encoding/json"
2525
"fmt"
26-
"github.com/dgraph-io/dgraph/gql"
2726
"sort"
2827
"time"
2928

29+
"github.com/dgraph-io/dgraph/gql"
30+
3031
"github.com/dgraph-io/dgo/v200/protos/api"
3132
"github.com/dgraph-io/dgraph/graphql/schema"
3233
"github.com/dgraph-io/ristretto/z"
@@ -42,36 +43,39 @@ func ResetCors(closer *z.Closer) {
4243
closer.Done()
4344
}()
4445

45-
req := &api.Request{
46-
Query: `query{
46+
req := &Request{
47+
req: &api.Request{
48+
Query: `query{
4749
cors as var(func: has(dgraph.cors))
4850
}`,
49-
Mutations: []*api.Mutation{
50-
{
51-
Set: []*api.NQuad{
52-
{
53-
Subject: "_:a",
54-
Predicate: "dgraph.cors",
55-
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: "*"}},
56-
},
57-
{
58-
Subject: "_:a",
59-
Predicate: "dgraph.type",
60-
ObjectValue: &api.Value{Val: &api.Value_StrVal{
61-
StrVal: "dgraph.type.cors"}},
51+
Mutations: []*api.Mutation{
52+
{
53+
Set: []*api.NQuad{
54+
{
55+
Subject: "_:a",
56+
Predicate: "dgraph.cors",
57+
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: "*"}},
58+
},
59+
{
60+
Subject: "_:a",
61+
Predicate: "dgraph.type",
62+
ObjectValue: &api.Value{Val: &api.Value_StrVal{
63+
StrVal: "dgraph.type.cors"}},
64+
},
6265
},
66+
Cond: `@if(eq(len(cors), 0))`,
6367
},
64-
Cond: `@if(eq(len(cors), 0))`,
6568
},
69+
CommitNow: true,
6670
},
67-
CommitNow: true,
71+
doAuth: NoAuthorize,
6872
}
6973

7074
for closer.Ctx().Err() == nil {
7175
ctx, cancel := context.WithTimeout(closer.Ctx(), time.Minute)
7276
defer cancel()
7377
ctx = context.WithValue(ctx, IsGraphql, true)
74-
if _, err := (&Server{}).doQuery(ctx, req, NoAuthorize); err != nil {
78+
if _, err := (&Server{}).doQuery(ctx, req); err != nil {
7579
glog.Infof("Unable to upsert cors. Error: %v", err)
7680
time.Sleep(100 * time.Millisecond)
7781
}
@@ -93,35 +97,41 @@ func AddCorsOrigins(ctx context.Context, origins []string) error {
9397
if err != nil {
9498
return err
9599
}
96-
req := &api.Request{
97-
Query: `query{
100+
req := &Request{
101+
req: &api.Request{
102+
Query: `query{
98103
cors as var(func: has(dgraph.cors))
99104
}`,
100-
Mutations: []*api.Mutation{
101-
{
102-
SetNquads: generateNquadsForCors(uid, origins),
103-
Cond: `@if(gt(len(cors), 0))`,
104-
DelNquads: []byte(`<` + uid + `>` + ` <dgraph.cors> * .`),
105+
Mutations: []*api.Mutation{
106+
{
107+
SetNquads: generateNquadsForCors(uid, origins),
108+
Cond: `@if(gt(len(cors), 0))`,
109+
DelNquads: []byte(`<` + uid + `>` + ` <dgraph.cors> * .`),
110+
},
105111
},
112+
CommitNow: true,
106113
},
107-
CommitNow: true,
114+
doAuth: NoAuthorize,
108115
}
109-
_, err = (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req, NoAuthorize)
116+
_, err = (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req)
110117
return err
111118
}
112119

113120
// GetCorsOrigins retrieve all the cors origin from the database.
114121
func GetCorsOrigins(ctx context.Context) (string, []string, error) {
115-
req := &api.Request{
116-
Query: `query{
122+
req := &Request{
123+
req: &api.Request{
124+
Query: `query{
117125
me(func: has(dgraph.cors)){
118126
uid
119127
dgraph.cors
120128
}
121129
}`,
122-
ReadOnly: true,
130+
ReadOnly: true,
131+
},
132+
doAuth: NoAuthorize,
123133
}
124-
res, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req, NoAuthorize)
134+
res, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req)
125135
if err != nil {
126136
return "", nil, err
127137
}
@@ -160,29 +170,32 @@ func GetCorsOrigins(ctx context.Context) (string, []string, error) {
160170

161171
// UpdateSchemaHistory updates graphql schema history.
162172
func UpdateSchemaHistory(ctx context.Context, schema string) error {
163-
req := &api.Request{
164-
Mutations: []*api.Mutation{
165-
{
166-
Set: []*api.NQuad{
167-
{
168-
Subject: "_:a",
169-
Predicate: "dgraph.graphql.schema_history",
170-
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: schema}},
171-
},
172-
{
173-
Subject: "_:a",
174-
Predicate: "dgraph.type",
175-
ObjectValue: &api.Value{Val: &api.Value_StrVal{
176-
StrVal: "dgraph.graphql.history"}},
173+
req := &Request{
174+
req: &api.Request{
175+
Mutations: []*api.Mutation{
176+
{
177+
Set: []*api.NQuad{
178+
{
179+
Subject: "_:a",
180+
Predicate: "dgraph.graphql.schema_history",
181+
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: schema}},
182+
},
183+
{
184+
Subject: "_:a",
185+
Predicate: "dgraph.type",
186+
ObjectValue: &api.Value{Val: &api.Value_StrVal{
187+
StrVal: "dgraph.graphql.history"}},
188+
},
177189
},
190+
SetNquads: []byte(fmt.Sprintf(`_:a <dgraph.graphql.schema_created_at> "%s" .`,
191+
time.Now().Format(time.RFC3339))),
178192
},
179-
SetNquads: []byte(fmt.Sprintf(`_:a <dgraph.graphql.schema_created_at> "%s" .`,
180-
time.Now().Format(time.RFC3339))),
181193
},
194+
CommitNow: true,
182195
},
183-
CommitNow: true,
196+
doAuth: NoAuthorize,
184197
}
185-
_, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req, NoAuthorize)
198+
_, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req)
186199
return err
187200
}
188201

@@ -213,13 +226,16 @@ func ProcessPersistedQuery(ctx context.Context, gqlReq *schema.Request) error {
213226
variables := map[string]string{
214227
"$sha": sha256Hash,
215228
}
216-
req := &api.Request{
217-
Query: queryForSHA,
218-
Vars: variables,
219-
ReadOnly: true,
229+
req := &Request{
230+
req: &api.Request{
231+
Query: queryForSHA,
232+
Vars: variables,
233+
ReadOnly: true,
234+
},
235+
doAuth: NoAuthorize,
220236
}
221237

222-
storedQuery, err := (&Server{}).doQuery(ctx, req, NoAuthorize)
238+
storedQuery, err := (&Server{}).doQuery(ctx, req)
223239

224240
if err != nil {
225241
glog.Errorf("Error while querying sha %s", sha256Hash)
@@ -249,33 +265,36 @@ func ProcessPersistedQuery(ctx context.Context, gqlReq *schema.Request) error {
249265
return errors.New("provided sha does not match query")
250266
}
251267

252-
req := &api.Request{
253-
Mutations: []*api.Mutation{
254-
{
255-
Set: []*api.NQuad{
256-
{
257-
Subject: "_:a",
258-
Predicate: "dgraph.graphql.p_query",
259-
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: query}},
260-
},
261-
{
262-
Subject: "_:a",
263-
Predicate: "dgraph.graphql.p_sha256hash",
264-
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: sha256Hash}},
265-
},
266-
{
267-
Subject: "_:a",
268-
Predicate: "dgraph.type",
269-
ObjectValue: &api.Value{Val: &api.Value_StrVal{
270-
StrVal: "dgraph.graphql.persisted_query"}},
268+
req = &Request{
269+
req: &api.Request{
270+
Mutations: []*api.Mutation{
271+
{
272+
Set: []*api.NQuad{
273+
{
274+
Subject: "_:a",
275+
Predicate: "dgraph.graphql.p_query",
276+
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: query}},
277+
},
278+
{
279+
Subject: "_:a",
280+
Predicate: "dgraph.graphql.p_sha256hash",
281+
ObjectValue: &api.Value{Val: &api.Value_StrVal{StrVal: sha256Hash}},
282+
},
283+
{
284+
Subject: "_:a",
285+
Predicate: "dgraph.type",
286+
ObjectValue: &api.Value{Val: &api.Value_StrVal{
287+
StrVal: "dgraph.graphql.persisted_query"}},
288+
},
271289
},
272290
},
273291
},
292+
CommitNow: true,
274293
},
275-
CommitNow: true,
294+
doAuth: NoAuthorize,
276295
}
277296

278-
_, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req, NoAuthorize)
297+
_, err := (&Server{}).doQuery(context.WithValue(ctx, IsGraphql, true), req)
279298
return err
280299

281300
}

0 commit comments

Comments
 (0)