Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Sec. index on json #3330

Merged
merged 36 commits into from
Jan 2, 2025
Merged

feat: Sec. index on json #3330

merged 36 commits into from
Jan 2, 2025

Conversation

islamaliev
Copy link
Contributor

Relevant issue(s)

Resolves #2280

Description

Enables json fields indexing.

JSON interface has been extended to allow traversing it with different configurations.

Indexing or documents has been refactor so that instead of acting based off of the fact that there is a special field in the index description (like array or json), we assign a field-specific generator so that every field is responsible for generating a value for the inde key. For example, if we have a composite index made up of fields of types int, array and json (complex composite index), we will generate all possible combinations where int generator will always generate 1 value, array generator will generate values for every element and json generator will generate values for every json node.

Added json encoding/decoding to our encoding package.

@islamaliev islamaliev added area/query Related to the query component perf Performance issue or suggestion labels Dec 16, 2024
@islamaliev islamaliev self-assigned this Dec 16, 2024
@islamaliev islamaliev requested review from a team and pradhanashutosh December 16, 2024 16:08
Copy link

codecov bot commented Dec 16, 2024

Codecov Report

Attention: Patch coverage is 86.57635% with 109 lines in your changes missing coverage. Please review.

Project coverage is 78.46%. Comparing base (cdba31b) to head (f237f2f).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
internal/db/fetcher/indexer_matchers.go 75.00% 52 Missing and 13 partials ⚠️
internal/db/index.go 81.05% 12 Missing and 6 partials ⚠️
client/json.go 96.15% 6 Missing and 2 partials ⚠️
internal/encoding/json.go 92.52% 5 Missing and 3 partials ⚠️
internal/encoding/errors.go 62.50% 2 Missing and 1 partial ⚠️
internal/encoding/field_value.go 90.91% 2 Missing and 1 partial ⚠️
internal/db/fetcher/errors.go 0.00% 2 Missing ⚠️
internal/db/fetcher/indexer.go 0.00% 1 Missing ⚠️
internal/db/fetcher/indexer_iterators.go 98.44% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3330      +/-   ##
===========================================
+ Coverage    78.00%   78.46%   +0.46%     
===========================================
  Files          389      392       +3     
  Lines        35446    35640     +194     
===========================================
+ Hits         27648    27964     +316     
+ Misses        6151     6044     -107     
+ Partials      1647     1632      -15     
Flag Coverage Δ
all-tests 78.46% <86.58%> (+0.46%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
client/normal_util.go 96.12% <100.00%> (+0.12%) ⬆️
internal/encoding/bool.go 100.00% <100.00%> (ø)
internal/encoding/encoding.go 100.00% <ø> (ø)
internal/encoding/null.go 100.00% <100.00%> (ø)
internal/encoding/type.go 100.00% <100.00%> (+11.76%) ⬆️
internal/planner/scan.go 90.04% <100.00%> (+0.16%) ⬆️
internal/db/fetcher/indexer.go 83.69% <0.00%> (ø)
internal/db/fetcher/indexer_iterators.go 85.01% <98.44%> (+10.06%) ⬆️
internal/db/fetcher/errors.go 37.04% <0.00%> (+21.04%) ⬆️
internal/encoding/errors.go 82.76% <62.50%> (-7.72%) ⬇️
... and 5 more

... and 13 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cdba31b...f237f2f. Read the comment docs.

@islamaliev islamaliev removed the request for review from pradhanashutosh December 16, 2024 17:28
@islamaliev islamaliev force-pushed the feat/sec-index-on-json branch from 06e78e4 to 8308672 Compare January 2, 2025 16:08
@islamaliev islamaliev merged commit 4318db4 into develop Jan 2, 2025
44 of 45 checks passed
@islamaliev islamaliev deleted the feat/sec-index-on-json branch January 2, 2025 20:52
@fredcarle fredcarle added this to the DefraDB v0.16 milestone Feb 18, 2025
ChrisBQu pushed a commit to ChrisBQu/defradb that referenced this pull request Feb 21, 2025
## Relevant issue(s)

Resolves sourcenetwork#2280

## Description

Enables json fields indexing.

JSON interface has been extended to allow traversing it with different
configurations.

Indexing or documents has been refactor so that instead of acting based
off of the fact that there is a special field in the index description
(like array or json), we assign a field-specific generator so that every
field is responsible for generating a value for the inde key. For
example, if we have a composite index made up of fields of types int,
array and json (complex composite index), we will generate all possible
combinations where int generator will always generate 1 value, array
generator will generate values for every element and json generator will
generate values for every json node.

Added json encoding/decoding to our encoding package.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/query Related to the query component perf Performance issue or suggestion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sec. Index: enable indexing of key-values of JSON fields
5 participants