Closed
Conversation
Add new query encryption API for searchable encryption: - encryptQuery(): Single value query encryption with index type control - createQuerySearchTerms(): Bulk query encryption with mixed index types - createJsonSearchTerms(): JSON path and containment query encryption Features: - Support for all index types: ore, match, unique, ste_vec - Lock context support for all query operations - SEM-only payloads (no ciphertext) optimized for database queries - Path queries (dot notation and array format) - Containment queries (contains/contained_by) Test coverage includes: - Lock context integration tests - Boundary conditions (empty strings, Unicode, emoji, large numbers) - Deep JSON nesting (5+ levels) - Bulk operation edge cases - Error handling scenarios
🦋 Changeset detectedLatest commit: 7658095 The changes in this PR will be included in the next version bump. This PR includes changesets to release 10 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Remove public API additions that diverged from requirements: - Requirements specified using existing createSearchTerms function - Requirements specified NOT changing the existing protectjs public API Removed: - encryptQuery(), createQuerySearchTerms(), createJsonSearchTerms() methods - Public type exports for query-specific types - Test files for removed public API Internal operation files remain for potential future use.
- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released
Add 32 tests covering JsonSearchTermsOperation including: - Path queries (string/array paths, deep paths, path-only) - Containment queries (simple/nested objects, multiple keys) - Bulk operations (mixed queries, multiple columns) - Lock context integration - Edge cases (unicode, deep nesting, special chars) - Error handling (missing ste_vec index) - Selector generation verification
Add missing public methods to ProtectClient: - encryptQuery: encrypt single value with explicit index type - createQuerySearchTerms: bulk query term encryption - createJsonSearchTerms: JSON path/containment query encryption Update tests to use public API instead of unsafe internal access. Export new operation types and search term types.
Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).
…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.
SearchTerm is now a union of SimpleSearchTerm, JsonPathSearchTerm, and JsonContainmentSearchTerm, enabling createSearchTerms to accept all search term types in a single call. - Add SimpleSearchTerm type alias for original behavior - Update SearchTerm to union type - Export SimpleSearchTerm from public API
SearchTermsOperation.execute() now handles JSON search terms: - Partitions terms by type (simple, JSON path, JSON containment) - Encrypts simple terms with encryptBulk (original behavior) - Encrypts JSON terms with encryptQueryBulk (ste_vec index) - Reassembles results in original order - Supports mixed batches of simple and JSON terms Also includes: - Type guards for SearchTerm variants - Helper functions (pathToSelector, buildNestedObject, flattenJson) - withLockContext support for JSON terms - Extracted shared logic into encryptSearchTermsHelper to reduce duplication
Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call
Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.
Remove the deprecated createJsonSearchTerms function and supporting code, consolidating JSON search functionality into the unified createSearchTerms API. - Remove createJsonSearchTerms method from ProtectClient - Delete json-search-terms.ts operation file - Remove JsonSearchTermsOperation export from index - Migrate comprehensive tests to search-terms.test.ts - Update documentation examples to use createSearchTerms
Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.
tobyhede
force-pushed
the
searchable-json-query-api
branch
from
66227cd to
c4f5d8c
Compare
Remove __RESOLVE_AT_BUILD__ placeholder in favor of inferring the ste_vec prefix from table/column context when not explicitly set. Changes: - searchableJson() now sets empty ste_vec object - ProtectTable.build() and buildEncryptConfig() infer prefix when missing - Simplified error checks in search-terms.ts - Enabled previously commented test for ste_vec index
Add tests to prevent regressions based on code review feedback: - Selector prefix resolution test verifying table/column prefix - encryptQuery(null) null handling verification - escaped-composite-literal return type for createQuerySearchTerms - ste_vec index with default queryOp for JSON object encryption
Set temporary column name prefix in searchableJson() to satisfy type requirements, then always overwrite with full table/column prefix during build. Update search-terms.ts to always derive prefix from table/column names rather than relying on column.build() which may have incomplete prefix. This fixes the DTS build error where prefix was required by the type but not set until table build time.
tobyhede
commented
Jan 20, 2026
calvinbrewer
approved these changes
Jan 20, 2026
Contributor
calvinbrewer
left a comment
calvinbrewer
left a comment
There was a problem hiding this comment.
This looks great @tobyhede good stuff
Contributor
|
Note: you don't have a changeset in here - might be worth bumping the major version just to be safe to account for all the different typescript targets |
coderdan
reviewed
Jan 20, 2026
coderdan
reviewed
Jan 20, 2026
coderdan
reviewed
Jan 20, 2026
coderdan
reviewed
Jan 20, 2026
coderdan
reviewed
Jan 20, 2026
Add 77 tests covering JSONB operations matching proxy test coverage: - jsonb_array_elements and jsonb_array_length extraction - Field access via -> operator and path queries - Contains (@>) and contained_by (<@) operations - jsonb_path_exists, jsonb_path_query, jsonb_path_query_first - Comparison operations (=, >, >=, <, <=) on extracted fields - Data types: strings, numbers, arrays, nested objects, null, boolean - Edge cases: deep nesting, unicode, special characters, large objects
- Add 30000ms timeout to all 77 tests for consistency - Add header comment explaining intentional test overlap - Remove unused expectSteVecSelector import - Add TEST COVERAGE MAPPING section linking describe blocks to proxy SQL
…atasets Close test coverage gaps identified in proxy parity assessment: Phase 1 - Unknown field/path edge cases: - array_elements with unknown field (empty result) - array_length with unknown field (empty result) - path_query with unknown path (empty set return) - path_query_first with unknown path (NULL return) Phase 2 - Large dataset containment tests: - 100 variations batch containment queries - Complex nested containment objects - Mixed contains/contained_by operations - 100-element array containment - Numeric edge cases (MAX/MIN_SAFE_INTEGER) - Subset vs exact match patterns Test count: 81 → 87 tests (147% of proxy coverage)
…test suite Add comprehensive validation tests across all 5 JSONB test files to verify: - Data stored in database is encrypted (not plaintext) - Encrypted structure contains expected ciphertext markers - Data can be decrypted and matches original values - Round-trip encryption/decryption preserves all fields Each file now includes: - Encryption Verification describe block - Decryption Verification describe block - Query Execution tests (jsonb-comparison.test.ts) Also adds shared test data fixtures in jsonb-test-data.ts.
Add CREATE TABLE IF NOT EXISTS statements in beforeAll hooks to ensure test tables exist before running tests. This fixes CI failures where the tables were not present in the test database.
Drop and recreate test tables with eql_v2_encrypted column type instead of native JSONB. This fixes test failures caused by encryptedType() serializing data for eql_v2_encrypted composite format, which PostgreSQL's native JSONB parser rejects as invalid JSON. Changes: - Add DROP TABLE IF EXISTS before CREATE TABLE - Change column type from JSONB to eql_v2_encrypted
tobyhede
force-pushed
the
searchable-json-query-api
branch
from
d3f47a1 to
649b883
Compare
Enable ste_vec index for JSONB containment and path queries directly from Drizzle schema definition. Automatically configures the index required for path selection (->, ->>) and containment (@>, <@) queries.
…yptQuery API
Migrate all searchable JSONB tests to use the new encryptQuery API:
- Replace SearchTerm type with QueryTerm
- Use contains/containedBy properties instead of containmentType
- Update helper functions to match new output format ({ sv: [...] })
- Enable searchableJson in test table schemas
Files updated:
- jsonb-containment.test.ts
- jsonb-array-operations.test.ts
- jsonb-path-operations.test.ts
- jsonb-field-access.test.ts
- jsonb-comparison.test.ts
Support test isolation by allowing test runs to tag their data.
Update jsonb-proxy-parity.test.ts and json-extraction-ops.test.ts: - Replace createSearchTerms with encryptQuery - Replace SearchTerm type with QueryTerm - Update containment syntax: value + containmentType → contains/containedBy
Add new test file covering: - Scalar queries (equality, range, free text search) - JSON path queries (selector-only, path+value, wildcards) - JSON containment queries (contains, containedBy) - Bulk operations and error handling - Edge cases (unicode, special chars, deep nesting)
Strengthen test assertions by replacing weak negative checks with positive EQL v2 structure validation: - Add expectEncryptedJsonPayload helper to verify i/v/c fields - Enhance expectJsonPathWithValue with EQL v2 and plaintext checks - Enhance expectJsonPathSelectorOnly with EQL v2 structure validation - Update all json-protect.test.ts assertions to use new helper - Pass path/value params to helpers for plaintext leak detection
coderdan
reviewed
Jan 29, 2026
coderdan
reviewed
Jan 29, 2026
coderdan
reviewed
Jan 29, 2026
| table: string | ||
| } | ||
|
|
||
| /** |
Contributor
There was a problem hiding this comment.
Is this still used anywhere? I can't see how its used.
Contributor
Author
There was a problem hiding this comment.
Is used, but only in tests that are explicitly preserved to test the deprecated functionality.
Updating to make clearer, and grouping the deprecated tests.
coderdan
reviewed
Jan 30, 2026
| const searchableSchema = csTable('drizzle_jsonb_array_ops_test', { | ||
| encrypted_jsonb: csColumn('encrypted_jsonb').searchableJson(), | ||
| // Array length extracted fields for range operations | ||
| "jsonb_array_length(encrypted_jsonb->'array_string')": csColumn( |
Contributor
There was a problem hiding this comment.
This is a gnarly column name (not sure if it'd be valid in postgres!).
Is this doing something clever for the test or has something gone awry?
Use standard JSONPath terminology. Updates function name and all usages in batch-encrypt-query.ts and search-terms.ts.
Comprehensive test coverage including: - Single and multi-segment array paths - Dot-separated string input - Bracket notation for special characters and spaces - Quote escaping - Empty path edge case
Clarify the semantic difference between string and array path inputs: - String paths split on dots (e.g., "a.b.c" → $.a.b.c) - Array paths preserve dots within segments (e.g., ["a.b.c"] → $["a.b.c"])
…rectory Organize test files by moving deprecated search-terms tests to packages/protect/__tests__/deprecated/ for better codebase structure.
…dule Update imports to reference search-terms from its new location in the deprecated subdirectory.
Add comprehensive tests for number value encryption including: - Default (auto-inferred) query type - Explicit equality query type - Explicit orderAndRange query type - Range query operators (lt, lte, gt, gte, between) - Negative numbers and zero values
…ed directory Move search-terms.ts to operations/deprecated/ subdirectory to match the organizational structure established for deprecated functionality.
…ctor - Fix path reference in deprecation comment (./encrypt-query.ts -> ../encrypt-query.ts) - Add explanatory comments for magic numbers in test utility functions - Add @deprecated JSDoc to SearchTermsOperation type export for IDE visibility
- Create shared helpers directory with: - jsonb-test-setup.ts: Factory function for test setup/teardown - jsonb-query-helpers.ts: Validation helpers (expectJsonPathSelectorOnly, etc.) - jsonb-e2e-helpers.ts: E2E query execution helpers - Consolidate tests into jsonb/ directory: - Move 5 operation test files (array-ops, comparison, containment, field-access, path-ops) - Extract encryption/decryption verification tests into encryption-verification.test.ts - Consolidate 45 Pattern B TODO tests into pattern-b-e2e.test.ts - Streamline operation test files: - Remove duplicated helper functions (use shared helpers) - Remove duplicated setup/teardown (use factory) - Remove encryption verification tests (moved to consolidated file) - Remove Pattern B tests (moved to consolidated file) Benefits: - ~50% code reduction in individual operation test files - Single source of truth for helpers and setup - Clear organization with operation tests focused on operations only
Contributor
|
Closing in favour of #276. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.