CLIENT-4216 Support IN operation for Lists#65
CLIENT-4216 Support IN operation for Lists#65agrgr wants to merge 11 commits intomainaerospike/expression-dsl-java:mainfrom CLIENT-4216-support-in-list-operationaerospike/expression-dsl-java:CLIENT-4216-support-in-list-operationCopy head branch name to clipboard
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for an IN operator in the Aerospike DSL so expressions like $.bin in [1,2,3] and "x" in $.listBin compile into Aerospike list existence expressions, while ensuring IN clauses are excluded from secondary-index filter selection.
Changes:
- Extend the ANTLR grammar and expression AST to recognize the
INoperator. - Generate Aerospike expressions for
INviaListExp.getByValue(ListReturnType.EXISTS, left, right)and excludeINfrom SI filter selection. - Add comprehensive expression and parsed-expression/index-selection tests for
IN.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/main/antlr4/com/aerospike/dsl/Condition.g4 | Adds IN token/rule and updates list constant parsing to support [] tokenization. |
| src/main/java/com/aerospike/dsl/parts/ExpressionContainer.java | Adds IN to the operation enum. |
| src/main/java/com/aerospike/dsl/visitor/ExpressionConditionVisitor.java | Builds IN expression containers and infers operand types for IN. |
| src/main/java/com/aerospike/dsl/visitor/VisitorUtils.java | Emits Aerospike IN expression, validates IN placeholder RHS, and excludes IN from index-cardinality filter selection. |
| src/test/java/com/aerospike/dsl/expression/InExpressionsTests.java | Adds parsing/Exp-building tests (including negatives) for IN. |
| src/test/java/com/aerospike/dsl/parsedExpression/InFilterTests.java | Adds parsed-expression tests to ensure IN is excluded from secondary-index Filter selection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (expr.getOperationType() == IN && rightIsPlaceholder) { | ||
| validateInPlaceholderValue((PlaceholderOperand) expr.getRight(), placeholderValues); | ||
| } |
There was a problem hiding this comment.
IN with a right-hand placeholder that resolves to a Java List will still fail later: this validation passes, but the placeholder is then resolved via OperandFactory.createOperand(...), which currently doesn't support List values (it throws UnsupportedOperationException). Consider either extending placeholder resolution to wrap list values into a ListOperand (and map values into MapOperand if needed), or special-casing IN placeholder resolution to build Exp.val(resolvedList) directly so $.bin in ?0 works when ?0 is a list.
|
|
||
| private static void inferInTypes(AbstractPart left, AbstractPart right) { | ||
| if (right.getPartType() == AbstractPart.PartType.BIN_PART) { | ||
| ((BinPart) right).updateExp(Exp.Type.LIST); |
There was a problem hiding this comment.
inferInTypes currently forces the right operand BIN_PART to Exp.Type.LIST unconditionally. This overwrites an explicitly provided type (e.g. $.x.get(type: INT)) and can mask user errors. Suggest only auto-setting LIST when the right bin type is not explicitly set; if it is explicitly set and not LIST, fail fast with the same parse exception used elsewhere for invalid IN right operands.
| ((BinPart) right).updateExp(Exp.Type.LIST); | |
| BinPart rightBin = (BinPart) right; | |
| if (!rightBin.isTypeExplicitlySet()) { | |
| rightBin.updateExp(Exp.Type.LIST); | |
| } |
|
|
||
| @Test | ||
| void placeholderAsRightOperand() { | ||
| parseFilterExp(ExpressionContext.of("$.name in ?0")); |
There was a problem hiding this comment.
placeholderAsRightOperand currently calls parseFilterExp(ExpressionContext.of("$.name in ?0")) without providing PlaceholderValues. In the existing placeholder tests, missing values causes a parse failure (PLACEHOLDER_OPERAND is unsupported when values are null), so this test will fail. Either provide placeholder values here (ideally a List value once placeholder list support is implemented) or change the test to assert the expected exception for missing placeholder values.
| parseFilterExp(ExpressionContext.of("$.name in ?0")); | |
| assertThatThrownBy(() -> parseFilterExp(ExpressionContext.of("$.name in ?0"))) | |
| .isInstanceOf(DslParseException.class); |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| QUOTED_STRING: ('\'' (~'\'')* '\'') | ('"' (~'"')* '"'); | ||
|
|
||
| listConstant: '[' unaryExpression? (',' unaryExpression)* ']'; | ||
| listConstant: '[' unaryExpression? (',' unaryExpression)* ']' | LIST_TYPE_DESIGNATOR; |
There was a problem hiding this comment.
The addition of LIST_TYPE_DESIGNATOR as an alternative in the listConstant rule appears unrelated to the IN operation support. The LIST_TYPE_DESIGNATOR token (defined as '[]' on line 417) is already used in the listPart rule for path operations. Adding it here would allow '[]' to be parsed as a list literal, but there are no tests demonstrating this functionality or explaining its purpose in relation to the IN operation. Consider removing this addition if it's not needed for the IN operation, or add tests and documentation if it's intentional.
| listConstant: '[' unaryExpression? (',' unaryExpression)* ']' | LIST_TYPE_DESIGNATOR; | |
| listConstant: '[' unaryExpression? (',' unaryExpression)* ']'; |
There was a problem hiding this comment.
This is done to handle empty list, added a comment
| static Exp.Type inferTypeFromListElements(ListOperand listOperand) { | ||
| List<Object> values = listOperand.getValue(); | ||
| if (values.isEmpty()) return null; | ||
| Object first = values.get(0); | ||
| if (first instanceof String) return Exp.Type.STRING; | ||
| if (first instanceof Boolean) return Exp.Type.BOOL; | ||
| if (first instanceof Float || first instanceof Double) return Exp.Type.FLOAT; | ||
| if (first instanceof Integer || first instanceof Long) return Exp.Type.INT; | ||
| if (first instanceof java.util.List) return Exp.Type.LIST; | ||
| if (first instanceof java.util.Map) return Exp.Type.MAP; |
There was a problem hiding this comment.
The inferTypeFromListElements method only checks the first element of the list to infer the type. This could lead to incorrect type inference if the list contains mixed types (e.g., [1, "string", 2]). While Aerospike may not support heterogeneous lists, consider adding a comment explaining this assumption or validating that all elements are of the same type to prevent silent type mismatches.
| static Exp.Type inferTypeFromListElements(ListOperand listOperand) { | |
| List<Object> values = listOperand.getValue(); | |
| if (values.isEmpty()) return null; | |
| Object first = values.get(0); | |
| if (first instanceof String) return Exp.Type.STRING; | |
| if (first instanceof Boolean) return Exp.Type.BOOL; | |
| if (first instanceof Float || first instanceof Double) return Exp.Type.FLOAT; | |
| if (first instanceof Integer || first instanceof Long) return Exp.Type.INT; | |
| if (first instanceof java.util.List) return Exp.Type.LIST; | |
| if (first instanceof java.util.Map) return Exp.Type.MAP; | |
| /** | |
| * Infer the Aerospike Exp.Type for a list operand by examining its elements. | |
| * <p> | |
| * This method assumes/enforces that all non-null elements in the list are of the same | |
| * logical type (e.g. all strings, all integers, etc.). If heterogeneous element types | |
| * are detected, a DslParseException is thrown to avoid silent type mismatches. | |
| */ | |
| static Exp.Type inferTypeFromListElements(ListOperand listOperand) { | |
| List<Object> values = listOperand.getValue(); | |
| if (values.isEmpty()) { | |
| return null; | |
| } | |
| Exp.Type inferredType = null; | |
| for (Object value : values) { | |
| if (value == null) { | |
| // Skip nulls for the purpose of type inference. | |
| continue; | |
| } | |
| Exp.Type currentType = inferElementType(value); | |
| if (currentType == null) { | |
| throw new DslParseException( | |
| "Unsupported element type in IN list: " + value.getClass().getName()); | |
| } | |
| if (inferredType == null) { | |
| inferredType = currentType; | |
| } else if (inferredType != currentType) { | |
| throw new DslParseException( | |
| "IN list elements must all be of the same type; found " + | |
| inferredType + " and " + currentType); | |
| } | |
| } | |
| return inferredType; | |
| } | |
| /** | |
| * Map a single Java object to the corresponding Aerospike Exp.Type. | |
| */ | |
| private static Exp.Type inferElementType(Object element) { | |
| if (element instanceof String) return Exp.Type.STRING; | |
| if (element instanceof Boolean) return Exp.Type.BOOL; | |
| if (element instanceof Float || element instanceof Double) return Exp.Type.FLOAT; | |
| if (element instanceof Integer || element instanceof Long) return Exp.Type.INT; | |
| if (element instanceof java.util.List) return Exp.Type.LIST; | |
| if (element instanceof java.util.Map) return Exp.Type.MAP; |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| && !((BinPart) left).isTypeExplicitlySet() | ||
| && right.getPartType() == AbstractPart.PartType.LIST_OPERAND) { | ||
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | ||
| if (inferredType != null) { | ||
| ((BinPart) left).updateExp(inferredType); | ||
| } |
There was a problem hiding this comment.
For IN, type compatibility between the left operand and the list element type is not validated when the left side has an explicit type (e.g. $.name.get(type: STRING) in [1,2] will currently parse/build an expression). Consider inferring the element type for LIST_OPERAND right-hand sides and calling validateComparableTypes() against the left operand type (at least when left is a BIN_PART with an explicitly set type), so mismatches fail fast like other comparison operators.
| && !((BinPart) left).isTypeExplicitlySet() | |
| && right.getPartType() == AbstractPart.PartType.LIST_OPERAND) { | |
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | |
| if (inferredType != null) { | |
| ((BinPart) left).updateExp(inferredType); | |
| } | |
| && right.getPartType() == AbstractPart.PartType.LIST_OPERAND) { | |
| BinPart leftBin = (BinPart) left; | |
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | |
| // If the list is empty (or only nulls), we cannot infer a type – nothing to do. | |
| if (inferredType == null) { | |
| return; | |
| } | |
| if (!leftBin.isTypeExplicitlySet()) { | |
| // Existing behavior: infer the left bin type from homogeneous list elements. | |
| leftBin.updateExp(inferredType); | |
| } else { | |
| // New behavior: when the left type is explicit, validate compatibility with | |
| // the inferred list element type so mismatches fail fast. | |
| Exp.Type leftType = leftBin.getExpType(); | |
| if (leftType != inferredType) { | |
| throw new DslParseException( | |
| "Type mismatch in IN operation: left operand is of type " | |
| + leftType + " but list elements are of type " + inferredType); | |
| } | |
| } |
| } else if (value instanceof List) { | ||
| return new ListOperand((List<Object>) value); | ||
| } else if (value instanceof SortedMap) { | ||
| return new MapOperand((SortedMap<Object, Object>) value); | ||
| } else if (value instanceof Map) { | ||
| return new MapOperand(new TreeMap<>((Map<Object, Object>) value)); | ||
| } else { | ||
| throw new UnsupportedOperationException(String.format("Cannot create operand from value of type %s, " + | ||
| "only String, boolean, float, double, long and integer values are currently supported", | ||
| throw new UnsupportedOperationException(String.format("Cannot create operand from value of type %s", | ||
| value.getClass().getSimpleName())); | ||
| } |
There was a problem hiding this comment.
createOperand() wraps any Map in a TreeMap, which will throw a ClassCastException at runtime if the map keys are not mutually comparable (or if mixed key types are present). Since this method is used for resolving placeholders, it would be better to fail deterministically with a DslParseException/UnsupportedOperationException that explains the requirement (or only accept SortedMap and reject other Map inputs).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | IN | ||
| ; | ||
|
|
There was a problem hiding this comment.
Adding IN as an alternative in the mapKey rule (and using ctx.IN() in MapKey.from()) is correct. However, MapKeyRange and MapKeyList (which also use mapKey) have not been updated to handle the IN token. When the IN keyword is used as a key in a range expression (like $.mapBin.{in-z}) or a key list ($.mapBin.{in,z}), the existing code in MapKeyRange.from() and MapKeyList.from() will throw a NullPointerException because it only checks for NAME_IDENTIFIER() and QUOTED_STRING(), not IN(). These classes should be updated with the same fix applied to MapKey.from() in this PR.
| | IN | |
| ; | |
| ; |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (left.getPartType() != BIN_PART | ||
| || right.getPartType() != LIST_OPERAND) { | ||
| return; | ||
| } | ||
| BinPart leftBin = (BinPart) left; | ||
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | ||
| if (inferredType == null) { | ||
| return; | ||
| } |
There was a problem hiding this comment.
inferTypeFromListElements() enforces homogeneous element types, but it’s only invoked via inferLeftBinTypeFromList(), which returns early unless the left operand is BIN_PART. As a result, expressions like "x" in [1, "y"] or ?0 in [1, "y"] won’t throw, while $.bin in [1, "y"] will—this is inconsistent with the stated rule and the negative tests. Consider validating LIST_OPERAND element types for all IN expressions (even when the left isn’t a bin) and add a test that covers a non-bin left operand with a mixed-type list.
| if (left.getPartType() != BIN_PART | |
| || right.getPartType() != LIST_OPERAND) { | |
| return; | |
| } | |
| BinPart leftBin = (BinPart) left; | |
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | |
| if (inferredType == null) { | |
| return; | |
| } | |
| // Only proceed when the right operand is a list; otherwise nothing to infer/validate. | |
| if (right.getPartType() != LIST_OPERAND) { | |
| return; | |
| } | |
| // Always validate list element types for IN expressions, regardless of the left operand. | |
| Exp.Type inferredType = inferTypeFromListElements((ListOperand) right); | |
| if (inferredType == null) { | |
| // Empty list: nothing to infer or validate further. | |
| return; | |
| } | |
| // If the left operand is not a bin, there is no bin type to infer or validate against. | |
| if (left.getPartType() != BIN_PART) { | |
| return; | |
| } | |
| // For bin operands, infer or validate the bin's Exp type against the list element type. | |
| BinPart leftBin = (BinPart) left; |
| Object value = placeholderValues.getValue(placeholder.getIndex()); | ||
| if (!(value instanceof List)) { | ||
| throw new DslParseException("IN operation requires a List as the right operand"); |
There was a problem hiding this comment.
The Javadoc says this validation happens "so that the error message references the placeholder index", but the thrown message doesn’t include the index (it’s the same for any placeholder). Either include the placeholder index in the exception message (e.g., ?0) or adjust the Javadoc to match the current behavior.
| Object value = placeholderValues.getValue(placeholder.getIndex()); | |
| if (!(value instanceof List)) { | |
| throw new DslParseException("IN operation requires a List as the right operand"); | |
| int index = placeholder.getIndex(); | |
| Object value = placeholderValues.getValue(index); | |
| if (!(value instanceof List)) { | |
| throw new DslParseException("IN operation requires a List as the right operand for placeholder ?" + index); |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * Validates that a placeholder used as the right operand of an IN expression | ||
| * resolves to a {@link java.util.List}. Called before placeholder resolution | ||
| * so that the error message references the placeholder index. | ||
| * | ||
| * @throws DslParseException if the placeholder value is not a List | ||
| */ | ||
| private static void validateInPlaceholderValue(PlaceholderOperand placeholder, | ||
| PlaceholderValues placeholderValues) { | ||
| int index = placeholder.getIndex(); | ||
| Object value = placeholderValues.getValue(index); | ||
| if (!(value instanceof List)) { | ||
| throw new DslParseException( | ||
| "IN operation requires a List as the right operand for placeholder ?" + index); | ||
| } |
There was a problem hiding this comment.
The Javadoc for validateInPlaceholderValue says it throws DslParseException when the placeholder value is not a list, but placeholderValues.getValue(index) can also throw IllegalArgumentException when the placeholder index is missing. Consider documenting that behavior here (or wrapping it into a DslParseException) so callers/tests don’t see an unexpected exception type.
| private static void inferInTypes(AbstractPart left, AbstractPart right) { | ||
| if (right.getPartType() == AbstractPart.PartType.BIN_PART) { | ||
| BinPart rightBin = (BinPart) right; | ||
| if (!rightBin.isTypeExplicitlySet()) { | ||
| rightBin.updateExp(Exp.Type.LIST); | ||
| } else if (rightBin.getExpType() != Exp.Type.LIST) { | ||
| throw new DslParseException( | ||
| "IN operation requires a List as the right operand"); | ||
| } | ||
| } else if (right.getPartType() == AbstractPart.PartType.PATH_OPERAND) { | ||
| Path rightPath = (Path) right; | ||
| if (rightPath.getPathFunction() != null) { | ||
| Exp.Type pathType = rightPath.getPathFunction().getBinType(); | ||
| if (pathType != null && pathType != Exp.Type.LIST) { | ||
| throw new DslParseException( | ||
| "IN operation requires a List as the right operand"); | ||
| } | ||
| } | ||
| } | ||
| Exp.Type inferredType = VisitorUtils.validateListHomogeneity(right); | ||
| VisitorUtils.inferBinTypeFromList(left, inferredType); | ||
| } |
There was a problem hiding this comment.
For IN, the right operand is required to evaluate to a LIST, but inferInTypes() only enforces this for BIN_PART (and PATH_OPERAND when a path function has an explicit non-LIST type). Plain PATH/VARIABLE operands with no explicit type info are always accepted, which means ListExp.getByValue(...) can be built over operands whose inferred/default type is not LIST (e.g., nested map paths default to STRING). Consider either requiring an explicit LIST type on PATH/VARIABLE right operands (e.g., .get(type: LIST) / designator) or coercing the right operand’s type metadata to LIST when used with IN, so validation and generated Exp types align with the operation contract.
| throw new UnsupportedOperationException(String.format("Cannot create operand from value of type %s", | ||
| value.getClass().getSimpleName())); |
There was a problem hiding this comment.
The fallback UnsupportedOperationException message became less actionable (it no longer tells callers which operand value types are supported). Consider restoring the previous detail or switching to a DslParseException with a clearer guidance message, since this error can surface for placeholder values at runtime.
| throw new UnsupportedOperationException(String.format("Cannot create operand from value of type %s", | |
| value.getClass().getSimpleName())); | |
| throw new DslParseException(String.format( | |
| "Unsupported operand value type '%s'. Supported types are: String, Boolean, Float/Double, Integer/Long, List, and Map/SortedMap. " | |
| + "If this value comes from a placeholder, ensure it is resolved to one of the supported types before operand creation.", | |
| value.getClass().getName())); |
|
Implicit type setting only applies to BIN_PART. Once the path becomes a PATH_OPERAND (by having CDT parts like There is a way to go with an operation-aware hint approach (effectively "default of a default") which seems to be the least bad of implicit approaches so far. We could store a When the right operand is a PATH_OPERAND, there's no list literal to inspect, so no cross-inference happens. With the mentioned hint an expression So overall, currently there seem to be two approaches:
What do you think, @tim-aero? |
|
I think it's valid to require types when they are abiguous, eg However, if we have already determined the type of the left operand, the type should not be necessary:
My confusion arises because |
No description provided.