[refractor] Using Spring jdbc to query Greptime log #3887
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request refactors the GreptimeDB integration to use JDBC (PostgreSQL protocol) instead of HTTP REST API, while also adding log content search filtering capabilities across the application. The main changes include migrating from HTTP-based queries to JDBC-based queries using Spring's JdbcTemplate with HikariCP connection pooling, and adding a new search parameter to filter logs by content.
- Migration from HTTP REST API to PostgreSQL JDBC protocol for GreptimeDB queries
- Addition of log content search filtering with case-insensitive substring matching
- Updates to E2E test infrastructure to expose and configure the PostgreSQL endpoint (port 4003)
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| web-app/src/assets/i18n/*.json | Added internationalized labels for log content search field across 5 languages |
| web-app/src/app/service/log.service.ts | Added search parameter to all log query methods |
| web-app/src/app/routes/log/log-stream/log-stream.component.* | Added log content filter input field and state management |
| web-app/src/app/routes/log/log-manage/log-manage.component.* | Added search content field and integrated with query/stats methods |
| hertzbeat-warehouse/pom.xml | Added spring-boot-starter-jdbc and postgresql driver dependencies |
| hertzbeat-warehouse/src/main/java/.../GreptimeSqlQueryExecutor.java | Complete rewrite to use JdbcTemplate with HikariCP instead of RestTemplate |
| hertzbeat-warehouse/src/main/java/.../GreptimeDbDataStorage.java | Updated to use parameterized queries and changed body field type to String |
| hertzbeat-warehouse/src/main/java/.../GreptimeProperties.java | Added postgresEndpoint configuration property |
| hertzbeat-warehouse/src/main/java/.../HistoryDataReader.java | Added searchContent parameter to log query interface methods with legacy overloads |
| hertzbeat-log/src/main/java/.../LogQueryController.java | Added search parameter to all API endpoints |
| hertzbeat-log/src/main/java/.../LogSseFilterCriteria.java | Added logContent field for SSE filter matching |
| hertzbeat-warehouse/src/test/java/.../GreptimeDbDataStorageTest.java | Updated mocks to use new JDBC query methods instead of HTTP execute |
| hertzbeat-warehouse/src/test/java/.../GreptimeSqlQueryExecutorTest.java | Completely rewritten to test JDBC-based implementation |
| hertzbeat-log/src/test/java/.../LogQueryControllerTest.java | Updated test mocks to include new searchContent parameter |
| hertzbeat-e2e/.../GreptimeLogStorageE2eTest.java | Added PostgreSQL port exposure and dynamic property registration |
| hertzbeat-e2e/.../LogPeriodicAlertE2eTest.java | Added PostgreSQL port exposure and dynamic property registration |
| script/docker-compose/.../application.yml | Added postgres-endpoint configuration for GreptimeDB |
| script/application.yml | Added postgres-endpoint configuration template |
| hertzbeat-startup/src/main/resources/application*.yml | Added postgres-endpoint configuration for test and production profiles |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...beat-warehouse/src/main/java/org/apache/hertzbeat/warehouse/db/GreptimeSqlQueryExecutor.java
Outdated
Show resolved
Hide resolved
hertzbeat-log/src/main/java/org/apache/hertzbeat/log/notice/LogSseFilterCriteria.java
Show resolved
Hide resolved
script/docker-compose/hertzbeat-postgresql-greptimedb/conf/application.yml
Outdated
Show resolved
Hide resolved
...beat-warehouse/src/main/java/org/apache/hertzbeat/warehouse/db/GreptimeSqlQueryExecutor.java
Show resolved
Hide resolved
...n/java/org/apache/hertzbeat/warehouse/store/history/tsdb/greptime/GreptimeDbDataStorage.java
Show resolved
Hide resolved
...n/java/org/apache/hertzbeat/warehouse/store/history/tsdb/greptime/GreptimeDbDataStorage.java
Show resolved
Hide resolved
...beat-warehouse/src/main/java/org/apache/hertzbeat/warehouse/db/GreptimeSqlQueryExecutor.java
Show resolved
Hide resolved
…se/db/GreptimeSqlQueryExecutor.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Logic <zqr10159@126.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Logic <zqr10159@126.com>
…lication.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Logic <zqr10159@126.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Logic <zqr10159@126.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Logic <zqr10159@126.com>
- Add comprehensive tests for log content filtering including exact match, partial match, case insensitivity, and non-match scenarios - Fix incorrect verify calls in GreptimeDbDataStorageTest from query() to delete() - Update constructor and setter tests to include log content field - Improve test coverage for null log entry bodies - Add test for multiple filter combinations including log content
- Replace System.nanoTime() with consistent nanosecond timestamp - Increase E2E test timeout from 30s to 60s for log processing - Extend query time window from 5 to 10 minutes in tests - Add logging for stored log size verification - Fix minor formatting in Javadoc comments - Update license header URL formatting
- Replace System.nanoTime() with consistent nanosecond timestamp - Increase E2E test timeout from 30s to 60s for log processing - Extend query time window from 5 to 10 minutes in tests - Add logging for stored log size verification - Fix minor formatting in Javadoc comments - Update license header URL formatting
Signed-off-by: Logic <zqr10159@dromara.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for log content search filtering and expands GreptimeDB integration to include the PostgreSQL endpoint in both E2E tests and configuration. The main changes are grouped into API enhancements for log search functionality and improvements to GreptimeDB test setup and configuration.
Log search functionality improvements:
searchparameter to log query API endpoints inLogQueryController, allowing users to filter logs by content keyword. The filtering is case-insensitive and matches substrings within log entries. [1] [2] [3] [4] [5] [6]searchparameter, ensuring that filtering logic and pagination work correctly with content-based queries. [1] [2] [3] [4] [5] [6] [7] [8] [9]GreptimeDB integration and configuration:
GREPTIME_PG_PORT, port 4003) in E2E test containers for both alert and log storage tests, including startup commands and port exposure. [1] [2] [3] [4]postgres-endpointproperty to the GreptimeDB configuration inapplication-test.ymlfor test environments.searchparameter.These changes collectively enhance log filtering capabilities for users and improve the reliability and completeness of GreptimeDB integration in the test suite.