Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

zhicwu
Copy link
Contributor

@zhicwu zhicwu commented Nov 29, 2021

This is derived from PR #747 focusing on below items:

  • basic JDBC escape syntax support
  • enhance prepared statement
    • multiple values for batch insertion
    • leverage JDBC escape syntax and external table for non-insert query(in a similar way as input function based approach)
  • share test cases for http and grpc clients
  • better support of AggregateFunction(and further optimization to bitmap)
  • implement unwrap methods to expose Java client API for JDBC users
  • enhance multi-format support(grpc is inconsistent with http)
  • fix reported issues in new JDBC driver

@github-actions
Copy link

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   268.593 ±  31.341  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   252.961 ±  24.348  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   256.464 ±  30.397  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   254.123 ±  23.342  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   271.505 ±  30.542  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   268.237 ±  30.897  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   266.413 ±  28.681  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   268.169 ±  30.725  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1462.354 ±  84.366  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1437.674 ± 109.896  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1445.086 ±  81.150  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1451.279 ±  96.342  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1011.632 ±  52.903  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   970.800 ±  85.199  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   998.666 ±  93.195  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   951.415 ±  75.458  ops/s

@zhicwu zhicwu self-assigned this Nov 29, 2021
@zhicwu zhicwu added this to the 0.3.2 Release milestone Nov 29, 2021
@zhicwu zhicwu added the jdbc-v1 JDBC driver label Nov 29, 2021
@enqueue
Copy link
Contributor

enqueue commented Nov 29, 2021

@zhicwu let me know if I can help.

@zhicwu
Copy link
Contributor Author

zhicwu commented Nov 30, 2021

@zhicwu let me know if I can help.

Thank you @enqueue as always. Could you help with below items?

  1. share test cases for all type of clients - enforce consistency and better quality
    clickhouse-grpc-client was implemented first along with a few tests, and then clickhouse-http-client with almost no test. As we'll soon add clickhouse-tcp-client(using native TCP protocol), we need an elegant way to share test cases among these modules, with the ability of adding and turning on/off specific tests for certain modules. I thought about creating a new module just for testing, like a toolkit for compatibility check, but it's inconvenient for CI and VSCode(does not support multiple maven module very well). Or shared git submodule along with a configuration file in upper directory? Perhap there's better way.

  2. review and test the Java client and new JDBC driver
    I didn't intent to rewrite everything in a minor release, but one thing led to another, I rewrote the code a few times in the past months :p I have to admit it's still kind of messy and inmature, for examples: 1) introducing ClickHouseInputStream to replace InputStream scarified usability with only 5% - 10% performance gain on tiny reads; 2) ClickHouse*Value classes should be removed/generated, they're there only because they provide better performance; 3) naming and API doc - maybe I should use google translator in the first place :D
    Please file issues to log your findings and we can address them one by one moving forward. I wouldn't mind to rewrite it again, but we'd better hold that until 0.3.3 or 0.3.2 will never be released :)

  3. add benchmarks to define ideal performance
    We need a set of benchmarks to define what's the ideal performance for reading and writing. They do not rely on any existing client, but directly connect to ClickHouse and issue query in the most efficient way.
    Based on my testing, 0.3.2 performs much better than before with less memory footprints. I think it's mainly come from RowBinary format and maybe the new read / write paths. However, we need to understand what's the limit and which lane is faster to guide future development.

@zhicwu zhicwu merged commit 1b0714a into ClickHouse:develop Nov 30, 2021
@zhicwu
Copy link
Contributor Author

zhicwu commented Nov 30, 2021

Created #769 for accidentally closed this one :p

@enqueue
Copy link
Contributor

enqueue commented Nov 30, 2021

Ad 1: In some gradle projects test fixtures are used. Perhaps a similar concept can be used in this maven project.

Ad 2: This will require some manual testing, e.g checking. how DBeaver or similar tools behave. We have added unit tests over the last couple of years, but of course they won't spot everything.

Ad 3: I don't know enough about benchmarking, so I won't be any help there. Sorry.

@zhicwu
Copy link
Contributor Author

zhicwu commented Nov 30, 2021

Ad 1: In some gradle projects test fixtures are used. Perhaps a similar concept can be used in this maven project.

Cool. Will you be able to spend some time to take care of this one?

Ad 2: This will require some manual testing, e.g checking. how DBeaver or similar tools behave. We have added unit tests over the last couple of years, but of course they won't spot everything.

Yes, I've been tried DBeaver and SQuirreL SQL. Actually I also tried DataGrip earlier but had to stop due to license expiration ;) Are you interested in code review? Any suggestion is greatly appreciated.

Ad 3: I don't know enough about benchmarking, so I won't be any help there. Sorry.

No problem. I think we can start a simple program and then add it into benchmarks.

Let me know if there's any specific item you're interested. I created a pre-release 0.3.1-test1last night, I think we just need one or two more in the coming days, and we should be able to roll out 0.3.2(in 2 or 3 weeks?).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jdbc-v1 JDBC driver

Projects

None yet

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.