Add pagination for list projects endpoint#1266
Add pagination for list projects endpoint#1266dttung2905 wants to merge 9 commits intolakekeeper:mainlakekeeper/lakekeeper:mainfrom dttung2905:add-pagination-list-projectsdttung2905/lakekeeper:add-pagination-list-projectsCopy head branch name to clipboard
Conversation
| pub struct ListProjectsRequest { | ||
| /// Next page token for pagination | ||
| #[serde(default)] | ||
| pub page_token: Option<String>, |
There was a problem hiding this comment.
Could you use page_token: PageToken for consistency with other paginated list endpoints?
lakekeeper/crates/lakekeeper/src/api/iceberg/types.rs
Lines 26 to 33 in f323c58
This might be easier when it's a Query rather than Request, see for instance ListTablesQuery.
There was a problem hiding this comment.
Thank you for pointing it out. Let me fix it
| transaction: <Self::Transaction as Transaction<Self::State>>::Transaction<'_>, | ||
| ) -> Result<Vec<GetProjectResponse>> { | ||
| list_projects(project_ids, &mut **transaction).await | ||
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { |
There was a problem hiding this comment.
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { | |
| ) -> Result<ListProjectsResponse> { |
Please import at the top instead of using such long fully qualified syntax.
| pagination: PaginationQuery, | ||
| connection: E, | ||
| ) -> Result<Vec<GetProjectResponse>> { | ||
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { |
There was a problem hiding this comment.
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { | |
| ) -> Result<ListProjectsResponse> { |
| .map(|project| GetProjectResponse { | ||
| project_id: ProjectId::from_db_unchecked(project.project_id), | ||
| name: project.project_name, | ||
| let has_more = projects.len() > usize::try_from(page_size).unwrap_or(usize::MAX); |
There was a problem hiding this comment.
Rather do something like the following as .len() > usize::MAX cannot be true.
lakekeeper/crates/lakekeeper/src/catalog/mod.rs
Lines 198 to 200 in f323c58
| } | ||
|
|
||
| #[sqlx::test] | ||
| async fn test_list_projects_pagination(pool: sqlx::PgPool) { |
There was a problem hiding this comment.
Would be nice to also test pagination when Some(HashSet) is passed to list_projects.
There was a problem hiding this comment.
sure. That would be next in my todo list
WalkthroughAdds server-side pagination for listing projects across API, service, and Postgres layers: new ListProjectsQuery, PaginationQuery plumbing, ListProjectsResponse.next_page_token, updated trait and handler signatures, cursor-based SQL (created_at, project_id, LIMIT N+1), updated SQLx descriptor, and OpenAPI schema change. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant HTTP as HTTP Handler (list_projects)
participant Api as ApiServer
participant Cat as Catalog
participant WH as Postgres Warehouse
participant DB as Postgres
Client->>HTTP: GET /projects?pageToken=&pageSize=
HTTP->>Api: list_projects(ListProjectsQuery, context, metadata)
Api->>Cat: list_projects(project_ids?, PaginationQuery, tx)
Cat->>WH: list_projects(project_ids?, PaginationQuery, conn)
WH->>DB: SELECT ... WHERE cursor ... ORDER BY created_at, project_id LIMIT N+1
DB-->>WH: rows
WH-->>Cat: ListProjectsResponse { projects, next_page_token? }
Cat-->>Api: ListProjectsResponse
Api-->>HTTP: ListProjectsResponse
HTTP-->>Client: 200 OK { projects, next_page_token? }
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Assessment against linked issues
Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (1)
docs/docs/api/management-open-api.yaml (1)
1022-1041: Document pagination request parameters on list-projectsThe endpoint advertises a paginated response but does not declare pageToken/pageSize query parameters. Please add them for parity with list-users/list-roles.
Apply this diff under /management/v1/project-list -> get:
/management/v1/project-list: get: tags: - project summary: List Projects description: Lists all projects that the requesting user has access to. operationId: list_projects + parameters: + - name: pageToken + in: query + description: Next page token + required: false + schema: + type: + - string + - 'null' + - name: pageSize + in: query + description: | + Signals an upper bound of the number of results that a client will receive. + Default: 100 + required: false + schema: + type: integer + format: int64 responses: '200': description: List of projects content: application/json: schema: $ref: '#/components/schemas/ListProjectsResponse'
♻️ Duplicate comments (2)
crates/lakekeeper/src/api/management/v1/project.rs (1)
50-61: Consistent pagination query type (PageToken) — goodUsing PageToken with serde/utoipa annotations aligns with other paginated endpoints. Defaulting page_size via management::v1::default_page_size is consistent.
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
442-444: Import ListProjectsResponse to shorten the return typeSame as earlier feedback: prefer importing the type over fully qualifying.
- ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { + ) -> Result<ListProjectsResponse> { list_projects(project_ids, pagination, &mut **transaction).await }Add this import near the top:
use crate::api::management::v1::project::ListProjectsResponse;
🧹 Nitpick comments (5)
.sqlx/query-ac395ee12ac90d563f6ed2b5afe52c2b2e8f6f0bce9792aae0a253a729d3fe16.json (1)
3-3: Keyset pagination query looks correctThe WHERE clause and ORDER BY match the cursor fields; semantics are good for stable keyset pagination and first-page behavior with null token. Two small recommendations:
- Explicitly use ORDER BY created_at ASC, project_id ASC for readability.
- Ensure an index exists to support this cursor: e.g. CREATE INDEX ON project (created_at ASC, project_id ASC).
crates/lakekeeper/src/service/catalog.rs (1)
598-602: Shorten the return type by importing ListProjectsResponseThe long fully qualified path is harder to scan. Import the type and use it directly.
- ) -> Result<crate::api::management::v1::project::ListProjectsResponse>; + ) -> Result<ListProjectsResponse>;Add this import near the other use lines:
use crate::api::management::v1::project::ListProjectsResponse;crates/lakekeeper/src/implementations/postgres/warehouse.rs (3)
455-457: Consider more robust architecture handling.While the expect message is descriptive, consider using a more graceful approach:
- let page_as_usize: usize = page_size - .try_into() - .expect("should be running on at least 32 bit architecture"); + let page_as_usize: usize = page_size + .try_into() + .map_err(|_| ErrorModel::internal( + "Page size exceeds platform limits", + "PageSizeConversionError", + None, + ))?;
416-416: Fix formatting issues.The static analysis tool detected formatting inconsistencies.
Run
cargo fmtto fix the formatting issues at lines 416 and 475.Also applies to: 475-475
1283-1353: Comprehensive pagination test coverage!The test thoroughly validates:
- Multi-page pagination with page_size=2
- Correct ordering of results across pages (addressing previous review feedback)
- Next page token generation and consumption
- Completeness of paginated results
Note: Fix the formatting issue at line 1335 by running
cargo fmt.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
.sqlx/query-04c6fd4a25005469a8342fe751db0302f8ff81b7046efaa73c0d2433e1867780.json(0 hunks).sqlx/query-ac395ee12ac90d563f6ed2b5afe52c2b2e8f6f0bce9792aae0a253a729d3fe16.json(1 hunks)crates/lakekeeper/src/api/management/mod.rs(1 hunks)crates/lakekeeper/src/api/management/v1/project.rs(4 hunks)crates/lakekeeper/src/implementations/postgres/catalog.rs(2 hunks)crates/lakekeeper/src/implementations/postgres/warehouse.rs(6 hunks)crates/lakekeeper/src/service/catalog.rs(1 hunks)docs/docs/api/management-open-api.yaml(1 hunks)
💤 Files with no reviewable changes (1)
- .sqlx/query-04c6fd4a25005469a8342fe751db0302f8ff81b7046efaa73c0d2433e1867780.json
🧰 Additional context used
🧬 Code Graph Analysis (4)
crates/lakekeeper/src/api/management/mod.rs (3)
crates/lakekeeper/src/api/management/v1/project.rs (1)
list_projects(232-256)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(438-444)crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
list_projects(414-487)
crates/lakekeeper/src/implementations/postgres/catalog.rs (4)
crates/lakekeeper/src/api/management/v1/project.rs (2)
get_project(170-202)list_projects(232-256)crates/lakekeeper/src/implementations/postgres/warehouse.rs (2)
get_project(238-270)list_projects(414-487)crates/lakekeeper/src/service/catalog.rs (3)
get_project(589-592)transaction(70-70)list_projects(597-601)crates/lakekeeper/src/api/management/mod.rs (1)
list_projects(565-571)
crates/lakekeeper/src/api/management/v1/project.rs (4)
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(438-444)crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
list_projects(414-487)crates/lakekeeper/src/service/catalog.rs (1)
list_projects(597-601)crates/lakekeeper/src/api/management/mod.rs (1)
list_projects(565-571)
crates/lakekeeper/src/implementations/postgres/warehouse.rs (4)
crates/lakekeeper/src/service/mod.rs (1)
from_db_unchecked(203-205)crates/lakekeeper/src/api/management/v1/project.rs (1)
list_projects(232-256)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(438-444)crates/lakekeeper/src/service/catalog.rs (2)
list_projects(597-601)begin_read(64-64)
🪛 GitHub Check: check-format
crates/lakekeeper/src/implementations/postgres/catalog.rs
[warning] 435-435:
Diff in /home/runner/work/lakekeeper/lakekeeper/crates/lakekeeper/src/implementations/postgres/catalog.rs
crates/lakekeeper/src/implementations/postgres/warehouse.rs
[warning] 475-475:
Diff in /home/runner/work/lakekeeper/lakekeeper/crates/lakekeeper/src/implementations/postgres/warehouse.rs
[warning] 416-416:
Diff in /home/runner/work/lakekeeper/lakekeeper/crates/lakekeeper/src/implementations/postgres/warehouse.rs
[warning] 1335-1335:
Diff in /home/runner/work/lakekeeper/lakekeeper/crates/lakekeeper/src/implementations/postgres/warehouse.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: docker / docker
- GitHub Check: test
- GitHub Check: sqlx-check
- GitHub Check: docker / docker
- GitHub Check: clippy
- GitHub Check: check-generated-openapi-matches
- GitHub Check: docker / docker
- GitHub Check: docker / docker
🔇 Additional comments (9)
crates/lakekeeper/src/api/management/v1/project.rs (3)
62-70: Helper to convert to PaginationQuery is cleanpagination_query() correctly preserves token and page_size. This keeps service-layer code tidy.
76-79: next_page_token on ListProjectsResponseAdding next_page_token (kebab-case) makes the response shape match documented behavior.
232-256: Service wiring passes AuthZ filter and pagination throughThe handler propagates AuthZ-scoped project_ids and a PaginationQuery to the catalog, returning the catalog’s paginated response as-is. Looks good.
docs/docs/api/management-open-api.yaml (1)
3260-3264: Schema addition for next-page-token is correctListProjectsResponse now exposes next-page-token; matches the Rust response (kebab-case).
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
430-435: Correct mapping to service-layer GetProjectResponseMapping project_name -> name resolves the field rename between API-layer and service-layer types.
crates/lakekeeper/src/api/management/mod.rs (1)
568-570: LGTM! Pagination parameters correctly integrated.The changes properly wire up the pagination query parameters through the axum Query extractor and pass them to the service layer, following the established pattern for paginated endpoints.
crates/lakekeeper/src/implementations/postgres/warehouse.rs (3)
414-418: Pagination implementation looks good!The function signature correctly accepts pagination parameters and returns the paginated response type.
420-450: Well-implemented cursor-based pagination logic.The implementation correctly:
- Enforces page size limits via CONFIG
- Parses pagination tokens with proper error handling
- Uses a stable cursor with (created_at, project_id) for consistent ordering
- Fetches an extra record to determine if more pages exist
889-952: Test updates correctly adapted to pagination API.All test modifications properly handle the new paginated response structure while maintaining the same test coverage and assertions.
| @@ -1,5 +1,5 @@ | ||
| use std::collections::{HashMap, HashSet}; | ||
|
|
||
| use axum::routing::options; |
There was a problem hiding this comment.
Remove unused import (may break builds with deny(warnings))
use axum::routing::options; is unused in this module. Please remove it.
-use axum::routing::options;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| use axum::routing::options; |
🤖 Prompt for AI Agents
In crates/lakekeeper/src/implementations/postgres/catalog.rs around line 2, the
import use axum::routing::options; is unused and will cause a warning/failure
under deny(warnings); remove that import line from the file; if the symbol is
needed later, replace with a used import or reference it where required,
otherwise delete the unused use statement so the module compiles without
warnings.
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
a247b35 to
8e9b72d
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
441-444: Import the return type; avoid fully-qualified path in signaturePrefer importing the type at the top for readability and consistency (echoing earlier feedback).
Apply this diff to the signature:
- ) -> Result<crate::api::management::v1::project::ListProjectsResponse> { + ) -> Result<ListProjectsResponse> {And add this import near the other
crate::apiimports:use crate::api::management::v1::project::ListProjectsResponse;crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
454-457: Simplify has_more calculation and avoid lossy conversionNo need to convert
page_sizetousize. Compare in i64 space to avoid the expect and make intent clearer.Apply this diff:
- let page_as_usize: usize = page_size - .try_into() - .expect("should be running on at least 32 bit architecture"); - let has_more = projects.len() > page_as_usize; + let has_more = (projects.len() as i64) > page_size;
🧹 Nitpick comments (3)
crates/lakekeeper/src/implementations/postgres/warehouse.rs (3)
435-441: Add composite index to support the pagination ORDER BYTo keep this query efficient at scale, ensure there is a composite index on
(created_at, project_id)on theprojecttable. Without it, cursor scans will degrade to sequential scans under load.Would you like me to open a follow-up issue or migration snippet to add this index?
886-896: LGTM: tests updated for paginated return shapeTests now unwrap the ListProjectsResponse correctly and validate membership. Consider extracting a small helper for constructing a default PaginationQuery to reduce repetition.
Also applies to: 910-920, 932-942
1281-1362: Add tests for filtered pagination and uneven last pageGreat coverage for the unfiltered, even-split case. Two additions would harden this:
- Filtered pagination: pass
Some(HashSet)and paginate to verify the WHERE clause and ordering behave with a filter.- Uneven page: e.g., 5 projects with page-size=2 to validate three pages (2,2,1) and a final None token.
I can draft these tests if helpful.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
.sqlx/query-04c6fd4a25005469a8342fe751db0302f8ff81b7046efaa73c0d2433e1867780.json(0 hunks).sqlx/query-ac395ee12ac90d563f6ed2b5afe52c2b2e8f6f0bce9792aae0a253a729d3fe16.json(1 hunks)crates/lakekeeper/src/api/management/mod.rs(1 hunks)crates/lakekeeper/src/api/management/v1/project.rs(4 hunks)crates/lakekeeper/src/implementations/postgres/catalog.rs(1 hunks)crates/lakekeeper/src/implementations/postgres/warehouse.rs(6 hunks)crates/lakekeeper/src/service/catalog.rs(1 hunks)docs/docs/api/management-open-api.yaml(1 hunks)
💤 Files with no reviewable changes (1)
- .sqlx/query-04c6fd4a25005469a8342fe751db0302f8ff81b7046efaa73c0d2433e1867780.json
🚧 Files skipped from review as they are similar to previous changes (4)
- .sqlx/query-ac395ee12ac90d563f6ed2b5afe52c2b2e8f6f0bce9792aae0a253a729d3fe16.json
- docs/docs/api/management-open-api.yaml
- crates/lakekeeper/src/service/catalog.rs
- crates/lakekeeper/src/api/management/mod.rs
🧰 Additional context used
🧬 Code Graph Analysis (3)
crates/lakekeeper/src/implementations/postgres/catalog.rs (4)
crates/lakekeeper/src/api/management/v1/project.rs (2)
get_project(169-201)list_projects(231-255)crates/lakekeeper/src/implementations/postgres/warehouse.rs (2)
get_project(238-270)list_projects(414-484)crates/lakekeeper/src/service/catalog.rs (3)
get_project(589-592)transaction(70-70)list_projects(597-601)crates/lakekeeper/src/api/management/mod.rs (1)
list_projects(565-571)
crates/lakekeeper/src/api/management/v1/project.rs (4)
crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
list_projects(414-484)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(438-444)crates/lakekeeper/src/service/catalog.rs (1)
list_projects(597-601)crates/lakekeeper/src/api/management/mod.rs (1)
list_projects(565-571)
crates/lakekeeper/src/implementations/postgres/warehouse.rs (4)
crates/lakekeeper/src/service/mod.rs (1)
from_db_unchecked(203-205)crates/lakekeeper/src/api/management/v1/project.rs (1)
list_projects(231-255)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(438-444)crates/lakekeeper/src/service/catalog.rs (2)
list_projects(597-601)begin_read(64-64)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: check-generated-openapi-matches
- GitHub Check: test
- GitHub Check: sqlx-check
🔇 Additional comments (8)
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
430-435: AI summary mismatch: this maps to service::GetProjectResponse (name), not API GetProjectResponse (project_name)Code maps
warehouse::get_project(API shape withproject_name) intoservice::GetProjectResponse { name: ... }. The AI summary claims the opposite. If this mapping is intentional to keep the Catalog trait stable, this is fine. Otherwise, update the return type/imports accordingly.crates/lakekeeper/src/api/management/v1/project.rs (4)
15-19: LGTM: standardized pagination typesSwitch to PageToken and PaginationQuery is consistent with other endpoints and addresses prior feedback.
49-69: LGTM: ListProjectsQuery shape and helperQuery params and the
pagination_query()helper look correct and consistent.
76-78: LGTM: next_page_token included in responseResponse now carries the cursor; naming aligns with kebab-case conventions.
231-255: LGTM: endpoint plumbing for paginated list-projectsAuthZ filter + direct pass-through to Catalog with PaginationQuery is clean and correct.
crates/lakekeeper/src/implementations/postgres/warehouse.rs (3)
10-11: LGTM: import relocationUsing API-layer GetProjectResponse and new ListProjectsResponse here is appropriate for this module.
265-266: LGTM: field rename alignmentMapping to
project_namematches the updated API struct.
458-473: LGTM: correct cursor emission (uses last included row after trimming)Pop extra row and derive the token from the last included row ensures no gaps/duplicates across pages.
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
443-447: Nit: align param naming with other paginated methodsOther methods in this impl (e.g., list_tables, list_views, list_tabulars) use pagination_query for the parameter name. Consider the same here for consistency.
Apply this diff:
- async fn list_projects( - project_ids: Option<HashSet<ProjectId>>, - pagination: PaginationQuery, + async fn list_projects( + project_ids: Option<HashSet<ProjectId>>, + pagination_query: PaginationQuery, transaction: <Self::Transaction as Transaction<Self::State>>::Transaction<'_>, ) -> Result<ListProjectsResponse> { - list_projects(project_ids, pagination, &mut **transaction).await + list_projects(project_ids, pagination_query, &mut **transaction).await }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
crates/lakekeeper/src/implementations/postgres/catalog.rs(2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
crates/lakekeeper/src/implementations/postgres/catalog.rs (5)
crates/lakekeeper/src/service/catalog.rs (3)
get_project(589-592)transaction(70-70)list_projects(597-601)crates/lakekeeper/src/api/management/v1/project.rs (2)
get_project(169-201)list_projects(231-255)crates/lakekeeper/src/implementations/postgres/warehouse.rs (2)
get_project(238-270)list_projects(414-484)crates/lakekeeper/src/service/authz/mod.rs (2)
list_projects(845-847)ListProjectsResponse(147-152)crates/lakekeeper/src/service/mod.rs (2)
ProjectId(146-146)ProjectId(164-206)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: docker / docker
- GitHub Check: test
- GitHub Check: sqlx-check
- GitHub Check: check-generated-openapi-matches
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: docker / docker
🔇 Additional comments (2)
crates/lakekeeper/src/implementations/postgres/catalog.rs (2)
34-37: Importing ListProjectsResponse here is the right cleanupBrings the type into scope and avoids FQNs. Also looks like the previously flagged unused axum import is gone—thanks for addressing that.
433-438: Field rename mapping verified
- The service’s GetProjectResponse (src/service/catalog.rs:149) defines
name, notproject_name.- The Postgres query returns a row with
project_name, and catalog.rs correctly mapsresponse.project_nameintoGetProjectResponse.name.- A repository-wide search shows no remaining initializations using the legacy
project_namefield on GetProjectResponse.All aligned—no further changes needed.
|
@mooori Could you help to review it again pls 🙏 ? |
| #[serde(default = "crate::api::management::v1::default_page_size")] | ||
| pub page_size: i64, | ||
| } | ||
| impl ListProjectsQuery { | ||
| #[must_use] | ||
| pub fn pagination_query(&self) -> PaginationQuery { | ||
| PaginationQuery { | ||
| page_token: self.page_token.clone(), | ||
| page_size: Some(self.page_size), | ||
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Other List*Querys have page_size: Option<i64> and then implement From<ListXQuery> for PaginationQuery. I think ListProjectsQuery should handle that in the same way. Example
| Query(request): Query<crate::api::management::v1::project::ListProjectsQuery>, | ||
| ) -> Result<ListProjectsResponse> { | ||
| ApiServer::<C, A, S>::list_projects(api_context, metadata).await | ||
| ApiServer::<C, A, S>::list_projects(request, api_context, metadata).await |
There was a problem hiding this comment.
| ApiServer::<C, A, S>::list_projects(request, api_context, metadata).await | |
| ApiServer::<C, A, S>::list_projects(query, api_context, metadata).await |
| async fn list_projects<C: Catalog, A: Authorizer + Clone, S: SecretStore>( | ||
| AxumState(api_context): AxumState<ApiContext<State<A, C, S>>>, | ||
| Extension(metadata): Extension<RequestMetadata>, | ||
| Query(request): Query<crate::api::management::v1::project::ListProjectsQuery>, |
There was a problem hiding this comment.
nit: import ListProjectsQuery at the top of the file
| get_project(project_id, transaction).await.map(|opt| { | ||
| opt.map(|response| GetProjectResponse { | ||
| project_id: response.project_id, | ||
| name: response.project_name, | ||
| }) | ||
| }) |
There was a problem hiding this comment.
Why is it now returning a different GetProjectResponse such that the map is needed?
| api::{ | ||
| iceberg::v1::PaginationQuery, | ||
| management::v1::{ | ||
| project::{GetProjectResponse, ListProjectsResponse}, |
There was a problem hiding this comment.
I think importing GetProjectResponse here from api instead of service causes get_project to return the wrong GetProjectResponse.
| None, | ||
| PaginationQuery { | ||
| page_size: None, | ||
| page_token: crate::api::iceberg::v1::PageToken::NotSpecified, |
There was a problem hiding this comment.
| page_token: crate::api::iceberg::v1::PageToken::NotSpecified, | |
| page_token: PageToken::NotSpecified, |
| Some(HashSet::from_iter(vec![project_id_1.clone()])), | ||
| PaginationQuery { | ||
| page_size: None, | ||
| page_token: crate::api::iceberg::v1::PageToken::NotSpecified, |
There was a problem hiding this comment.
| page_token: crate::api::iceberg::v1::PageToken::NotSpecified, | |
| page_token: PageToken::NotSpecified, |
| let page_as_usize: usize = page_size | ||
| .try_into() | ||
| .expect("should be running on at least 32 bit architecture"); | ||
| let has_more = projects.len() > page_as_usize; |
| pagination: PaginationQuery, | ||
| transaction: <Self::Transaction as Transaction<Self::State>>::Transaction<'_>, | ||
| ) -> Result<Vec<GetProjectResponse>>; | ||
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse>; |
There was a problem hiding this comment.
| ) -> Result<crate::api::management::v1::project::ListProjectsResponse>; | |
| ) -> Result<ListProjectsResponse>; |
| assert!(page2.next_page_token.is_none()); | ||
|
|
||
| // Paginated results are return in the expected order | ||
| let all_projects = list_projects( |
There was a problem hiding this comment.
The ordering for paginated project is (created_at, project_id). So the expected order corresponds to project_ids (ids are pushed there after creation).
So I would remove all_projects and just check something like
assert_eq!(page1.projects[0], project_ids[0]);
...
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
🚨 PR Title Needs FormattingThe title of this PR needs to be formatted correctly.
We only use types: |
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
crates/lakekeeper/src/api/management/mod.rs (1)
26-28: Use the imported type instead of a fully-qualified path in the handler.You already import
ListProjectsQuery; use it directly for consistency/nit.- Query(query): Query<crate::api::management::v1::project::ListProjectsQuery>, + Query(query): Query<ListProjectsQuery>,Also applies to: 568-571
crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
454-457: Avoid lossy i64→usize conversion; compare in i64 directly.Converting
page_size: i64tousizeis unnecessary and adds an infallibility assumption. Compare ini64to keep types aligned.- let page_as_usize: usize = page_size - .try_into() - .expect("should be running on at least 32 bit architecture"); - let has_more = projects.len() > page_as_usize; + let has_more = (projects.len() as i64) > page_size;
🧹 Nitpick comments (4)
crates/lakekeeper/src/api/management/mod.rs (1)
553-564: Document query params in OpenAPI for list_projects.The endpoint is now query-driven but the OpenAPI annotation doesn’t declare params. Add
params(ListProjectsQuery)to exposepageToken/pageSizein the spec (consistent with other list endpoints like users/roles).#[utoipa::path( get, tag = "project", path = ManagementV1Endpoint::ListProjects.path(), + params(ListProjectsQuery), responses( (status = 200, description = "List of projects", body = ListProjectsResponse), (status = "4XX", body = IcebergErrorResponse), ) )]crates/lakekeeper/src/api/management/v1/project.rs (1)
49-59: Harden query deserialization and keep it consistent with other endpoints.
- Consider defaulting
page_tokento avoid deserialization errors when omitted.- Optionally skip serializing
page_sizewhenNoneto match other list queries.#[derive(Debug, Clone, Deserialize, utoipa::IntoParams)] #[serde(rename_all = "camelCase")] pub struct ListProjectsQuery { /// Next page token for pagination - #[serde(skip_serializing_if = "PageToken::skip_serialize")] + #[serde(skip_serializing_if = "PageToken::skip_serialize")] + #[serde(default)] #[param(value_type=String)] pub page_token: PageToken, /// Signal an upper bound of the number of results that a client will receive /// Default: 100 - pub page_size: Option<i64>, + #[serde(skip_serializing_if = "Option::is_none")] + pub page_size: Option<i64>, }crates/lakekeeper/src/implementations/postgres/warehouse.rs (2)
442-449: Optional: bind UUIDs directly instead of strings forANY($1).You currently pass
Vec<String>of UUIDs. Binding asVec<uuid::Uuid>(and optionally castingANY($1::uuid[])) avoids string conversions and can help the planner. Only do this ifProjectIdexposes the inner UUID cheaply.
1279-1325: Tests cover happy-path pagination and ordering; slight flakiness risk remains.Order is
(created_at, project_id). If two rows share the samecreated_atat DB precision, ordering falls back toproject_idwhich may not match insertion order. This is unlikely but possible. If this ever flakes, consider ensuring distinctcreated_at(e.g., tiny sleep) or relaxing the exact index-based assertions.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (5)
crates/lakekeeper/src/api/management/mod.rs(2 hunks)crates/lakekeeper/src/api/management/v1/project.rs(4 hunks)crates/lakekeeper/src/implementations/postgres/catalog.rs(2 hunks)crates/lakekeeper/src/implementations/postgres/warehouse.rs(5 hunks)crates/lakekeeper/src/service/catalog.rs(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/lakekeeper/src/implementations/postgres/catalog.rs
🧰 Additional context used
🧬 Code graph analysis (3)
crates/lakekeeper/src/api/management/mod.rs (4)
crates/lakekeeper/src/api/management/v1/project.rs (1)
list_projects(229-249)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(436-442)crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
list_projects(414-483)crates/lakekeeper/src/service/catalog.rs (1)
list_projects(600-604)
crates/lakekeeper/src/api/management/v1/project.rs (4)
crates/lakekeeper/src/api/management/mod.rs (2)
from(1548-1553)list_projects(565-571)crates/lakekeeper/src/implementations/postgres/warehouse.rs (2)
from(687-692)list_projects(414-483)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(436-442)crates/lakekeeper/src/service/catalog.rs (1)
list_projects(600-604)
crates/lakekeeper/src/implementations/postgres/warehouse.rs (3)
crates/lakekeeper/src/api/management/v1/project.rs (2)
list_projects(229-249)from(61-66)crates/lakekeeper/src/implementations/postgres/catalog.rs (1)
list_projects(436-442)crates/lakekeeper/src/service/catalog.rs (2)
list_projects(600-604)begin_read(67-67)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: docker / docker
- GitHub Check: test
- GitHub Check: clippy
- GitHub Check: sqlx-check
- GitHub Check: check-generated-openapi-matches
🔇 Additional comments (5)
crates/lakekeeper/src/api/management/mod.rs (1)
570-571: Forwarding query to ApiServer looks correct.The new call signature
ApiServer::<C, A, S>::list_projects(query, api_context, metadata)aligns with the trait change. LGTM.crates/lakekeeper/src/service/catalog.rs (1)
28-32: Cataloglist_projectssignature and usage verified
- Confirmed the
Catalogtrait inservice/catalog.rsdefinesasync fn list_projects(project_ids: Option<HashSet<ProjectId>>, pagination: PaginationQuery, …)and that the Postgres implementation inimplementations/postgres/catalog.rsmatches this updated signature.- Inspected all direct invocations of
C::list_projects(e.g., insrc/api/management/v1/project.rs) andPostgresCatalog::list_projects(inimplementations/postgres/warehouse.rs) to ensure the newpaginationparameter (and transaction argument where required) are passed.- No other
Catalogimplementors or mocks were found in the codebase; alternate backends or in-memory mocks are not present.All backends and call sites have been updated correctly.
crates/lakekeeper/src/api/management/v1/project.rs (2)
74-76: Response shape with next_page_token is clear and matches the SQL-layer behavior.No issues spotted.
245-249: Plumbing pagination through to Catalog and returningListProjectsResponseis correct.Transaction handling and authZ filter look good.
crates/lakekeeper/src/implementations/postgres/warehouse.rs (1)
414-483: Cursor-based pagination implementation looks solid.
- Correct keyset condition on
(created_at, project_id)withLIMIT page_size + 1.- Properly trims the sentinel and constructs the next-page token from the last visible row.
Hi team,
My goal is to add pagination to list projects endpoint
I have also tested out with a minimal example
Partially fixes #812