Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
This repository was archived by the owner on Apr 25, 2026. It is now read-only.

Try reconnect IAM systems if failed initially - #20333

#20333
Merged
harshavardhana merged 8 commits into
minio:masterminio/minio:masterfrom
shtripat:reconnect-openidshtripat/minio:reconnect-openidCopy head branch name to clipboard
Apr 2, 2025
Merged

Try reconnect IAM systems if failed initially#20333
harshavardhana merged 8 commits into
minio:masterminio/minio:masterfrom
shtripat:reconnect-openidshtripat/minio:reconnect-openidCopy head branch name to clipboard

Conversation

@shtripat

Copy link
Copy Markdown
Contributor

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

If while start of MinIO server, some of IAM systems (e.g. OpenID/ldap) are not reachable, start a thread to check their availability runtime and re-initialize the configurations properly. This makes sure if openid/ldap comes online later, console would allow using the those creds for login.

Motivation and Context

Fixes: #20118

How to test this PR?

Follow instructions at #20118

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

Comment thread cmd/globals.go Outdated

@klauspost klauspost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL.

Comment thread cmd/server-main.go Outdated
Comment thread cmd/server-main.go Outdated
Comment thread cmd/server-main.go Outdated
Comment thread cmd/server-main.go Outdated
@harshavardhana
harshavardhana force-pushed the reconnect-openid branch 2 times, most recently from 5ce5e85 to 12f748c Compare September 7, 2024 10:46
@harshavardhana

Copy link
Copy Markdown
Member

I simplified the overall behavior @klauspost instead of having many goroutines, it is done in one self contained place PTAL.

@harshavardhana

Copy link
Copy Markdown
Member

I simplified the overall behavior @klauspost instead of having many goroutines, it is done in one self contained place PTAL.

PTAL @shtripat

@taran-p taran-p left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IAMsys stays uninitialized if one of the subsystems is down, so actions by non-root users fail and IAM mc actions hang

@harshavardhana

Copy link
Copy Markdown
Member

IAMsys stays uninitialized if one of the subsystems is down, so actions by non-root users fail and IAM mc actions hang

that is to be expected, you need to bring those endpoints online @taran-p

@harshavardhana

Copy link
Copy Markdown
Member

the idea is that those endpoints do come online lazily, and we can operate without them being online when the server is coming up.

@shtripat

shtripat commented Sep 9, 2024

Copy link
Copy Markdown
Contributor Author

I tried verifying the latest changes. With this change, when openid is down and we restart MinIO server, it keeps waiting for opendid to come online and keeps logging below error till openid comes online.

API: SYSTEM.iam
Time: 04:12:30 UTC 09/09/2024
DeploymentID: 35d9d4e7-0850-4994-ab92-423d2b5e15ba
Error: Unable to initialize OpenID: Get "http://localhost:5557/dex/.well-known/openid-configuration": dial tcp 127.0.0.1:5557: connect: connection refused (*fmt.wrapError)
       6: internal/logger/logger.go:268:logger.LogIf()
       5: cmd/logging.go:29:cmd.iamLogIf()
       4: cmd/iam.go:277:cmd.(*IAMSys).Init()
       3: cmd/server-main.go:980:cmd.serverMain.func14.1()
       2: cmd/server-main.go:561:cmd.bootstrapTrace()
       1: cmd/server-main.go:979:cmd.serverMain.func14()
INFO: Waiting for OpenID to be initialized.. (retrying in 1.280339766s)

API: SYSTEM.authZ
Time: 04:12:31 UTC 09/09/2024
DeploymentID: 35d9d4e7-0850-4994-ab92-423d2b5e15ba
Error: The policies "[readwrite]" mapped to role ARN arn:minio:iam:::role/domXb70kze7Ugc1SaxaeFchhLP4 are not defined - this role may not work as expected. (*errors.errorString)
       7: internal/logger/logger.go:268:logger.LogIf()
       6: cmd/logging.go:54:cmd.authZLogIf()
       5: cmd/iam.go:519:cmd.(*IAMSys).validateAndAddRolePolicyMappings()
       4: cmd/iam.go:366:cmd.(*IAMSys).Init()
       3: cmd/server-main.go:980:cmd.serverMain.func14.1()
       2: cmd/server-main.go:561:cmd.bootstrapTrace()
       1: cmd/server-main.go:979:cmd.serverMain.func14()

During this time console doesnt get initialized and we cannot open it. Once openid comes online everything starts working as expected.

If we dont restart MinIO while openid goes offline temporarily, the console falls back to userid/paswd based login and if we bring openid back online, it starts using the same for login.
Hope this is acceptable behavior and as

  • while start of MinIO if configured openid down, it doesnt start well
  • if openid eventually gets offline while MinIO is running the login sxreen falls back to userid/passwd based login and once openid is online again it starts using the same.

Approving the changes based on the same.

@harshavardhana

Copy link
Copy Markdown
Member
  • while start of MinIO if configured openid down, it doesnt start well
  • if openid eventually gets offline while MinIO is running the login sxreen falls back to userid/passwd based login and once openid is online again it starts using the same.

I mean it is precisely doing what we wanted to do, it isn't about not working well. SSO down is catastrophic, we cannot allow access to resources.

We could initialize the console earlier than IAM, which is okay. Then, it will work the same way as when SSO goes offline while the server is running.

@harshavardhana

Copy link
Copy Markdown
Member

We could initialize the console earlier than IAM, which is okay. Then, it will work the same way as when SSO goes offline while the server is running.

@shtripat can you move the code around for starting Console UI in server-main.go

@shtripat

Copy link
Copy Markdown
Contributor Author

We could initialize the console earlier than IAM, which is okay. Then, it will work the same way as when SSO goes offline while the server is running.

@shtripat can you move the code around for starting Console UI in server-main.go

Juts by moving the console start before IAM init, would start the UI server but still would root creds work for login till we start openid as its already configured?
Also even after successful start and init of openid, we would need to re-init the console to make sure it starts using openid creds for login.

@shtripat

Copy link
Copy Markdown
Contributor Author

With new commit now behavior is as below

  • in start when openid down, it uses root creds
  • when we bring openid up in middle, after few secs it starts using openid
  • if we bring down openid in middle, it falls back to root creds
  • if we bring back openid again, after few secs it again starts using openid creds

shtripat and others added 7 commits September 10, 2024 21:09
Fixes: minio#20118

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat

Copy link
Copy Markdown
Contributor Author

With new commit now behavior is as below

  • in start when openid down, it uses root creds
  • when we bring openid up in middle, after few secs it starts using openid
  • if we bring down openid in middle, it falls back to root creds
  • if we bring back openid again, after few secs it again starts using openid creds

Reverted this change as not too safe way to handle

@harshavardhana harshavardhana left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay with this change but will confirm once with @abperiasamy

@harshavardhana

Copy link
Copy Markdown
Member

@shtripat please move this PR to EOS only

@mircea-pavel-anton

Copy link
Copy Markdown

Hi, @harshavardhana !

I see this PR got closed, though I was very interested in this functionality being present, since currently if MinIO starts up before my IDP, then I simply lose all access to the console for my users unless I restart the k8s pod which in turn creates some downtime.

Are there any updates on this? What is the current status?

@harshavardhana

Copy link
Copy Markdown
Member

Hi, @harshavardhana !

I see this PR got closed, though I was very interested in this functionality being present, since currently if MinIO starts up before my IDP, then I simply lose all access to the console for my users unless I restart the k8s pod which in turn creates some downtime.

Are there any updates on this? What is the current status?

PR will be made available in our EOS releases.

@mircea-pavel-anton

Copy link
Copy Markdown

Hi, @harshavardhana !

I see this PR got closed, though I was very interested in this functionality being present, since currently if MinIO starts up before my IDP, then I simply lose all access to the console for my users unless I restart the k8s pod which in turn creates some downtime.

Are there any updates on this? What is the current status?

PR will be made available in our EOS releases.

So this will be an enterprise only feature?

@harshavardhana

Copy link
Copy Markdown
Member

So this will be an enterprise only feature?

Correct.

@mircea-pavel-anton

Copy link
Copy Markdown

So this will be an enterprise only feature?

Correct.

Well that's just a shame... The work was already done over here. I find it weird that SSO is NOT an enterprise only feature but rather it working properly is...

@harshavardhana

harshavardhana commented Sep 22, 2024

Copy link
Copy Markdown
Member

Well that's just a shame... The work was already done over here. I find it weird that SSO is NOT an enterprise only feature but rather it working properly is...

@mircea-pavel-anton you are free to work on it, the code is available keep it in your fork.

@harshavardhana harshavardhana reopened this Apr 2, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Reconnect OpenID configuration at startup, if fails

6 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.