…e#44429)

The original paralleled cache init will causing unstable pick of cache
base path because the choice depends on the order of init which could be
different after each BE reboot. Thus, cause cache missing and duplicate
cache block across multiple caches (disk space waste).

This commit will serialize the init process of multiple cache and using
fixed order, i.e. the order explicitly declared in be conf:
file_cache_path.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

… (#44942)

…e#44429) (apache#3674)

The initialization of the file cache involves asynchronous loading logic and synchronous upgrade directories. The latter mainly handles the conversion from version1 to version2 format and some fallback logic for problematic directories, which involves a large number of directory traversals and can be very slow.

Previously, in PR apache#44429, we changed the initialization of multiple cache directories from parallel to serial to avoid the disorder caused by concurrent initialization, which led to a long cache initialization time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades and does not need to be executed on every restart. Therefore, if we detect that the version file has been successfully written, we consider the cache directory to have completed the upgrade and skip these redundant directory traversals

Of course, we could further optimize the directory traversal process to make it asynchronous and not block the BE startup. However, this would result in three concurrent operations on the file system: asynchronous loading, asynchronous updating, and lazy loading on query. This would increase code complexity, the likelihood of errors, and the difficulty of troubleshooting. Considering that old clusters are not very common and that a cluster only needs to go through such an upgrade once in its lifecycle, we assessed that this optimization would have low cost-effectiveness and decided not to pursue it.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

The initialization of the file cache involves asynchronous loading logic and synchronous upgrade directories. The latter mainly handles the conversion from version1 to version2 format and some fallback logic for problematic directories, which involves a large number of directory traversals and can be very slow.

Previously, in PR #44429, we changed the initialization of multiple cache directories from parallel to serial to avoid the disorder caused by concurrent initialization, which led to a long cache initialization time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades and does not need to be executed on every restart. Therefore, if we detect that the version file has been successfully written, we consider the cache directory to have completed the upgrade and skip these redundant directory traversals

Of course, we could further optimize the directory traversal process to make it asynchronous and not block the BE startup. However, this would result in three concurrent operations on the file system: asynchronous loading, asynchronous updating, and lazy loading on query. This would increase code complexity, the likelihood of errors, and the difficulty of troubleshooting. Considering that old clusters are not very common and that a cluster only needs to go through such an upgrade once in its lifecycle, we assessed that this optimization would have low cost-effectiveness and decided not to pursue it.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

The initialization of the file cache involves asynchronous loading logic
and synchronous upgrade directories. The latter mainly handles the
conversion from version1 to version2 format and some fallback logic for
problematic directories, which involves a large number of directory
traversals and can be very slow.

Previously, in PR #44429, we changed the initialization of multiple
cache directories from parallel to serial to avoid the disorder caused
by concurrent initialization, which led to a long cache initialization
time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades
and does not need to be executed on every restart. Therefore, if we
detect that the version file has been successfully written, we consider
the cache directory to have completed the upgrade and skip these
redundant directory traversals

Of course, we could further optimize the directory traversal process to
make it asynchronous and not block the BE startup. However, this would
result in three concurrent operations on the file system: asynchronous
loading, asynchronous updating, and lazy loading on query. This would
increase code complexity, the likelihood of errors, and the difficulty
of troubleshooting. Considering that old clusters are not very common
and that a cluster only needs to go through such an upgrade once in its
lifecycle, we assessed that this optimization would have low
cost-effectiveness and decided not to pursue it.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

The initialization of the file cache involves asynchronous loading logic
and synchronous upgrade directories. The latter mainly handles the
conversion from version1 to version2 format and some fallback logic for
problematic directories, which involves a large number of directory
traversals and can be very slow.

Previously, in PR #44429, we changed the initialization of multiple
cache directories from parallel to serial to avoid the disorder caused
by concurrent initialization, which led to a long cache initialization
time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades
and does not need to be executed on every restart. Therefore, if we
detect that the version file has been successfully written, we consider
the cache directory to have completed the upgrade and skip these
redundant directory traversals

Of course, we could further optimize the directory traversal process to
make it asynchronous and not block the BE startup. However, this would
result in three concurrent operations on the file system: asynchronous
loading, asynchronous updating, and lazy loading on query. This would
increase code complexity, the likelihood of errors, and the difficulty
of troubleshooting. Considering that old clusters are not very common
and that a cluster only needs to go through such an upgrade once in its
lifecycle, we assessed that this optimization would have low
cost-effectiveness and decided not to pursue it.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

The initialization of the file cache involves asynchronous loading logic
and synchronous upgrade directories. The latter mainly handles the
conversion from version1 to version2 format and some fallback logic for
problematic directories, which involves a large number of directory
traversals and can be very slow.

Previously, in PR apache#44429, we changed the initialization of multiple
cache directories from parallel to serial to avoid the disorder caused
by concurrent initialization, which led to a long cache initialization
time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades
and does not need to be executed on every restart. Therefore, if we
detect that the version file has been successfully written, we consider
the cache directory to have completed the upgrade and skip these
redundant directory traversals

Of course, we could further optimize the directory traversal process to
make it asynchronous and not block the BE startup. However, this would
result in three concurrent operations on the file system: asynchronous
loading, asynchronous updating, and lazy loading on query. This would
increase code complexity, the likelihood of errors, and the difficulty
of troubleshooting. Considering that old clusters are not very common
and that a cluster only needs to go through such an upgrade once in its
lifecycle, we assessed that this optimization would have low
cost-effectiveness and decided not to pursue it.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

The initialization of the file cache involves asynchronous loading logic
and synchronous upgrade directories. The latter mainly handles the
conversion from version1 to version2 format and some fallback logic for
problematic directories, which involves a large number of directory
traversals and can be very slow.

Previously, in PR apache#44429, we changed the initialization of multiple
cache directories from parallel to serial to avoid the disorder caused
by concurrent initialization, which led to a long cache initialization
time and affected the startup speed of the BE.

We found that the upgrade directory is only meaningful during upgrades
and does not need to be executed on every restart. Therefore, if we
detect that the version file has been successfully written, we consider
the cache directory to have completed the upgrade and skip these
redundant directory traversals

Of course, we could further optimize the directory traversal process to
make it asynchronous and not block the BE startup. However, this would
result in three concurrent operations on the file system: asynchronous
loading, asynchronous updating, and lazy loading on query. This would
increase code complexity, the likelihood of errors, and the difficulty
of troubleshooting. Considering that old clusters are not very common
and that a cluster only needs to go through such an upgrade once in its
lifecycle, we assessed that this optimization would have low
cost-effectiveness and decided not to pursue it.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>