Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

Describe the issue

When EnableAOF is enabled in Garnet, writing a value of around 4 MB using Microsoft.Extensions.Caching.StackExchangeRedis (IDistributedCache.SetString) causes a RedisConnectionException (SocketClosed).

If EnableAOF is disabled, the same operation works without any issue.
I’m wondering whether this behavior is expected, or if there’s some configuration I should adjust to handle larger payloads when AOF is enabled.


Steps to reproduce

  1. Enable EnableAOF in Garnet configuration.
  2. Use Microsoft.Extensions.Caching.StackExchangeRedis version 9.0.8 (with StackExchange.Redis 2.7.27).
  3. Run the following code (.net core web api):
using Microsoft.Extensions.Caching.Distributed;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "127.0.0.1:6379,DefaultDatabase=0";
});

builder.Services.AddHostedService<StartupService>();

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

public class StartupService(IDistributedCache cache, ILogger<StartupService> logger) : IHostedService
{
    public async Task StartAsync(CancellationToken cancellationToken)
    {
        //var value = new string('1', 1024 * 1024 * 4 - 88); // OK
        var value = new string('1', 1024 * 1024 * 4 - 87); // RedisConnectionException

        await cache.SetStringAsync("1", value).ContinueWith(p =>
        {
            if (p.IsFaulted)
            {
                logger.LogError($"Error setting cache");
                if (p.Exception is AggregateException ae)
                {
                    logger.LogError(ae.InnerException, $"Error setting cache");
                }
            }
        });
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}
  1. Execution will throw RedisConnectionException (SocketClosed).

Expected behavior

The value should be stored successfully without closing the connection.


Actual behavior

Connection is closed during the write operation, producing this error:

StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on 127.0.0.1:6380/Interactive, Idle/MarkProcessed, last: ECHO, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 8 of 10 available, in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.7.27.49176
at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.SetImplAsync(String key, ReadOnlySequence`1 value, DistributedCacheEntryOptions options, CancellationToken token)

Log from Garnet.worker.exe

05::54::15 crit: Session[0] [127.0.0.1:11698] [00B235F6] ProcessMessages threw an exception: Tsavorite.core.TsavoriteException: Entry does not fit on page    at Tsavorite.core.TsavoriteLog.Enqueue[THeader,TInput](THeader userHeader, SpanByte& item1, TInput& input, Int64& logicalAddress) in /_/libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLog.cs:line 1024
    at Garnet.server.ObjectSessionFunctions.WriteLogRMW(Byte[]& key, ObjectInput& input, Int64 version, Int32 sessionID) in /_/libs/server/Storage/Functions/ObjectStore/PrivateMethods.cs:line 59
    at Garnet.server.ObjectSessionFunctions.PostInitialUpdater(Byte[]& key, ObjectInput& input, IGarnetObject& value, GarnetObjectStoreOutput& output, RMWInfo& rmwInfo) in /_/libs/server/Storage/Functions/ObjectStore/RMWMethods.cs:line 86
    at Tsavorite.core.TsavoriteKV`4.CreateNewRecordRMW[TInput,TOutput,TContext,TSessionFunctionsWrapper](TKey& key, TInput& input, TValue& value, TOutput& output, PendingContext`3& pendingContext, TSessionFunctionsWrapper sessionFunctions, OperationStackContext`4& stackCtx, RecordInfo& srcRecordInfo, Boolean doingCU) in /_/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Implementation/InternalRMW.cs:line 384
    at Garnet.server.StorageSession.RMWObjectStoreOperation[TObjectContext](Byte[] key, ObjectInput& input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in /_/libs/server/Storage/Session/ObjectStore/Common.cs:line 31
    at Garnet.server.StorageSession.HashSet[TObjectContext](Byte[] key, ObjectInput& input, ObjectOutputHeader& output, TObjectContext& objectStoreContext) in /_/libs/server/Storage/Session/ObjectStore/HashOps.cs:line 381
    at Garnet.server.GarnetApi`2.HashSet(Byte[] key, ObjectInput& input, ObjectOutputHeader& output) in /_/libs/server/API/GarnetApiObjectCommands.cs:line 458    at Garnet.server.RespServerSession.HashSet[TGarnetApi](RespCommand command, TGarnetApi& storageApi) in /_/libs/server/Resp/Objects/HashCommands.cs:line 51
    at Garnet.server.RespServerSession.ProcessArrayCommands[TGarnetApi](RespCommand cmd, TGarnetApi& storageApi) in /_/libs/server/Resp/RespServerSession.cs:line 893
    at Garnet.server.RespServerSession.ProcessMessages() in /_/libs/server/Resp/RespServerSession.cs:line 596
    at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /_/libs/server/Resp/RespServerSession.cs:line 444

Environment

  • Garnet: 1.0.77 and 1.0.82 (both reproduce the issue)
  • OS: Windows 11 24H2, Windows Server 2022 Standard (both reproduce the issue)
  • Microsoft.Extensions.Caching.StackExchangeRedis: 9.0.8
    • StackExchange.Redis: 2.7.27
  • EnableAOF: true (works fine if set to false)

Questions

  • Is this an expected limitation when AOF is enabled?
  • Could it be related to Garnet’s handling of large writes in AOF mode?
  • Are there specific configurations (e.g., buffer sizes, timeouts) required for large value writes?
You must be logged in to vote

Try adjusting the PageSize, ObjectStorePageSize or AofPageSize accordingly.
In your case I think it is the AofPageSize.

Replies: 2 comments · 1 reply

Comment options

Try adjusting the PageSize, ObjectStorePageSize or AofPageSize accordingly.
In your case I think it is the AofPageSize.

You must be logged in to vote
1 reply
@sjhank
Comment options

Thank you for your helpful answer! Much appreciated.

Answer selected by sjhank
Comment options

The call stack you posted says HSET was being called, whereas your repro shows SET is being invoked. Just double checking to make sure this isn't a problem, rather is just a mismatch between your repro and your error log.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.