I currently use Redis Timeseries for pooling and accessing data that is streamed from an external source.

@robswc Would you be willing to say more. What type of aggregation are you doing? What is the size of the data? When do you truncate the data?

Sure! (sorry for the delay). Basically just signal processing. I don't use any aggregation or truncation, just setting an expiration. Something like weather data, recording the temperature every second, then pulling the last X and processing in an application.

We use it for real time calculation of customer attributes based on user behaviour. Like total purchase in last x hours/days.
We use different aggregation types, and also down sampling feature. This is currently used at a fairly large scale.

+1!

I'd love to be pointed to a reference on how I could do this as well, as I am facing the same problem!

I'd love to be pointed to a reference on how I could do this as well, as I am facing the same problem!

What are your requirements beyond what's in redis-timeseries ? I have code implementing parts of this, but my focus is slightly different. If you'd like some pointers, I'd be glad to help.

You can review similar RFCs here: https://github.com/valkey-io/valkey-rfc. If you want to create a draft, people can update it later.

Actually the aim is to implement a superset or the API

@m2b ,

1. Does all timeseries data need to be stored in memory. ... Obviously the older the data the least frequently accessed but still needed for analysis of events.

Currently yes. I'm using the infra provided by valkey itself, which has no analog of Redis On Flash. The only alternative at the moment is better compression. A method which I'm toying with is using a columnar block type and encoding with pcodec , which according to docs should offer near the theoretical limit of compression. In addition, there's configurable decimal/significant digits per timeseries which may help in cases where full precision is not needed.

1. I would be very interested in not only querying for time filters but for values (e.g. get me all the timestamps where the metric is less than X). This is quite useful in analyzing data. For example, all the times that weather temperature dropped below 0 or rose above 50

We support redis-style range queries which have value filters of the form FILTER_BY_VALUE min max

1. Leveraging something like ZRAM in linux as a means to store the less frequently used data in compressed form.

A stretch goal is to have configurable tiered storage where less frequently accessed data uses blocks with higher levels of compression. Any OS level support will have to happen in Valkey itself.

@madolson where does this feature stand?