How to get data as a stream from a WebAPI (.NET)
A remote web API can be consumed as a stream, which can take longer, depends an other server process, and the necessary time to generate the requested content.
It is a real scenario, if we get event, data pieces from a services, etc.
In this project I will demonstrate, how we can use a streaming API, how we can iterate through on the incoming dataset with the IAsyncEnumerable interface.
This controller generates a set of weather data, than send the dataset back to the client as a stream. Every data item serialized into Json format, and finally it sends '[DONE]' string to indicate, the streaming is over.
[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
private static readonly string[] Summaries = new[]
{
"Freezing", "Bracing", "Chilly", "Cool", "Mild",
"Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};
private readonly ILogger<WeatherForecastController> _logger;
public WeatherForecastController(ILogger<WeatherForecastController> logger)
{
_logger = logger;
}
[HttpGet]
public async Task GetAsync()
{
Response.StatusCode = 200;
Response.ContentType = "text/html";
List<WeatherForecast> list = Enumerable.Range(1, 5)
.Select(index => new WeatherForecast
{
Date = DateTime.Now.AddDays(index),
TemperatureC = Random.Shared.Next(-20, 55),
Summary = Summaries[Random.Shared.Next(Summaries.Length)]
})
.ToList();
StreamWriter sw;
await using ((sw = new StreamWriter(Response.Body))
.ConfigureAwait(false))
{
foreach (WeatherForecast item in list)
{
// Thread.Sleep simulates a long running process,
// which generates some kind of output
Thread.Sleep(1000);
await sw.WriteLineAsync(item.ToString()).ConfigureAwait(false);
await sw.FlushAsync().ConfigureAwait(false);
}
await sw.WriteLineAsync("[DONE]").ConfigureAwait(false);
};
}
}On the client side, we need a caller code, which can read and handle the incoming data. Check out the 'yield' instruction when passing every incoming data item towards the API caller. If the code detects the '[DONE]' maker, the reading ends.
public class Api
{
public async IAsyncEnumerable<string> StreamedAsync([System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "https://localhost:7176/weatherforecast"))
{
using (HttpClient httpClient = new HttpClient())
{
//httpClient.Timeout = Timeout.InfiniteTimeSpan;
using (HttpResponseMessage response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false))
{
if (response.IsSuccessStatusCode)
{
using (Stream contentStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
using (StreamReader reader = new StreamReader(contentStream))
{
string? line = null;
while ((line = await reader.ReadLineAsync().ConfigureAwait(false)) != null &&
!cancellationToken.IsCancellationRequested)
{
if ("[DONE]".Equals(line.Trim())) break;
if (!string.IsNullOrWhiteSpace(line))
{
yield return line;
}
}
}
}
}
else
{
string? jsonResult = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
yield return jsonResult;
}
}
}
}
}
}Finally, a code can use the Api class to acquire the data, using IAsyncEnumerable on a proper way.
This code iterates on the incoming data asynchronously and display the given data as a text, one by one, after a data item arrived. This approch does not wait to arrive all of the data, it immediatelly displays what we got from the server.
static async Task Main(string[] args)
{
Console.WriteLine("Press a key to start");
Console.ReadKey();
Api api = new Api();
await foreach (string line in api.StreamedAsync(CancellationToken.None))
{
Console.WriteLine(line);
}
Console.WriteLine("[DONE] press a key to exit");
Console.ReadKey();
}