Serilog.Sinks.Async spawns thousands of threads

Programming
, ,
1

We are seeing server-wide issues and it looks like thousands of threads are being blocked by the Serilog.Sinks.Async.BackgroundWorkerSink. We believe the issue is most likely something with the file sink, and are wondering:

  • Is it wise to have “shared:true” for the file sink?
  • Is it wise to combine the elastic search sink with the async sink?
  • Isn’t Serilog supposed to use the Treadpool? Why then do we have ~3000 threads?
var logCfg =
            new LoggerConfiguration()
            .Enrich.WithProperty("machine", System.Environment.MachineName)
            .WriteTo.Map(keyPropertyName: "$filename", defaultKey: "fallback",
                configure: (fileName, wt) =>
                     wt.Async(c =>
                         c.File(formatter: formatter
                         , path: logOptions.AuditPath
                         , shared: true
                         , fileSizeLimitBytes: fileSizeLimitBytes ?? 41943040
                         , rollingInterval: RollingInterval.Day
                         , rollOnFileSizeLimit: true
                         )
                )
            )
            .WriteTo.Async(c =>
                       c.Elasticsearch(new Serilog.Sinks.Elasticsearch.ElasticsearchSinkOptions(new Uri(elasticUrl))
                       {
                           IndexFormat = "my-audit-" + DateTime.Now.Year,
                           ModifyConnectionSettings = x => x.MyAuthentication(elasticCreds[0], elasticCreds[1])
                       }
                   )
            );

Nuget versions:

  • Serilog.Sinks.Async 1.5.0.0
  • Serilog.sinks.Elasticsearch 8.4.1
  • Serilog.Sinks.File 5.0.0

We are seeing server-wide issues and it looks like thousands of threads are being blocked by the Serilog.Sinks.Async.BackgroundWorkerSink. Our configuration is as follows:

var logCfg =
            new LoggerConfiguration()
            .Enrich.WithProperty("machine", System.Environment.MachineName)
            .WriteTo.Map(keyPropertyName: "$filename", defaultKey: "fallback",
                configure: (fileName, wt) =>
                     wt.Async(c =>
                         c.File(formatter: formatter
                         , path: logOptions.AuditPath
                         , shared: true
                         , fileSizeLimitBytes: fileSizeLimitBytes ?? 41943040
                         , rollingInterval: RollingInterval.Day
                         , rollOnFileSizeLimit: true
                         )
                )
            )
            .WriteTo.Async(c =>
                       c.Elasticsearch(new Serilog.Sinks.Elasticsearch.ElasticsearchSinkOptions(new Uri(elasticUrl))
                       {
                           IndexFormat = "my-audit-" + DateTime.Now.Year,
                           ModifyConnectionSettings = x => x.MyAuthentication(elasticCreds[0], elasticCreds[1])
                       }
                   )
            );

We are using a Serilog file sink and an Elasticsearch sink, both using a Serilog async sink. We are wondering:

  • Is it wise to have “shared:true” for the file sink?
  • Is it wise to combine the Elasticsearch sink with the async sink?
  • Isn’t Serilog supposed to use the Threadpool? Why then do we have ~3000 threads?

Nuget versions:

  • Serilog.Sinks.Async 1.5.0.0
  • Serilog.sinks.Elasticsearch 8.4.1
  • Serilog.Sinks.File 5.0.0
2

Based on the provided information, here are the answers to your questions:

  1. Is it wise to have “shared:true” for the file sink?

    • Yes, it is wise to have “shared:true” for the file sink when using Serilog.Sinks.File version 5.0.0. This allows multiple processes to write to the same log file.

  2. Is it wise to combine the Elasticsearch sink with the async sink?

    • Yes, it is wise to combine the Elasticsearch sink with the async sink when using Serilog.Sinks.Elasticsearch version 8.4.1. The async sink improves performance by offloading the logging operations to a separate background worker thread.

  3. Isn’t Serilog supposed to use the Threadpool? Why then do we have ~3000 threads?

    • Serilog does use the Threadpool by default, but the issue here seems to be caused by the Serilog.Sinks.Async.BackgroundWorkerSink. This sink can create multiple background worker threads which might lead to a high number of threads. You may need to investigate further to determine why there are approximately 3000 threads being blocked.