Connection reset by peer when using > 5 concurrent requests

[alexellis]

Is there a reason I'm missing why apache bench 'crashes' with more than 5 concurrent requests to a Kestrel hosted .NET app?

Here's my whole folder:

$ cat *
FROM microsoft/dotnet:latest
WORKDIR /root/
ADD project.json ./
RUN dotnet restore
ADD *.cs ./
RUN dotnet build
EXPOSE 5000
CMD ["dotnet", "run", "--server.urls=http://0.0.0.0:5000"]
using System;
using Microsoft.AspNetCore.Hosting;

namespace aspnetcoreapp
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var host = new WebHostBuilder()
            .UseUrls("http://0.0.0.0:5000/")
                .UseKestrel()
                .UseStartup<Startup>()
                .Build();

            host.Run();
        }
    }
}

using System;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;

namespace aspnetcoreapp
{
    public class Startup
    {
        public void Configure(IApplicationBuilder app)
        {
            app.Run(context =>
            {
                return context.Response.WriteAsync("Hello from ASP.NET Core!");
            });
        }
    }
}
{
  "version": "1.0.0-*",
  "buildOptions": {
    "emitEntryPoint": true
  },
  "dependencies": {
    "Microsoft.NETCore.App": {
      "type": "platform",
      "version": "1.0.0-rc2-3002702"
    },
    "Microsoft.AspNetCore.Server.Kestrel": "1.0.0"
  },
  "frameworks": {
    "netcoreapp1.0": { }
  }
}

Error:

$ ab -n 1000 -c 8 http://localhost:5000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection reset by peer (54)

With 5 requests:

$ ab -n 1000 -c 5 http://localhost:5000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Kestrel
Server Hostname:        localhost
Server Port:            5000

Document Path:          /
Document Length:        24 bytes

Concurrency Level:      5
Time taken for tests:   1.275 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      97000 bytes
HTML transferred:       24000 bytes
Requests per second:    784.29 [#/sec] (mean)
Time per request:       6.375 [ms] (mean)
Time per request:       1.275 [ms] (mean, across all concurrent requests)
Transfer rate:          74.29 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       4
Processing:     1    6   3.6      5      32
Waiting:        1    6   3.5      5      32
Total:          2    6   3.6      5      33

Percentage of the requests served within a certain time (ms)
  50%      5
  66%      6
  75%      7
  80%      8
  90%     10
  95%     13
  98%     18
  99%     21
 100%     33 (longest request)

[alexellis]

The application remains running, but does not error in the logs.

$ docker logs 50d0
Project root (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Hosting environment: Production
Content root path: /root/bin/Debug/netcoreapp1.0
Now listening on: http://0.0.0.0:5000
Application started. Press Ctrl+C to shut down.

[Tratcher]

related? #952

[cesarblum]

@alexellis Do you see the same when sending requests from a separate machine?

[muratg]

Putting it in 1.1 for investigation.

[alexellis]

@caesarBS sorry haven't been able to test yet.

[cesarblum]

@alexellis Did you get a chance to test this?

[muratg]

Closing for now. If you still have problems please re-file

[alexellis]

That was quick. I saw @CesarBS 's message and have just opened laptop to try it again.

[alexellis]

Issue still reproducing against localhost:

$ ab -n 1000 -c 8 http://localhost:5000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
apr_socket_recv: Connection reset by peer (54)
Total of 857 requests completed

When running from another machine on the network it appears to fulfil the workload with 0% failure.

[benaadams]

They are non-keep alive requests? (no -k switch)

[cesarblum]

@alexellis Do you see the same issue on multiple machines? I've just tried reproing once more and no luck. All requests complete successfully.

What are you running as your Docker host?

[alexellis]

Thanks for looking into this for me.

Hi @CesarBS I tried building the project on my Intel Nuc with Linux and got a segmentation fault / core dump (below). When running natively on my Mac outside of Docker engine I couldn't reproduce the issue: apr_socket_recv: Connection reset by peer (54)

Also when I connected from my Intel Nuc to my Mac running in Docker I couldn't reproduce the issue either. @benaadams -k wasn't used.

I am wondering if this is a timing issue or something introduced by using the virtual machine created by Docker for Mac?

info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.hosting.abstractions/index.json 144ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.hosting.abstractions/1.0.0/microsoft.aspnetcore.hosting.abstractions.1.0.0.nupkg
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.http/index.json 148ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.http/1.0.0/microsoft.aspnetcore.http.1.0.0.nupkg
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/index.json 150ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.extensions.options/1.0.0/microsoft.extensions.options.1.0.0.nupkg
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.extensions.fileproviders.physical/index.json 155ms
Segmentation fault (core dumped)
The command '/bin/sh -c dotnet restore' returned a non-zero code: 139

[cesarblum]

I can't reproduce on Docker for Mac either (if I understand correctly, that's your Docker host).

BTW, I had to change project.jsonto use version 1.0.0 of Microsoft.NETCore.App. I'm using CLI version 1.0.0-preview2-003121. What version are you using?

The segfault you got looks like a NuGet issue. I suggest you capture a dump and file an issue with them.

[benaadams]

May not be related, but may have some suggestions on how to investigate; and tweaks to push stress levels, they were experiencing localhost loopback congestion (linux) O_o https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/