why the plaintext test make the content-length of response is '15' in the example website ?

[AkazawaYun]

HTTP/1.1 200 OK
Content-Length: 15
Content-Type: text/plain; charset=UTF-8
Server: Example
Date: Wed, 17 Apr 2013 12:00:00 GMT

Hello, World!

"Hello, World!" is actually 13 length.
Some framword can not go pass the plaintext test, is it the reason ?

https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview

[MDA2AV]

Content-Length: 13 works fine. Maybe the example is assuming a \r\n termination

[AkazawaYun]

ok, I see that asp.net core uses 'transfer-encoding: chunked' instead of 'content-length: 13' in his test, Both the code are written, but the first one was seleted to work as the test api...I feel confused about it.
And, my framwork "akazawayun.pro" always get the result of 1090 rps at every test per week, I try to test it via JMeter on my PC, it work fine...its result is nearly to the rps of asp.netcore, i am very very confused about it..

[MDA2AV]

I tried running your benchmark locally but not working, is the request for plain text a GET to http://localhost:2022/post/plaintext?

Edit: Also just checked asp net platform test for plaintext and it returns a Content-Length : 13

[AkazawaYun]

I change it to GET http://localhost:8080/plaintext 2 hour ago, and pull request just now, but the build/verify failed..., so the code that you tried to run may be not the newest version...
I will download the old code from TechEmpower to my computer, and run my framework to confirm , wait a minute please, 3q

[AkazawaYun]

The old version akazawayun.pro also work fine, i run it with Visual Studio 2019 on windows 10, and use an ApiTestor Tool to request http://localhost:2022/post/plaintext , it got the response successfully.

https://akazawayun.cn/screenshot.jpg
https://akazawayun.cn/screenshot2.jpg

[MDA2AV]

Oh, I don't have Windows or Visual Studio, I am running it on a linux machine using curl and wrk (same tool techempower uses for load generator), can you try run

for GET case
curl http://localhost:2022/post/plaintext

for POST case
curl -X POST http://localhost:2022/post/json

or just use postman

in your command line on windows with the server running? It should work if the webserver is correctly configured. Not sure what ApiTestor Tool is. Ill try run from your PR

[MDA2AV]

Ok it is working with your changes

i built using your docker file and got for

Json test: 300k RPS
plaintext: 200k RPS

ran both with wrk with 512 concurrency

for comparison with asp net platform i get

Json test: 2.6 million RPS
plaintext: 26 million RPS

probably your framework does not support http pipelining which is used in the plaintext tests

For reference my machine is a linux-x64 with i9 14900k with 64GB 6400Mhz ram

[AkazawaYun]

oh yes, it does not support http pipelining, the next request only can be accepted after the current request's response is sent to client .
but the json test is also so slow...does the json test use the http-piplining ?

[MDA2AV]

oh yes, it does not support http pipelining, the next request only can be accepted after the current request's response is sent to client . but the json test is also so slow...does the json test use the http-piplining ?

No, the Json test does not use pipelining, one request at a time

/Desktop/Repos/wrk/scripts$ wrk -t32 -c512 -d15s http://localhost:8080/json
Running 15s test @ http://localhost:8080/json
  32 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.92ms    1.98ms  68.32ms   99.37%
    Req/Sec     8.80k   757.72    15.64k    97.18%
  4227161 requests in 15.10s, 697.42MB read
Requests/sec: 279957.34
Transfer/sec:     46.19MB

for asp net platform (aot)

 wrk -t32 -c512 -d15s http://localhost:8080/json
Running 15s test @ http://localhost:8080/json
  32 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.13ms    4.99ms 252.27ms   96.85%
    Req/Sec    79.69k    26.72k  239.64k    77.87%
  38061403 requests in 15.10s, 5.18GB read
Requests/sec: 2521190.73
Transfer/sec:    351.04MB

[AkazawaYun]

The difference is so huge !! ..cry .. can you help to copy the launch-log of my framwork to me . There is an index named concurrent of the ISemaphore component, is it value be 28 ?

[MDA2AV]

Docker logs for /json

akaza_json_log.txt

Edit: 300k isn't so bad.. better than java spring :D

[AkazawaYun]

thank you very much for the help and comfort. but still so depressed, I write this framwork is because I feel asp.net is complex and difficult to config and code when I first study how to develop a web server app, so I want to develop an easy and happy framwork to make partners develop webserver happily and quickly .

I use JMeter to test it for a long time, the result in JMeter shows that 'Throughput per second' is nearly with asp.net , even perform better at 10000 "Number of Thread (users)" and not select "Same user on each iteraion" setting in the Thread Group panel. ( asp.net appear failed connections in the two case )

wrk shows the different result with Jmeter, I need to find out the reason, and upgrade my framwork, fighting

[MDA2AV]

I am happy to help but dont feel depressed or discouraged, this is just one benchmark on a simple test, it may not be the best measure for the framework you built! 300k is a good result, also I compared with asp net platform benchmark which isn't a real framework, if we compare with asp net core mvc (controllers) the values goes down to 1.3 million plus these /json or /plaintext tests are far from a real world scenario load, you framework may perform better in those cases.

Nonetheless a tip I give you, if you want to participate in these kind of benchmarks you should test under similar circumstances, since you have a Windows machine try using docker but I am not sure how limited docker is on Windows, the best is to use on linux and don't use docker desktop.

Another thing is that i am using only 512 concurrency and these tests only create 512 connections and bombard them with keep-alive requests, maybe in Jmeter you are constantly creating new connections which is closer to a real world scenario. It could be that your webserver is not optimized for high load keep-alive connections

Edit: Plus, the results i showed you are just on my machine, wait for techempower data to make a real comparison since its hardware is more stable, or even better run them yourself in your machine

[AkazawaYun]

I run asp.net minimalAPI and my framwork both on a Linux cloud computer, it has only 2 physical core and 2Gb memory, but the result is this...
https://akazawayun.cn/wrk-linux.png

By the way , request-response on keep-alive tcp connections is faster about 10 times than create new tcp connection. For example, the rps on keep-alive is about 40k, and it decrease to 4k when change to create new tcp for each request in JMeter test my framwork.

[MDA2AV]

I run asp.net minimalAPI and my framwork both on a Linux cloud computer, it has only 2 physical core and 2Gb memory, but the result is this... https://akazawayun.cn/wrk-linux.png

By the way , request-response on keep-alive tcp connections is faster about 10 times than create new tcp connection. For example, the rps on keep-alive is about 40k, and it decrease to 4k when change to create new tcp for each request in JMeter test my framwork.

Ahh I see the issue!

When I run with 2 threads your framework has the same performance in my 16 core 32 thread machine.. you have closer performance to asp net core when number of threads is low, this shows that your framework does not scale with cpu cores

Desktop/Repos/wrk/scripts$ wrk -t2 -c512 http://localhost:8080/json
Running 10s test @ http://localhost:8080/json
  2 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.95ms  429.39us   9.56ms   90.56%
    Req/Sec   131.94k     8.87k  142.58k    84.00%
  2624208 requests in 10.04s, 432.96MB read
Requests/sec: 261449.18
Transfer/sec:     43.14MB
/Desktop/Repos/wrk/scripts$ wrk -t32 -c512 http://localhost:8080/json
Running 10s test @ http://localhost:8080/json
  32 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.54ms    5.09ms 116.56ms   99.02%
    Req/Sec     5.09k     0.96k   15.02k    81.02%
  1630763 requests in 10.10s, 269.05MB read
Requests/sec: 161470.64
Transfer/sec:     26.64MB

And sometimes with high thread count it even gets worse performance. You never noticed because you run it in a 2 core system

For comparison asp net platform with 2 thread

~/Desktop/Repos/wrk/scripts$ wrk -t2 -c512 -d5s http://localhost:8080/json
Running 5s test @ http://localhost:8080/json
  2 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   434.37us    1.06ms  52.57ms   99.67%
    Req/Sec   338.50k    18.44k  361.74k    62.00%
  3368028 requests in 5.05s, 468.95MB read
Requests/sec: 666877.74
Transfer/sec:     92.85MB

with minimal apis this value would probably be around 450-500k, still a lot closer to your.

Also if your machine is 2 core, running wrk with -t32 is not going to give you relevant data as you can see in your image, the performance is even lower due to overhead "shifting" threads since you can't run them in parallel.

As for the reason why you outscore minimal apis still in your system, my guess is you probably optimized your framework while testing in your system, this happened to me too, when i started developing my framework I was doing on a Windows machine and testing with bombardier which yields different results than running wrk in linux.