Forum OpenACS Development: Build a web server (HTTPS) in 20 lines of code

Greetings to all,

For those that like to experiment, here is a TCL extension that helps you build a Web Server in 20 lines of code: https://github.com/jerily/twebserver

In short, it is NOT an application server, it is a TCL loadable module, and it does NOT try to do things that other extensions already accomplish (or will accomplish in the future).

Here is an example without threads:


package require twebserver

proc process_request {request_dict} { return "HTTP/1.1 200 OK\n\ntest message request_dict=$request_dict\n" }

proc process_conn {conn addr port} { if { [catch { set request [::twebserver::read_conn $conn] set reply [process_request [::twebserver::parse_request $request]] ::twebserver::write_conn $conn $reply } errmsg] } { puts "error: $errmsg" } ::twebserver::close_conn $conn }

set config_dict [dict create] set server_handle [::twebserver::create_server $config_dict process_conn] ::twebserver::add_context $server_handle localhost "../certs/host1/key.pem" "../certs/host1/cert.pem" ::twebserver::add_context $server_handle www.example.com "../certs/host2/key.pem" "../certs/host2/cert.pem" ::twebserver::listen_server $server_handle 4433 ::twebserver::destroy_server $server_handle

And, here is an example with threads (requires installing Thread extension):


package require twebserver
package require Thread

set thread_script { package require twebserver proc thread_process_request {request_dict} { return "HTTP/1.1 200 OK\n\ntest message request_dict=$request_dict\n" } proc thread_process_conn {conn addr port} { after 1000 [list ::twebserver::close_conn $conn] if { [catch { set request [::twebserver::read_conn $conn] set reply [thread_process_request [::twebserver::parse_request $request]] ::twebserver::write_conn $conn $reply } errmsg] } { puts "error: $errmsg" } ::twebserver::close_conn $conn }

} set pool [::tpool::create -minworkers 5 -maxworkers 20 -idletime 40 -initcmd $thread_script]

proc process_conn {conn addr port} { global pool ::tpool::post -detached -nowait $pool [list thread_process_conn $conn $addr $port] }

set max_request_read_bytes [expr { 10 * 1024 * 1024 }] set max_read_buffer_size [expr { 1024 * 1024 }] set config_dict [dict create max_request_read_bytes $max_request_read_bytes max_read_buffer_size $max_read_buffer_size] set server_handle [::twebserver::create_server $config_dict process_conn] ::twebserver::add_context $server_handle localhost "../certs/host1/key.pem" "../certs/host1/cert.pem" ::twebserver::add_context $server_handle www.example.com "../certs/host2/key.pem" "../certs/host2/cert.pem" ::twebserver::listen_server $server_handle 4433 ::twebserver::destroy_server $server_handle

In my opinion, it is also good for educational purposes. The extension is not more than 1500 lines of C code and about half of it are the tables for encoding and decoding URI components. Furthermore, it allows you to build your framework to serve the requests prepared by twebserver without relying on the Web Server for auxiliary commands. I kind of like it. I hope you will too.

Collapse
Posted by Neophytos Demetriou on
Added return_conn command. It accepts a response dictionary (very similar to AWS api gateway response) and streams the response back to the client. In short, you can do the following to echo the request back to the client (including a file):


proc process_request {request_dict} {
    set content_type [dict get $request_dict headers content-type]
    set is_base64_encoded [dict get $request_dict isBase64Encoded]
    set body [dict get $request_dict body]
    set response [dict create \
        statusCode 200 \
        headers [dict create Content-Type $content_type] \
        multiValueHeaders {} \
        isBase64Encoded $is_base64_encoded \
        body $body]
    return $response
}

# and then use return_conn instead of write_conn, e.g.: ::twebserver::return_conn $conn $response

Collapse
Posted by Neophytos Demetriou on
I had some great feedback by Holger Ewert at the TCL wiki about this yesterday. As a result, the extension no longer blocks the event loop after the listen_server command and it no longer kills the application when there are errors. That means that you need a vwait forever after the listen_server command in the examples. It also means that you can start multiple servers if u want. The extension already supports multiple certificates for different hosts.

I must mention that the extension is still in an experimental mode and there will be more improvements, especially in the documentation, as I get a chance to review the feedback more carefully.

Collapse
Posted by Neophytos Demetriou on
First release of twebserver is out. You can find it here: v1.47.0.

Features

  • High performance web server (HTTPS) written in C and Tcl.
  • It uses a highly efficient event-driven model with fixed number of threads to manage connections.
  • It can be easily extended.
  • It is a TCL loadable module.
  • It is the absolute minimum.
  • It supports multiple certificates for different hosts (SNI).
  • Keepalive connections
  • Compression (gzip)

Benchmarks

Here are some benchmarks with and without keepalive:

With keepalive (example-best-with-native-threads.tcl - uses parse_conn/return_conn): 89 microseconds per request

# gohttpbench -v 10 -n 1000000 -c 10 -t 1000 -k "https://localhost:4433/example";

Document Path: /example Document Length: 251 bytes Concurrency Level: 10 Time taken for tests: 8.88 seconds Complete requests: 1000000 Failed requests: 0 HTML transferred: 251000000 bytes Requests per second: 112609.68 [#/sec] (mean) Time per request: 0.089 [ms] (mean) Time per request: 0.009 [ms] (mean, across all concurrent requests) HTML Transfer rate: 27602.55 [Kbytes/sec] received

Without keepalive (example-best-with-native-threads.tcl - uses parse_conn/return_conn): 1.6 millisecond per request

# gohttpbench -v 10 -n 1000000 -c 10 -t 1000 "https://localhost:4433/example";

Concurrency Level: 10 Time taken for tests: 155.66 seconds Complete requests: 1000000 Failed requests: 0 HTML transferred: 246000000 bytes Requests per second: 6424.29 [#/sec] (mean) Time per request: 1.557 [ms] (mean) Time per request: 0.156 [ms] (mean, across all concurrent requests) HTML Transfer rate: 1543.33 [Kbytes/sec] received

PS. Not bad for a month's worth of work. I will fix any issues and improve it further. All feedback is welcome. Many thanks to Holger Ewert for the constructive feedback so far.

Collapse
Posted by Neophytos Demetriou on
macOS support in the latest release (*). Download it from here: twebserver v1.47.2.

(*) Using kqueue for macOS/FreeBSD. On Linux, it is using epoll mechanism for I/O multiplexing.

Collapse
Posted by Neophytos Demetriou on
1. Just a heads up that twebserver 1.47.35 is out that fixes many previous issues and has new features: https://github.com/jerily/twebserver

2. All feedback is welcome.

3. You might want to try the following:


proc open_conn {} {
    set sock [socket localhost 8080]
    puts -nonewline $sock whatever
}

while {true} { catch {open_conn} }