Showtime: library to be proposed as Boost.Http!

It’s been two months already since my last post on this blog. All this time (and more), I’ve been (among other tasks) working on my 2014 GSoC project, an HTTP server proposal to Boost. I’ve finally reached a point where I feel it’s ready for major review/feedback.

If you’re a C++ programmer, a native speaker or an HTTP researcher (or just a little of everything) and you want to help, I’d like to ask you to review the project (interface-wise) and give me feedback.

You can find all the documentation and code at github.

Experience

This isn’t my first time on GSoC, but the experience was very different. The communities, development model, targeted audience, knowledge domain, to name a few, were completely different. Also, this year’s project wasn’t as challenging as the last one (although this is very subjective).

I improved as a programmer, but this is limiting. Once you become able to write algorithms for turing-machine compatible devices, there isn’t much room for improvement and you need to hunt other skills to continue improving (e.g. security, parallelism…). Coding for the sake of coding solves no problems. I decided to take a break (not exactly now, but in a few weeks) to make a shift and start to increase the efforts I dedicate to my academic skills.

Next step

I aim to integrate this library into Boost, so I still want to invest effort in this project.

GSoC 2014/Boost

I was accepted for GSoC 2014 and I’ll be working on the Boost project.

I created a new category on this blog to track the progress, so you’ll be able to have a separate rss feed for these posts. The new category URL is https://vinipsmaker.wordpress.com/category/computacao/gsoc2014-boost/.

Palestra sobre o Tufão no FISL14

Tive uma palestra aceita no FISL14 e o tema da palestra é a framework Tufão.

Espero mostrar rapidamente o histórico do projeto e usar o tempo restante para a motivação, reservando 10 minutos ao final para perguntas. A motivação deve incluir uma pequena explicação de como funciona a web e seus principais pontos, comparações, becnhmarks (esse é um item mais difícil), limitações, exemplos e um “tutorial”.

Devo preparar uma nova aplicação de exemplo que explora características presentes no Tufão.

Tufão 1.0.0

After a long time developing Tufão, it finally reached 1.0 version some hours ago. I’ve spent a lot of time cleaning up the API and exploring the features provided by C++11 and Qt5 to release this version.

This is the first Tufão release that:

• … breaks config files, so you’ll need to update your config files to use them with the new version
• … breaks ABI, so you’ll need to recompile your projects to use the new version
• … breaks API, so you’ll need to change your source code to be able to recompile your previous code and use the new version
• … breaks previous documented behaviour, so you’ll need to change the way you use features that were available before. But don’t worry, because the list of these changes are really small and are well documented below.

Porting to Tufão 1.0

From now on, you should link against tufao1 instead of tufao. The PKGCONFIG, qmake and CMake files were renamed also, so you can have different Tufão libraries in the same system if their major version differs.

The list of behavioural changes are:

• Headers are being stored using a Hash-table, so you can’t easily predict (and shouldn’t) the order of the headers anymore. I hope this change will improve the performance.
• HttpServerRequest::ready signal auto-disconnects all slots connected to the HttpServerRequest::data and HttpServerRequest::end slots before being emitted.
• HttpFileServer can automatically detect the mime type from the served files, so if you had your own code logic to handle the mimes, you should throw it away and enjoy a lesser code base to maintain.

Most of the errors related to API changes are caught at compile-time and you can use the Qt5’s new connection syntax and C++11’s override keyword to catch the remaining errors.

News

The list of changes:

• The project finally have a logo (made by me in Inkscape)
• Deprecated API was removed
• Url and QueryString removed in favor of QUrl
• Headers refactored to inherit from QMultiHash instead of QMultiMap
• HttpServerResponse
  • Constructor’s options argument is optional now
  • setOptions method added
  • Constructor takes a reference to a QIODevice instead a pointer
• HttpServerRequest
  • Constructor takes a reference to a QAbstractSocket instead a pointer
  • socket method returns a reference instead a pointer
  • url returns a QUrl
  • data signal was changed and you must use readBody method to access body’s content
  • the upgrade’s head data is accessed from the request body from now on
  • now the object auto-disconnects slots from data and end signals right before emit ready
  • setCustomData and customData methods added
    • Now HttpServerRequestRouter use these methods to pass the list of captured texts
• HttpServer uses reference instead of pointers in several places
• AbstractHttpServerRequestRouter refactored to explore lambdas features
• Tufão’s plugin system fully refactored
  • It’s using JSON files as configuration
• AbstractHttpServerRequestHandler::handleRequest
  • It uses references instead pointers
  • It receives 2 arguments instead of 3
• One more abstraction to sessions created to explore lambdas
• WebSocket
  • startServerHandshake is taking references instead pointers
• LESS POINTERS and MORE REFERENCES
  • This change exposes a model more predictive and natural
  • I’m caring less about Qt style and more about C++ style
    • But don’t worry, I’ll maintain a balance
• Using scoped enums
• HttpFileServer uses/sends mime info
• Interfaces don’t inherit from QObject anymore, so you can use multiple inheritance to make the same class implement many interfaces
• HttpUpgradeRouter introduced
  • HttpServer::setUpgradeHandler also
• Updated QtCreator plugin to work with QtCreator 2.7.0 and Qt 5

The future

I want to improve the Tufão’s stability and performance, so now I’ll focus on a minor relase with unit testing and some minor changes.

After Tufão 1.0.1, I’ll focus on BlowThemAll.

Bonus

You can see a visualization video based on the history of commits below.

This release deserves a wallpaper, so I made one. See it below:

You can download the source of the previous wallpaper here.

What are you still doing here? Go download the new version right now!

Understanding Tufão, part 1

This post is dedicated to Tufão, but before talk about Tufão, I’ll tell a small preparatory text. Once while surfing on the web, I found Node.js. What is Node.js? An excelent asynchronous JavaScript web API. Why is it excelent? It allows you write little code to do a lot of web-related things. There are lots of web frameworks in this level, but they usually don’t provide the right level of abstraction. Why not? They don’t support even old modern HTTP (the web protocol) like chunked entities and 100-continue, what about real modern HTTP features like WebSocket, WebRTC, among other then? You just can’t use modern features. You’ll have problems even with basic examples like doing live stream from a webcam. After this day, the day I knew a decent web API, I started to care about web development. And then, I created the Tufão project.

Tufão is a tool to work with web in C++. Why C++? I like C++ and this is enough to me, but there are other reasons to create a web framework for C++. First, computing is changing again. In the last years, the clock speed of the processors stalled and again, you need to care about performance. Not only that, but the nature of techniques to gain performance are changing too. Today, you find processor with more and more cores, you can see more general purpose enabled GPUs and there are more optimized instruction sets in processors. C++ is the right tool for the job, and with the language update in the last year, I believe that the language is becoming stronger. Second, I don’t see a de facto standard to create web applications using C++. Third, there are lots of good libraries and APIs in C++ that you can use to implement cool ideas and develop web apps in C++ would let you to use them.

So, instead of porting a not decent API to create a C++ web framework, I choose Node.js to create the API core. I believe that C++ programmers love controlness, so they will like the API’s power. Besides the core, I’m implementing other facilites to help in the use of application servers, file upload, forms parsing, WebSocket, WebRTC and other modern standards, web services, …

How does Tufão compare to other frameworks? In this post, I’ll compare it through some different points of view.

The programming language

My idea about programming language is that languages should be designed to express ideas, and if they don’t have a good expressive power, then they are useless. Today we have lots and lots of programming languages. Personally, I don’t like the direction the Java language is taking, but, on the other side, I like Python. I already see some languages created only to develop web applications. I don’t believe that this is a good move. I don’t believe that the web development’s problem is the lack of an appropriate language. There are lots of good languages available today and I don’t believe that create new languages is necessary.

I’m developing Tufão in C++. C++ has a good expressive power and the last year’s update improved it. The language isn’t popular among web development, but web development is very dynamic and change rapidly from time to time. How I said before, the problem, in my opinion, is not in the lack of an appropriate language, but in the lack of an appropriate framework.

Modules

Tufão can have extra modules as any other framework, but compared to Node.js, I see a problem with ease of the installation process. In Node.js, we have npm and all we need to do is npm install <module>, but in Tufão you need rely in the operating system installation process or compile the modules. This isn’t a problem in some GNU/Linux distributions like ArchLinux, but this might not be the case in other systems.

I don’t see an ease way to solve this problem, so I’ll try to evolve Tufão to be self-sufficient in most of the cases. And this isn’t really difficult as Tufão is built on top of Qt, that already provides lots and lots of common needed features to any application.

Routing mechanism

In Tufão, the rounting is built on top of two classes. The AbstractHttpServerRequestHandler interface and the HttpServerRequestRouter class. The AbstractHttpServerRequestHandler interface requires you to create a slot to handle a request and the HttpServerRequestRouter is itself is an AbstractHttpServerRequestHandler. This design let you to build things like nested routing and use the same router in several different combinations (serving to HTTP and HTTPS servers).

In Tufão, the basic building block for routing request is the same as several other frameworks. You just need to use regular expression to filter the path and pass a handler to that request, and, optionally, specify which methods the object will handle.

The plugin-based application server

Well, when you develop a web application, you want to do it available 24h per day, every day. Imagine how you’d feel if you had to stop the server, recompile the application and run it again just to add your company’s name at the bottom of the page.

We could just use a dynamic interpreted language and solve the problem, but the language isn’t the problem. In some web frameworks there is the notion of an application server, that usually is integrated in the routing mechanism. The role of the application server is to dynamically register new applications and remove old ones.

In C++, a dynamic handler is just a plugin stored in a shared library loaded at runtime by the application. It’s just the old *.dll files on Windows or *.so on Linux. Tufão provides an application server too, but, different from most of the frameworks I’ve seen, it’s not melted in the routing mechanism.

The Tufão application server is plugin-based, stores its configuration in a file (chosen by you) and provides an external program to update this file. It’s easy to use, but you’ll have to add code to define when the application server must reload the config file. If you don’t have any special requeriments, the default application template code is enough and you won’t need to write this piece of code at all.

MVC separation

Some frameworks just don’t separate logic and ui design, such as most of old PHP, embedding the logic code and the html (or some custom view language code) code in the same file.

There are frameworks that allow you separate the logic and the ui design, but do a wrong implementation of the idea, requiring the designer to have some knowledge about the framework internals, trapping you to use the tools from the same framework forever.

What is the idea of other frameworks? Other frameworks let you export application data to application ui design (through web services or other), allowing you to use any language to generate your HTML/CSS and import the data through JavaScript. This idea by alone don’t only help to lower the security problems (designers thinking about programming) and the application ugliness (programmers thinking about ui design), but also help to use HTTP features to decrease the network traffic needed by you application, making the use of cache more efficient (in HTTP level). How this happens? This is the old Ajax technique.

1. Write ui design code in pure HTML, CSS and JavaScript and put the files in the public folder.
2. Write logic to generate and serve the data in C++, without caring about ui design, and export the data in some format simple to import from JavaScript, like JSON.
3. Write JavaScript code to import the data from the services and populate the views.

Using this technique, when the user access your site, the browser will request the pages server through the Tufão’s static file server, making most part of the page cacheable, saving precious network traffic from your application . Also, the view isn’t generated at runtime, so you will save server’s processing power too, increasing the maximum numbers of requests per second.

Another advantage is the freedom to choose technologies. The UI is just a bunch of static files, isn’t something chained to specific Tufão internals, so you can use any tool you want to create them. We don’t take away your freedom.

As an advice, you should implement each page handler as a plugin, so you can change the running code at runtime.

IDE

Web frameworks usually don’t develop custom IDEs and this isn’t an exception for Tufão. But, Tufão is built on top of Qt, and Qt programmers are used to QtCreator, so I created a QtCreator plugin to help the development of Tufão applications. The plugin just help to create new applications, providing application templates. I think this is enough, but if I change my mind, I’ll extend the plugin.

There are 3 application templates currently:

• The first is a full featured web application, using the Tufão’s static file server, plugin server and a custom handler to treat 404-pages.
• The second is a plugin that responds with a “Hello World” and is as small as possible to keep things simple.
• The last is just a Hello World application.

Development time

Well, It’s not an easy task to measure the development productivity, but I can talk about some points. If what you want is a creative web app, you problably need a powerfull low-limited web API to help you, and my aim is to provide such API, so I highly invite you to test Tufão and give me some feedback.

If you plan to create some common-case application, there are frameworks such as Django that already provide several integrated facilities (such as mvc with integrated mapping of database schemas, template systems, already implemented authentacion sytems, … …) and right now, Tufão can only compete if you use other libraries to help you.

Improve Tufão to compete with Django isn’t my priority and I don’t even know if I care about facilitate the creation process of boring common projects, but is an important area and if the demand or the help increase, I’ll create tools to help in this use case too.

Framework’s future

Tufão is an open source project under the LGPL license. The documentation, examples and public headers are under the MIT license. This means that anyone can contribute to Tufão development and you can use it in your commercial projects.

I created Tufão all alone and tried to do I really good job in every piece of it, but maybe the documentation isn’t good enough, so if you have any question that cannot be solved reading the documentation, just ask me and in the next version the documentation might be better. The same rule applies to bugs (and I wrote lots of tests to prevent it, but I just didn’t realize how to automate all tests, like little endian and big endian tests…).

That’s it

In the next part of this article, I’ll demonstrate some benchmarks and how to create a web application using as much Tufão’s features as possible. Thank you for reading about the project, I really appreciate, and if you like the project, spread it among your friends.

I ackowledge Ryan Lienhart Dahl for developing Node.js. Before its API, I thought web development was tedious.

Tufão (the Qt/C++ web framework) 0.2 released

Hello good fellows, I’m not used to write posts in english, but this is a habit that I must change. Well, just today I’m releasing a new version of my C++/Qt web framework, Tufão.

For the newcomers, Tufão was, initially, a port of the excellent Node.js API to C++ built on top of Qt. It was a good choice because the Node.js assynchronous API fits well in Qt’s object communication system (signals & slots).

The new 0.2 version brings the following changes:

• Code is more stable
• Documentation a bit improved
• HttpServerResponse class is easier to use (stream operator)
• WebSocket support
• Added TUFAO_VERSION_MAJOR and TUFAO_VERSION_MINOR macros
• A QtCreator plugin
• New examples

The most problematic areas of Tufão now is its lack of:

• An application server
• A file server

In future versions these gaps will be filled. For more details take a look at the project roadmap in the README file.

I welcome everyone to compile, test and use Tufão.

Tufão 0.1

Como vários de vocês não sabem, eu não passo de um preguiçoso movido pelo tédio. Ultimamente, venho aperfeiçoando o projeto do qual falei no meu último post (o Tufão), que visa eliminar o vazio do desenvolvimento web que há para C++ utilizando a framework Qt como base. Após muito trabalho e dedicação o Tufão passou na bateria de testes e está implementando toda a API proposta, que se iguala a API oferecida pelo Node.js. Assim sendo, estou anunciando o lançamento de sua primeira versão estável, a versão 0.1.

Vocês podem baixar essa versão na página do Tufão no googlecode e eu vos convido a testarem um dos exemplos que acompanham o Tufão.

	// Programadores de Qt devem estar muito familiarizados com esse código
	#include "mainhandler.h"
	#include <Tufao/HttpServerRequest>
	MainHandler::MainHandler(QObject *parent) :
	QObject(parent)
	{
	}
	void MainHandler::handleRequest(Tufao::HttpServerRequest *request,
	Tufao::HttpServerResponse *response)
	{
	response->writeHead(Tufao::HttpServerResponse::OK);
	response->end("Hello " + request->url());
	}

view raw tufao.cpp hosted with ❤ by GitHub

O código anterior poderá ser melhorado no Qt5, que fará um melhor uso de funcionalidades encontradas no novo padrão do C++, como o uso de lambdas. Para aqueles que ficaram curiosos, recomendo esse link.

De agora em diante, estarei trabalhando nas seguintes funcionalidades:

• Plugins para o QtCreator
• Servidor de aplicações baseado em plugins
• Suporte a WebSocket
• Suporte mais fácil a multithread (atualmente já é possível utilizar threads de forma eficiente no Tufão para tirar proveito de várias CPUs)
• Descoberta e descrição de serviços
• Implementação de alguns RPCs comuns
• Backend HTML5 para o Qt

Introduzindo o Tufão

INTRODUÇÃO (a parte chata)

Atualmente a internet possui uma importância imensurável para o ato de compartilhamento de informações e nesse cenário o protocolo http ainda se destaca por ser a base responsável por garantir a interoperabilidade de computadores que foram projetados em 1997 e computadores que foram vendidos ontem, garantindo assim que tais informações trocadas entre esses nós possam ser preservadas e passadas adiante.

Para os desenvolvedores, uma das partes mais legais é que o protocolo foi projetado de forma que qualquer pessoa pode criar uma implementação compatível com o mesmo. Algumas características do HTTP são:

• É baseado, em sua maior parte, em texto
• É um protocolo bem definido
• Está publicamente disponível
• Não possui mais de 6000 páginas (esse ponto na verdade é só uma crítica a uma especificação não relacionada ao HTTP que existe por aí)

Caso você opte por respeitar o padrão, o que quer que você desenvolva estará seguindo um padrão suportado por todos os servidores webs e http user agents (isso inclui o firefox, o chromium, o internet explorer, …) existentes. Barreiras dificultadoras que só beneficiam algumas entidades não são impostas a você, como ocorre no desenvolvimento de aplicativos para iPhone e outras app-stores.

O projeto Tufão (a parte legal)

Durante o desenvolvimento de um dos projetos nos quais estive envolvido eu tive a oportunidade de trabalhar com node.js, uma API para javascript que fornece uma abstração do HTTP. Ela é flexível o suficiente para a criação de aplicações HTTP não-convencionais para as quais muitas frameworks não estão preparadas, como, por exemplo, streaming de conteúdo gerado dinamicamente no servidor ou mesmo o uso de WebSockets.

Havia em mim um sentimento de insatisfação devido a falta de uma framework web para C++ que fosse, ao mesmo tempo, produtiva e boa o suficiente. Então, pouco tempo após conhecer o projeto node.js resolvi portar sua API para C++. Assim nasceu o projeto Tufão.

Para me poupar tempo e não reinventar a roda, resolvi optar pelo uso do Qt, pois o mesmo disponibiliza um loop de eventos assíncrono e implementações multiplataformas de conceitos necessários, tais como sockets. Acho que sua popularidade entre muitos desenvolvedores de C++ deve ajudar na aceitação do Tufão.

Diferente dos meus projetos anteriores, desenvolvi o design do Tufão antes de sua implementação, para garantir tanta qualidade e facilidade de uso quanto possível. Assim, no começo havia somente um monte de arquivos de cabeçalho e comentários que o doxygen podia usar para gerar documentação, mas após várias e várias revisões a API chegou num ponto estável e bom o suficiente para uma primeira versão.

Então, após muitas e muitas horas de esforço, apesar de não implementar ainda toda a API proposta, o Tufão chegou em sua primeira versão funcional e eu fiz um pequeno vídeo demonstrando um exemplo com o firefox:

Pretendo, no futuro:

• Terminar a implementação seguindo a proposta de API criada
• Automatizar/facilitar o processo de instalação
• Avaliar e implementar mais protocolos utilizados na web, como o WebSocket e o SOCKET.IO
• Criar um backend HTML5 do Qt
• …

O Tufão está licenciado sob a LGPL, permitindo a criação de aplicações comerciais e pode ser encontrado em http://tufao.googlecode.com/ . Acho que com esse texto deixei claro o objetivo do Tufão e assim podemos dizer que o projeto está oficialmente anunciado.

🙂