Wednesday, October 12, 2011

HipHop for PHP at Hyves

HipHop is Facebook’s open source C++ compiler for PHP. HipHop (also known as HPHP) will compile your PHP code to highly optimized C++ code, which you can then compile (with g++) into a big binary that will run your web site (it includes a web server). If you want to know more about Hiphop and how it was created, you can check out the Facebook Engineering Blog article or the Github project.

After its introduction many bloggers have written background articles regarding Hiphop but few seem to have actually used it in a production environment. At Hyves, we use HipHop to run our web servers and our daemons, which are also written in in PHP. In this blog, I will detail some of our experiences and results.

Compiler and interpreter

Hiphop includes both and interpreter and a compiler. This is something that is not frequently mentioned in blog posts on Hiphop. The interpreter (called HPHPi) can be used for developing, thus saving you the hassle and the delays associated with compiling every time you change a single file. In general is has the same behavior as the compiled version of Hiphop. It is a bit slower than using PHP with an opcode cache, but it is not that bad and with a fast laptop it is still workable.

The compiler can be used to create an optimized binary for your code base. The Hyves code base is in the order of 3.5M LOC in PHP. The conversion from PHP to C++ has to be done by a single server, but we use a dedicated cluster of sixteen servers for compiling the C++ source code to an executable binary. This takes about seven minutes including the building of the web site (e.g. minimizing css and Javascript files).

The resulting binary is approximately 500MB size. This cannot be deployed to our web servers in a serial fashion, it would take too long. To deploy this to our web servers, we use a bittorent based p2p deploy system. In case of an emergency, we can roll out a fix in approximately ten minutes to our 300 web servers.

Behavioral differences

The HPHP interpreter, the compiled version and the official PHP binary all have slightly different behavior. HipHop is about 99% compatible with pure PHP. 99% times 3.5M LOC is a lot of problems. If you want to run HipHop, make sure that your unit tests are ran against all three versions automatically or you will drive yourself and your team members insane, especially in a mixed PHP / HPHP environment.

Some PHP 5.3 stuff is still not completely supported in Hiphop, such as namespaces, of which at the moment nearly every feature is broken. Extensions such as PHP’s SOAP extension have been ported but some behavior differences exist between the variants.

Converting the Hyves code base to run on HipHop took a big effort, several of our software engineers worked on this projects for months.

Hiphop has some really cool features. There is a call_user_func_async() function (it does exactly what it says), and you can have two versions of you web site running at the same time, so that you can deploy a new version without downtime. Also, you can catch fatal (E_FATAL) errors with their stack traces with HPHP, a feature sorely lacking from PHP.

There is only sparse online documentation of these specific features so you will have to figure a lot of it our by looking at the source.

Performance

Performance of our Hiphop enabled web site has been very good. In particular our API has strongly benefited, with API calls being twice times as fast and requiring only a quarter of the cpu power compared to plain PHP. This has resulted in much faster mobile clients and better performance of the parts of our web site that use internal API calls.

In the top graph you can see our web servers 95% and 99% response times. We enabled HipHop for all our web servers in March 2011. The higher response times just before the switch are due to internal network congestions because of some issues with the p2p deploy system we use.
The bottom graph shows the CPU load of our web servers, which has dropped by a factor of four after the switch to HipHop. This will enabled us do deprecate a big portion of our web servers in the future.

Open source

HipHop is an open source project, available on Github. If you file a bug, Facebook’s maintainers can be pretty responsive, well reported bugs can be fixed in a matter of days. Of course you can always fix it yourself on Github and try to get the fix pulled into the project.

Al though Facebook has reported that Wordpress and Mediawiki are experimenting with HipHop, not many people actually seem to be using Hiphop in production. For now, the community is pretty small outside of Facebook.

Also Hyves contributed to the HipHop project, we contributed a number of bugfixes and the ability to use distcc so you can compile your web site on multiple machines.

Conclusion

All in all the switch to HipHop has been a big success for Hyves. Al though switching comes with a steep cost in terms of implementation and tooling, for a big web site such as Hyves it is definitely worth the effort.