log4erl 是一个 log4* 的 erlang 实现。对于广大习惯了 log4* 的 erlang 使用者来说，确是一个福音。不过，从某种程度上讲，这也可以算是某种“重复建设”—— erlang 自身已经内置了相当强大的日志机制。注意，只是强大，我并没有说易用。因为对于广大 log4* 爱好者们而言，只有能使用 grep/awk/perl 这些标准 unix tool chain 来处理的，格式化的，每条一行的(问题是，以我的经验来说，”每条一行”这种限制常常会逼人多费半天劲，或者多写一大堆)才是易用的日志。而 erlang 自身内置的日志机制说实话也毫不逊色，只是用来颇有一些不同(加之文档组织得别出心裁，让人不易找到)，各自属于不同的风格。既被拔到了“风格”层面，也就参杂了情感因素，无法以简单的方式来区分高下(君不见 csdn 上 x vs y 的口水仗层出不穷无休无止?)。本是见仁见智的事，大家看个人的喜好各取所需就是了。

顺便说说 erlang 自身的日志机制，极简的例子看上去象这样：

这好比是 log 的配置文件，然后，这样用：

这样，在 debug.log 里面就有了：

差不多就是这样了，更多配置，相关工具，以及更 “The Erlang Way” 的用法，可以参看文档 SASL 的部分。

btw，升级 wp 时从 incoming link 里发现一个 “默默无闻的领悟Erlang传说” 的中文 blog ToQuick (图快?)，主人 cheng 同学治“程”(程序)态度严谨，细密，文章写得勤也写的好，值得推荐。下面对于 log4erl 的代码观感就转自图快：

log4erl代码非常精简，保持了erlang代码的风格，log4erl包括1个supervisor和2个gen_server和1个gen_event。
log4erl.erl为gen_server和application behaviour的callback，其作为主要的module，
log4erl_sup.erl为supervisor,启动log4erl,初始时将通过log4erl_sup:add_file_logger启动default_logger 日志记录器。
log4erl_sup:add_file_logger/1会动态添加file_logger_guard和file_logger两个child process。

have fun.

说起来，这个新出的 Kai 乃与是我们“一衣带水”的日本同行们的作品。加之此前听说的 Orto，以及零星听说的具有超强并发能力的 tokyocabinet，等。老实说，对于他们在 opensource 领域所表现出来的整体活力，确实有些惊讶。可惜不懂日文，未能对此作进一步的了解。(召唤日语达人整理一下)

话说这几个项目共同关注的是一个相同的问题，然而各自的侧重又略有不同。比如说，Kai 项目实现的乃是(又是) memcache 的访问接口和协议，主要侧重在利用 Erlang 来提升系统在“分布和容错”方面的特性。而 couchDB 则首先关注于实现 key-value 的数据存储本身，在此基础上建立完全不同于 RDBMS 的查询机制，并以优雅的 RESTful 访问接口而著称(Alexander 的项目资料不多，尚不明朗)。应该说是各有特色，而且具有很好的互补性，实在让人很难在这两者之间进行取舍。所以当 couchDB 团队的 Jan Lehnardt 向 Kai 团队的 Takeru Inoue 提出 Collaborate 的建议时，对于广大使用者而言，确实是最为“喜闻乐见”的情形。

或许我们很快就能见到 CouchDB Powered Kai 又或者是 Kai Powered CouchDB，两个项目的特性得到融合，也许，两者之间会形成一种如同 innoDB 之于 mysql 那样的合作关系，谁知道呢？未来充满可能，值得我们期待。开源的 Erlang 就是这么精彩。

由 Jamaal 同学补充：
在 video.google.com 上搜索 Erlang eXchange 能找到一堆演示视频，虽然没有 PPT/PDF 会略有遗憾，但胜在真人出演，也相当不错。懒得搜索的同学可以点[这里]。

Armstrong on Software: Erlang & SMP,
Joe Armstrong

Introducing Erlang to Motorola: The Journey to Success,
Nicholas Gunder & Torben Hoffman

Erlang- D-Trace,
Garry Bulmer

Erlang & Robotics: The ROSEN Framework at the Eurobot 2008 Competion,
Corrado Santoro

Erlang/OTP Vs Google App Engine, The CEO View,
Gordon Guthrie

Building Web Applications in Erlang,
Xingdong Bian & Michal Slaski

Erlang in Financial Applications,
Dr. Erik Stenman

Erlang and Ajax Web Applications,
Roberto Saccon

Quick Check for Erlang,
John Hughes

Introduction using Faxien & Sinan: Erlang Project Build & Packaging Systems,
Eric Merrit & Martin Logan

Roktalk, Erlang Powered Mobile Conferencing,
Jay Fenton

Wrangerl, The Erlang Refactoring Tool,
Simon Thompson

Enterprise Integration,
Steve Vinoski

Erlang & Tail-F,
Klacke Wikström

Presenting RabbitMQ: An Erlang Based Implementation of AMQP,
Matthias Radestock

EUnit – Lightweight Unit Testing for Erlang,
Richard Carlsson,

ejabberd for web 2.0 development,
Mickael Remond

Load Testing of Web Applications,
Karthik Ramachandra,

Using Jinterface to Bridge Erlang and Java,
Dennis Byrne

Quick Check Tutorial: Using QuickCheck to Test Erlang Programs,
John Hughes & Thomas Arts,

Using Faxien and Sinan, A Hands-on Approach,
Martin Logan & Eric Merrit

Couch DB at 10,000 feet,
Jan Lehnardt

Building a transactional distributed data store with Erlang,
Alexander Reinefeld

Tsung Tutorial,
Mickael Remond,

Erlang Enterprise Integration Panel Discussion,
Garry Bulmer

Joel Reymont 就是那位在 05 年写出惊到大家的《Writing Low-Pain Massively Scalable Multiplayer Servers》一文的作者。(此文本站亦有中文翻译《轻松实现可伸缩性，容错性，和负载平衡的大规模多人在线系统》，感谢译者“神宗冥浩”)。他这次带给大家的是一个让人惊叹的大礼包——超乎想象的 mnesia 补丁包 mnesiaex 。这个东西解除了加在 mnesia 数据库系统上所有的限制(虽说上面已经提到，实际上 mnesia 代码本身没有什么真正的限制)——你现在可以用 SleepyCat/BerkeleyDB/MySQL/Amazon S3/Tokyo Cabinet/… 甚至是你自己喜欢的某种东西来当作 mnesia 的后端，就像 ets/dets 一样。而访问的接口仍保持不变——继续沿用 mnesia 的接口，一行也不用改。 DIY 这种扩展也变得相当容易，写一个 behavior 就成了。

感谢 Joel Reymont 将这些工作回馈到开源社区。让我们一起祈祷 OTP Team 将这堆 patch 合并到 Erlang 的下一个发布版本中去吧。

顺便 blah 一下：

关于 Erlang

Erlang 就好像是 Ericsson 的私生子，从出生之日起就一直不得宠。在 AXD301 中的耀眼光芒，还是逃脱不了被弃用的命运(Ericsson 又转回去用 C 写交换机了，别让我猜中是因为公司政治)。失败了的 Joe 一伙人被迫离开自组 BlueTail 公司，绝望之中以 Open Source 协议公布了 Erlang 的代码，这个挫折使得它在编程语言的坟场寂寞的躺了多年，但仍然保留着翻盘的火种。默默无闻的完善了多年(加入SMP支持之类)，一直不为人所知。直到碰上 CPU 多核变革的机遇，这才重新捡回半条命，并渐渐被人提起。但别忘了，Erlang 直到现在仍然都是由 Ericsson 所拥有(整个的 OTP Team 都是他的员工)和操纵的(你能看到 Erlang 的 souce code 但能访问 Erlang source code 的 SVN 么？)。而比 Sun 的 Java 更加糟糕的是老态龙钟的 Ericsson 从来也没有意识到 Erlang 这个私生子身上所蕴含的潜力。麻烦哪位消息人士请一定转告 Ericsson 的老爷爷们，现在连 Sun 都已经完全开源了 Java ，请抓紧赶上吧，把那些没用的遮遮掩掩全都扔掉。因为对于一个程序设计语言而言，只有 Open Source Community 的程序员们，只有这些人，才是它生命力的真正源泉。在此祈祷 Open Source Erlang 项目朝着更 Open Source Way 的方向前进。

关于 Mnesia

因为工作关系，最近又有机会再来近距离审视 mnesia 这坨神奇的东西。Joe 老头在他的书中说：“关于 Mnesia 的更多内容，恐怕还要再写一本书才能讲得清楚”，现在我(部分地)知道这句话的分量了——我发现自己之前对于 Mnesia 的认识完全错了，而基于新的认识，好多东西都要推翻了重来(害我多做了那么多蹩脚的实现，写了那么多苍白的代码)。我的感觉(现在的)是—— mnesia 根本就不是什么数据库，这只是一个善意的谎言(以它出现的时代来说，太激进，会把人都吓跑了)。实际上，它根本就是一个 Erlang 的 hibernate 。换句话说，这个东西就不应该被拿来当作“数据库”用，而是应该拿来当作“数据层”用。一字之差，谬以千里，熟悉 Java/SSH 编程的同学们相信都能明白我在说什么。实际上，我私下里在怀疑这是 mnesia 最初的设计目的之一，但为了某种原因而故意不去点破这一层。但愿在这个问题上我只是个可耻的阴谋论者。

这篇文章是 Robert Zubek 对 Joe Armstrong 在 07 年某个会上 show 的 “A History of Erlang” PDF 的评论(更象是写了一个“注”)。基本上，相比著名的 “Joe Thesis 2003” (本站提供了中文翻译，感谢段先德同学)而言，这篇 PDF 中并没有太多的新内容，只能算是一个精编浓缩 remix 版而已。

但对 Joe Thesis 2003 这篇略嫌艰深的博士论文来说， Robert Zubek 的文章提供了很好的“补充视角”。也就是说，从 Joe Thesis 中我们能看到 Erlang 在“一系列问题”上做出了与众不同的选择。但如果没有一定的背景知识，我们或许没怎么觉得这“一系列问题”又有什么大不了的。而这一篇正好(部分地)补充了这些背景知识。

照例，对于这篇文章的干货，还是打捞一下。

But this paper really conveys the feel for what that means: huge collections of loosely coupled finite-state machines, all running in parallel, doing little bits of protocol validation here and there, but mostly staying dormant.

英文的描述，如果写得好了，会有一种准确精妙难以言表的感觉，通常很难原汁原味的译为相对应的中文。不过，这句太经典，我还是想翻来看看。

这篇文档所传达的要义(用 Erlang 构造系统的感觉)是：一大堆并行运行着的有限状态机，各处做一做协议验证，彼此松散耦合，通常大部分的有限状态机都处在休眠状态。

作者提到他对这篇 “A History of Erlang” 中“有感觉”的几点是：

First, there’s strong emphasis on easily tracking and manipulating the flow of control, rather than data.

Second, I enjoyed the explanation of one particular detail of Erlang: in the message-passing implementation, individual processes don’t handle messages immediately.

Third, Erlang is somewhat infamous for its distributed error handling mechanism.

这只是起头，精妙在于剖析。任何一个设计决策都是有 trade off 的，对这些设计决策正反两面的分析，或许能够部分解释“为什么 Erlang 会这么怪异”，相当值得一读。

顺便说一句，我“有感觉”的一点是 “Erlang 天生就是以消息驱动模型来架构的系统” 而 “消息驱动的编程模型” 如何应用于互联网编程环境，正是值得我们深思的问题。

下面是我们的前方记者从 “不明地带” 发回的 —— 现场 “流水帐” 。

第一回

启动：

连接：

得到：

看来和一个 crypto 啥的有关系。关门，放 google ……。原来 ssh 的加密体系依赖于 crypto app 需要先启动它。

第二回

启动：

连接：

得到：

这个 eacces 貌似很可疑，关门，再放 google ……。原来 sshd 默认会访问 /etc/ssh 以获取 “host files that identifies the host for ssh” ，而 /etc/ssh 只有 root 能访问，当然是会失败的。必须加上一个 system_dir 参数指定一个新位置。

第三回

启动：

连接：

得到：

enoent，想起来 “host files that identifies the host for ssh” 在 . 目录下根本没有，需要生成之。关门，又放 google ……。原来要如此这般的生成 ssh host 文件：

得到了 4 个 ssh_host_xxx 文件，分别对应 rsa 和 dsa 加密算法。

第四回

启动：

连接：

得到：

BINGO! 搞定！

延伸问题

上述方式启动的 ssh_sshd 仍然存在问题——当 q(). 退出 erlang shell in ssh 时，会让 ssh_sshd 退出，但同时也会报错。why？how to fix？

上述方式启动的 sshd 登录时，如何采用 public key 或者用户名/密码验证？

有兴趣了解的同学，自己做一下实验吧。

BTW. ssh application 目前正在 refactor 之中，有 OTP team 的 mail 为证：

Hi!

Thank you for reporting this and other issues in previous mails. We are in the process of refactoring the ssh-application and there is much room for improvement of the documentation, we will look over this.

Regards Ingela - Erlang/OTP, Ericsson

主要的 refactor 方向是（基于情报进行合理的大胆假设）：“更优雅的整和 ssh_sshd 和 ssh_sftpd 系统” 以及更 “customize 友好的架构” ，比如：ssh/sftp 连上去的不再是 erlang shell/real file system 而是一个 “定制的环境” 。ssh as a protocol (like http)? sounds great!

为方便阅读，将 mail 内容盗版如下：

Well, if you had time you could dig through my various IEEE Internet Computing columns from the past 6 years and find many reasons listed there. For example, “RPC Under Fire“(note that it’s PDF) from the Sep/Oct 2005 lists a number of problems:

Also, pretty much any of my columns that cover REST to any degree would also contain mentions of RPC’s shortcomings. All the columns can be found here:

But if you don’t have the time or energy, the fundamental problem is that RPC tries to make a distributed invocation look like a local one. This can’t work because the failure modes in distributed systems are
quite different from those in local systems, so you find yourself having to introduce more and more infrastructure that tries to hide all the hard details and problems that lurk beneath. That’s how we got
Apollo NCS and Sun RPC and DCE and CORBA and DSOM and DCOM and EJB and SOAP and JAX-RPC, to name a few off the top of my head, each better than what came before in some ways but worse in other ways, especially footprint and complexity. But it’s all for naught because no amount of infrastructure can ever hide those problems of distribution. Network partitions are real, timeouts are real, remote host and service
crashes are real, the need for piecemeal system upgrade and handling version differences between systems is real, etc. The distributed systems programmer *must* deal with these and other issues because
they affect different applications very differently; no amount of hiding or abstraction can make these problems disappear. As I said about such systems in a recent column:

“The layers of complexity required to maintain the resulting leaky illusion of local/remote transparency are reminiscent of the convoluted equations that pre-Copernican astronomers used to explain how the Sun and other planets revolved around the Earth.” (from “Serendipitous Reuse“)

RPC systems in C++, Java, etc. also tend to introduce higher degrees of coupling than one would like in a distributed system. Typically you have some sort of IDL that’s used to generate stubs/proxies/skeletons
– code that turns the local calls into remote ones, which nobody wants to write or maintain by hand. The IDL is often simple, but the generated code is usually not. That code is normally compiled into each app in the system. Change the IDL and you have to regenerate the code, recompile it, and then retest and redeploy your apps, and you typically have to do that atomically, either all apps or none, because versioning is not accounted for. In an already-deployed production system, it can be pretty hard to do atomic upgrades across the entire system. Overall, this approach makes for brittle, tightly-coupled systems.

Such systems also have problems with impedance mismatch between the IDL and whatever languages you’re translating it to. If the IDL is minimal so that it can be used with a wide variety of programming
languages, it means advanced features of well-stocked languages like Java and C++ can’t be used. OTOH if you make the IDL more powerful so that it’s closer to such languages, then translating it to C or other
more basic languages becomes quite difficult. On top of all that, no matter how you design the IDL type system, all the types won’t — indeed, can’t — map cleanly into every desired programming language. This turns into the need for non-idiomatic programming in one or more of the supported languages, and developers using those languages tend to complain about that. If you turn the whole process around by using a programming language like Java for your RPC IDL in an attempt to avoid the mismatch problems, you find it works only for that language, and that translating that language into other languages is quite difficult.

There’s also the need with these systems to have the same or similar infrastructure on both ends of the wire. Earlier posters to this thread complained about this, for example, when they mentioned having to have CORBA ORBs underneath all their participating applications. If you can’t get the exact same infrastructure under all endpoints, then you need to use interoperable infrastructure, which obviously relies on interoperability standards. These, unfortunately, are often problematic as well. CORBA interoperability, for example, eventually became pretty good, but it took about a decade. CORBA started out with
no interoperability protocol at all (in fact, it originally specified no network protocol at all), and then we suffered with interop problems for a few years once IIOP came along and both the protocol itself and implementations of it matured.

Ultimately, RPC is a leaky abstraction. It can’t hide what it tries to hide, and because of that, it can easily make the overall problem more difficult to deal with by adding a lot of accidental complexity.

In my previous message I specifically mentioned Erlang as having gotten it right. I believe that to be true not only because the handling of distribution is effectively built in and dealt with directly, but also because Erlang makes no attempt to hide those hard problems from the developer. Rather, it makes them known to the
developer by providing facilities for dealing with timeouts, failures, versioning, etc. I think what Erlang gives us goes a very long way and is well beyond anything I’ve experienced before. Erlang really doesn’t provide RPC according to the strict definition of the term, BTW, because remote calls don’t actually look like local ones.

(BTW, this is the kind of stuff I’ll be talking about at Erlang eXchange next month.)

1 作者放出 firebug 发现，http header 的 server 字段赫然写着 MochiWeb/1.0 的字样。

2 作者开动脑筋，在 mochiweb 基础之上，写了一个“最简单”的 facebook webim 概念模型，mochi loop 的核心代码如下：

简而言之就是：在 browser 和 mochiweb 之间保持 10 秒的长连接，这(10s的)期间收到的任何消息都会即时发送给 browser ，然后再由 browser 内的代码再次发起下一个连接。作者提供了完整例子代码的下载。PS.关于这一实现方式的更多说明也可以参考拙作“the google way”。

当然这只是一个“概念模型”，距离“实用价值”仍有一段距离。比如：这个 web 层如何与 ejabberd 接起来，如何识别同一个用户，如何增加更多的“聊天逻辑”。

有兴趣(有时间)深入研究的读者可以移步阅读原文。如果看不到原文，请留言，以便“进行盗版”。

1. Facebook 总算推出了 developer’s blog，在大家都推出自己的“开发者博”之后。
2. 这个“facebook 开发者博”透出来的第一条消息是——纯正 PHP/LAMP 血统的 Facebook 在高度 realtime 的 Chat 应用中，终于用了 Erlang 来提高性能( ejarbberd backend? 估计是 process-one 的那帮法国人最近一直都在搞的定制项目 )，我知道他迟早都会要用的。

如果觉得“干货”不过瘾，时间充裕的朋友可以移步阅读更“ juicy ”的原文—— 《Real-World Erlang Applications - Scaling Facebook Chat to 70 Million Active Users Almost Overnight》。

PS. 这篇 blog 亦可看作是《challenges of scaling up a realtime application》，所以，如果有人花时间将其 i8n 一下，那将会非常棒。

update 20080516：有人说打不开，那就“被迫盗版”一下吧。[节约版面起见，请点“继续阅读”]

Facebook Chat

by Eugene Letuchy Tuesday, May 13, 2008 at 10:56pm

One of the things I like most about working at Facebook is the ability to launch products that are (almost) immediately used by millions of people. Unlike a three-guys-in-a-garage startup, we don’t have the luxury of scaling out infrastructure to keep pace with user growth; when your feature’s userbase will go from 0 to 70 million practically overnight, scalability has to be baked in from the start. The project I’m currently working on, Facebook Chat, offered a nice set of software engineering challenges:

Real-time presence notification:

The most resource-intensive operation performed in a chat system is not sending messages. It is rather keeping each online user aware of the online-idle-offline states of their friends, so that conversations can begin.

The naive implementation of sending a notification to all friends whenever a user comes online or goes offline has a worst case cost of O(average friendlist size * peak users * churn rate) messages/second, where churn rate is the frequency with which users come online and go offline, in events/second. This is wildly inefficient to the point of being untenable, given that the average number of friends per user is measured in the hundreds, and the number of concurrent users during peak site usage is on the order of several millions.

Surfacing connected users’ idleness greatly enhances the chat user experience but further compounds the problem of keeping presence information up-to-date. Each Facebook Chat user now needs to be notified whenever one of his/her friends
(a) takes an action such as sending a chat message or loads a Facebook page (if tracking idleness via a last-active timestamp) or
(b) transitions between idleness states (if representing idleness as a state machine with states like “idle-for-1-minute”, “idle-for-2-minutes”, “idle-for-5-minutes”, “idle-for-10-minutes”, etc.).
Note that approach (a) changes the sending a chat message / loading a Facebook page from a one-to-one communication into a multicast to all online friends, while approach (b) ensures that users who are neither chatting nor browsing Facebook are nonetheless generating server load.

Real-time messaging:

Another challenge is ensuring the timely delivery of the messages themselves. The method we chose to get text from one user to another involves loading an iframe on each Facebook page, and having that iframe’s Javascript make an HTTP GET request over a persistent connection that doesn’t return until the server has data for the client. The request gets reestablished if it’s interrupted or times out. This isn’t by any means a new technique: it’s a variation of Comet, specifically XHR long polling, and/or BOSH.

Having a large-number of long-running concurrent requests makes the Apache part of the standard LAMP stack a dubious implementation choice. Even without accounting for the sizeable overhead of spawning an OS process that, on average, twiddles its thumbs for a minute before reporting that no one has sent the user a message, the waiting time could be spent servicing 60-some requests for regular Facebook pages. The result of running out of Apache processes over the entire Facebook web tier is not pretty, nor is the dynamic configuration of the Apache process limits enjoyable.

Distribution, Isolation, and Failover:

Fault tolerance is a desirable characteristic of any big system: if an error happens, the system should try its best to recover without human intervention before giving up and informing the user. The results of inevitable programming bugs, hardware failures, et al., should be hidden from the user as much as possible and isolated from the rest of the system.

The way this is typically accomplished in a web application is by separating the model and the view: data is persisted in a database (perhaps with a separate in-memory cache), with each short-lived request retrieving only the parts relevant to that request. Because the data is persisted, a failed read request can be re-attempted. Cache misses and database failure can be detected by the non-database layers and either reported to the user or worked around using replication.

While this architecture works pretty well in general, it isn’t as successful in a chat application due to the high volume of long-lived requests, the non-relational nature of the data involved, and the statefulness of each request.

For Facebook Chat, we rolled our own subsystem for logging chat messages (in C++) as well as an epoll-driven web server (in Erlang) that holds online users’ conversations in-memory and serves the long-polled HTTP requests. Both subsystems are clustered and partitioned for reliability and efficient failover. Why Erlang? In short, because the problem domain fits Erlang like a glove. Erlang is a functional concurrency-oriented language with extremely low-weight user-space “processes”, share-nothing message-passing semantics, built-in distribution, and a “crash and recover” philosophy proven by two decades of deployment on large soft-realtime production systems.

Glueing with Thrift:

Despite those advantages, using Erlang for a component of Facebook Chat had a downside: that component needed to communicate with the other parts of the system. Glueing together PHP, Javascript, Erlang, and C++ is not a trivial matter. Fortunately, we have Thrift. Thrift translates a service description into the RPC glue code necessary for making cross-language calls (marshalling arguments and responses over the wire) and has templates for servers and clients. Since going open source a year ago (we had the gall to release it on April Fool’s Day, 2007), the Thrift project has steadily grown and improved (with multiple iterations on the Erlang binding). Having Thrift available freed us to split up the problem of building a chat system and use the best available tool to approach each sub-problem.

Ramping up:

The secret for going from zero to seventy million users overnight is to avoid doing it all in one fell swoop. We chose to simulate the impact of many real users hitting many machines by means of a “dark launch” period in which Facebook pages would make connections to the chat servers, query for presence information and simulate message sends without a single UI element drawn on the page. With the “dark launch” bugs fixed, we hope that you enjoy Facebook Chat now that the UI lights have been turned on.

Eugene is a Facebook Engineer