$Z

From NMDC Protocol

$Z / ZLine

The $Z command is used by the NMDC ZLine Extension, a proposed extension to the NMDC protocol

Abstract
This proposal is by Jove and has been discussed in the DCDev Public hub, but is being documented here.

The proposal is to add a ZLine ($Z) command that returns a block of zipped commands.

  $Z <blob>|

would unzip to

  $Search ... |$MyINFO...|...

The proposal for now is to use simple zlib compression and adding a minimal escaping. The escaping for now is: replace "\" by "\\" and "|" by "\P" . It would be trivial to use bzip instead.

This would only be used for message passing from hub to client. The other way around is not useful and only generates extra load on the hub.

The Client must send a $Supports flag called "ZLine"

Evaluation
I have performed a few basic tests to evaluate the effectiveness of the extension.

The load test program runs in 2 stages. First, it joins all the users in the test, then all those users start generating "events". An event is:

• a $MyINFO
• a chat message
• a $Search
• a disconnect/connect

Delays between user joins and events can be programmed. Default settings are 100ms between user joins (10joins/s) and 50ms between events (20 events/s). Unless specified, the test was performed with default settings.

A short test (about 700s) with 100 users.

 Without ZLine
   Total traffic since boot: In 1986998, Out 58506319
   Average over 718 seconds: In 16.140272 kbps, Out 475.764798 kbps

 With ZLine
   Total traffic since boot: In 1549938, Out 15913487
   Average over 719 seconds: In 16.159448 kbps, Out 165.737719 kbps

This test has 1000 users connected to the hub. The test ran over 22000 seconds (a little over 6 hours) in both cases.

 Without ZLine
   Total traffic since boot: In 58562006, Out 18932614671
   Average over 28078 seconds: In 16.682808 kbps, Out 5393.331647 kbps

 With ZLine
   Total traffic since boot: In 48040395, Out 3987767021
   Average over 22318 seconds: In 16.989433 kbps, Out 1410.093335 kbps

This test has 4000 users connected to the hub. The test ran over 5000 seconds in both cases.

 Without ZLine
   Total traffic since boot: In 11675512, Out 14732322239
   Average over 5472 seconds: In 16.910506 kbps, Out 21337.779879 kbps

 With ZLine
   Total traffic since boot: In 11102441, Out 2805387234
   Average over 5042 seconds: In 17.197069 kbps, Out 4343.892306 kbps

Below tests were conducted with different messages. The basic message as above and a number of random characters. This results in larger messages that are harder to compress.

1000 users

 Without ZLine
   Total traffic since boot: In 26426307, Out 8802244496
   Average over 9358 seconds: In 22.585492 kbps, Out 7522.500261 kbps

 With Zline
   Total traffic since boot: In 27070505, Out 4003114560
   Average over 9359 seconds: In 22.837603 kbps, Out 3376.263386 kbps

Conclusion The bandwidth saved lies anywhere between 50% and 80% depending on the amount of users and the exact kind of traffic. I would also like to point out that the hub software used (my own, called Aquila) does not zip everything. The hub software aims at a nice balance between bandwidth and CPU/memory resources. In essence, only blocks of commands send to all users are zipped. This keeps the extra CPU/memory used almost negligible

Deployment
Hubs:

Aquila (Jove's hub software)
BDCH (planned)
DDCH
tkhub
YnHub (promised)
NitroHub (over 0.7)

clients:

DCDM++
iDC++
McDC++
Z++

bots:

StatBot (2.6.6 <) [[1]]