qemu/docs/colo-proxy.txt

COLO-proxy
----------
Copyright (c) 2016 Intel Corporation
Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
Copyright (c) 2016 Fujitsu, Corp.

This work is licensed under the terms of the GNU GPL, version 2 or later.
See the COPYING file in the top-level directory.

This document gives an overview of COLO proxy's design.

== Background ==
COLO-proxy is a part of COLO project. It is used
to compare the network package to help COLO decide
whether to do checkpoint. With COLO-proxy's help,
COLO greatly improves the performance.

The filter-redirector, filter-mirror, colo-compare
and filter-rewriter compose the COLO-proxy.

== Architecture ==

COLO-Proxy is based on qemu netfilter and it's a plugin for qemu netfilter
(except colo-compare). It keep Secondary VM connect normally to
client and compare packets sent by PVM with sent by SVM.
If the packet difference, notify COLO-frame to do checkpoint and send
all primary packet has queued. Otherwise just send the queued primary
packet and drop the queued secondary packet.

Below is a COLO proxy ascii figure:

 Primary qemu                                                           Secondary qemu
+--------------------------------------------------------------+       +----------------------------------------------------------------+
| +----------------------------------------------------------+ |       |  +-----------------------------------------------------------+ |
| |                                                          | |       |  |                                                           | |
| |                        guest                             | |       |  |                        guest                              | |
| |                                                          | |       |  |                                                           | |
| +-------^--------------------------+-----------------------+ |       |  +---------------------+--------+----------------------------+ |
|         |                          |                         |       |                        ^        |                              |
|         |                          |                         |       |                        |        |                              |
|         |  +------------------------------------------------------+  |                        |        |                              |
|netfilter|  |                       |                         |    |  |   netfilter            |        |                              |
| +----------+ +----------------------------+                  |    |  |  +-----------------------------------------------------------+ |
| |       |  |                       |      |        out       |    |  |  |                     |        |  filter execute order      | |
| |       |  |          +-----------------------------+        |    |  |  |                     |        | +------------------->      | |
| |       |  |          |            |      |         |        |    |  |  |                     |        |   TCP                      | |
| | +-----+--+-+  +-----v----+ +-----v----+ |pri +----+----+sec|    |  |  | +------------+  +---+----+---v+rewriter++  +------------+ | |
| | |          |  |          | |          | |in  |         |in |    |  |  | |            |  |        |              |  |            | | |
| | |  filter  |  |  filter  | |  filter  +------>  colo   <------+ +-------->  filter   +--> adjust |   adjust     +-->   filter   | | |
| | |  mirror  |  |redirector| |redirector| |    | compare |   |  |    |  | | redirector |  | ack    |   seq        |  | redirector | | |
| | |          |  |          | |          | |    |         |   |  |    |  | |            |  |        |              |  |            | | |
| | +----^-----+  +----+-----+ +----------+ |    +---------+   |  |    |  | +------------+  +--------+--------------+  +---+--------+ | |
| |      |   tx        |   rx           rx  |                  |  |    |  |            tx                        all       |  rx      | |
| |      |             |                    |                  |  |    |  +-----------------------------------------------------------+ |
| |      |             +--------------+     |                  |  |    |                                                   |            |
| |      |   filter execute order     |     |                  |  |    |                                                   |            |
| |      |  +---------------->        |     |                  |  +--------------------------------------------------------+            |
| +-----------------------------------------+                  |       |                                                                |
|        |                            |                        |       |                                                                |
+--------------------------------------------------------------+       +----------------------------------------------------------------+
         |guest receive               | guest send
         |                            |
+--------+----------------------------v------------------------+
|                                                              |                          NOTE: filter direction is rx/tx/all
|                         tap                                  |                          rx:receive packets sent to the netdev
|                                                              |                          tx:receive packets sent by the netdev
+--------------------------------------------------------------+

1.Guest receive packet route:

Primary:

Tap --> Mirror Client Filter
Mirror client will send packet to guest,at the
same time, copy and forward packet to secondary
mirror server.

Secondary:

Mirror Server Filter --> TCP Rewriter
If receive packet is TCP packet,we will adjust ack
and update TCP checksum, then send to secondary
guest. Otherwise directly send to guest.

2.Guest send packet route:

Primary:

Guest --> Redirect Server Filter
Redirect server filter receive primary guest packet
but do nothing, just pass to next filter.

Redirect Server Filter --> COLO-Compare
COLO-compare receive primary guest packet then
waiting secondary redirect packet to compare it.
If packet same,send queued primary packet and clear
queued secondary packet, Otherwise send primary packet
and do checkpoint.

COLO-Compare --> Another Redirector Filter
The redirector get packet from colo-compare by use
chardev socket.

Redirector Filter --> Tap
Send the packet.

Secondary:

Guest --> TCP Rewriter Filter
If the packet is TCP packet,we will adjust seq
and update TCP checksum. Then send it to
redirect client filter. Otherwise directly send to
redirect client filter.

Redirect Client Filter --> Redirect Server Filter
Forward packet to primary.

== Components introduction ==

Filter-mirror is a netfilter plugin.
It gives qemu the ability to mirror
packets to a chardev.

Filter-redirector is a netfilter plugin.
It gives qemu the ability to redirect net packet.
Redirector can redirect filter's net packet to outdev,
and redirect indev's packet to filter.

                    filter
                      +
          redirector  |
             +--------------+
             |        |     |
             |        |     |
             |        |     |
  indev +---------+   +---------->  outdev
             |    |         |
             |    |         |
             |    |         |
             +--------------+
                  |
                  v
                filter

COLO-compare, we do packet comparing job.
Packets coming from the primary char indev will be sent to outdev.
Packets coming from the secondary char dev will be dropped after comparing.
COLO-compare needs two input chardevs and one output chardev:
primary_in=chardev1-id (source: primary send packet)
secondary_in=chardev2-id (source: secondary send packet)
outdev=chardev3-id

Filter-rewriter will rewrite some of secondary packet to make
secondary guest's tcp connection established successfully.
In this module we will rewrite tcp packet's ack to the secondary
from primary,and rewrite tcp packet's seq to the primary from
secondary.

== Usage ==

Here is an example using demonstration IP and port addresses to more
clearly describe the usage.

Primary(ip:3.3.3.3):
-netdev tap,id=hn0,vhost=off
-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off
-chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off
-chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off
-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off
-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
-object iothread,id=iothread1
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1

Secondary(ip:3.3.3.8):
-netdev tap,id=hn0,vhost=off
-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
-chardev socket,id=red0,host=3.3.3.3,port=9003
-chardev socket,id=red1,host=3.3.3.3,port=9004
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
-object filter-rewriter,id=f3,netdev=hn0,queue=all

If you want to use virtio-net-pci or other driver with vnet_header:

Primary(ip:3.3.3.3):
-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off
-chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off
-chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off
-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off
-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0,vnet_hdr_support
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out,vnet_hdr_support
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0,vnet_hdr_support
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,vnet_hdr_support

Secondary(ip:3.3.3.8):
-netdev tap,id=hn0,vhost=off
-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
-chardev socket,id=red0,host=3.3.3.3,port=9003
-chardev socket,id=red1,host=3.3.3.3,port=9004
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0,vnet_hdr_support
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1,vnet_hdr_support
-object filter-rewriter,id=f3,netdev=hn0,queue=all,vnet_hdr_support

Note:
  a.COLO-proxy must work with COLO-frame and Block-replication.
  b.Primary COLO must be started firstly, because COLO-proxy needs
    chardev socket server running before secondary started.
  c.Filter-rewriter only rewrite tcp packet.