





sFlow.org                                                    Peter Phaal
http://www.sFlow.org/                                        InMon Corp.
info@sflow.org
                                                           Robert Jordan
                                                     ImageMovers Digital




                                                               July 2010


                         sFlow Host Structures



Copyright Notice

Copyright (C) sFlow.org (2010). All Rights Reserved.

Abstract

This memo describes sFlow version 5 structures for exporting host
related data.

Table of Contents

1. Overview ......................................................  1
2. Reference Model ...............................................  2
3. sFlow Datagram Extensions .....................................  3
4. References .................................................... 12
5. Author's Addresses ............................................ 13

1. Overview
   This document describes additional structures that allow an sFlow
   agent to export information about host resources.

   Current trends toward virtualization and convergence tightly link
   networking and system performance and blur the line between the
   network and the servers. For example, virtualization places switching
   and routing functions on servers. Monitoring the network in this
   environment requires that the servers are also monitored. Similarly
   anyone concerned with application and server performance must now
   also be concerned about network performance since the application
   features (such as virtual machine migration) can significantly affect
   and be affected by network performance.

   Currently, performance monitoring of servers and applications is



FINAL                           sFlow.org                       [Page 1]

FINAL                     sFlow Host Structures                July 2010


   highly fragmented. Each server vendor, operating system vendor and
   application developer creates agents and software for performance
   monitoring, none of which interoperate. Monitoring performance of a
   server might require the same (or similar but incompatible metrics)
   be monitored more than once since each management application
   requires its own agent. In addition, there may be other agents
   monitoring different hardware and software elements within the
   server.

   Host sFlow extends sFlow's scalable measurement protocol to include
   server performance monitoring. An interoperable standard for
   exporting host performance metrics breaks the linkage between agents
   and performance analysis applications, providing a standard set of
   metrics that can be collected once and shared among different
   performance analysis tools.

   Extending sFlow to monitor servers unifies network and system
   performance monitoring, delivering the real-time, integrated view of
   performance needed to manage converged services.

   sFlow version 5 is an extensible protocol that allows the addition of
   new data structures without impacting existing collectors. This
   document does not change the sFlow version 5 protocol [1], it simply
   defines additional, optional, data structures that a host device can
   use to report on host resources.

2. Reference Model
   The figure below shows the basic elements of a host.  In this
   abstract model a host consists of one or more physical machines (PM),
   each of which may contain zero or more virtual machines (VM) and/or
   application services (AS).

   +--------------------------------------+
   |   +------------------------------+   |
   |   | PM 1  +------+     +------+  |   |
   |   |       | AS 1 | ... | AS J |  |   |
   |   |       +------+     +------+  |   |
   |   |       +------+     +------+  |   |
   |   |       | VM 1 | ... | VM K |  |   |
   |   |       +------+     +------+  |   |
   |   +------------------------------+   |
   |                     .                |
   |                     .                |
   |                     .                |
   |   +------------------------------+   |
   |   | PM I                         |   |
   |   +------------------------------+   |
   +--------------------------------------+



FINAL                           sFlow.org                       [Page 2]

FINAL                     sFlow Host Structures                July 2010


   The term application service is used to refer to applications
   providing networked services. Examples would include: a web server
   implementing the HTTP protocol [2], a storage array supporting the
   NFS protocol [3], a distributed cache implementing the memcached
   protocol [4], or a distributed computing facility such as Hadoop [5].

   A stand-alone server would just have a single physical machine. A
   blade server may have a large number of physical machines (one per
   blade).

   Physical machines are indicated as PM 1 .. PM I. Application services
   are indicated as AS 1 .. AS J. Virtual machines are indicated as VM 1
   .. VM K.

   Each physical machine, application service and virtual machine is
   monitored by at least one sFlow data source. In this model there is a
   strict containment hierarchy, each physical machine contains
   application services and/or virtual machines. Each virtual machine
   may in turn contain application services.

   The objective of Host sFlow is to export statistics relating to the
   PM, AS and VM entities using a unified data model that permits
   correlation between the host statistics and network statistics
   provided by sFlow agents residing in physical and virtual switches.

   The sFlow version 5 standard requires the implementation of Packet
   Flow Sampling and Counter Sampling within the packet forwarding
   function of a device. In order to be compliant, a host must implement
   sFlow within any physical or virtual switching functions implemented
   within the host.

   However, in the case where no switching function is performed by the
   host, either because it does not contain virtual machines or because
   the inter-VM switching function has been offloaded to the adjacent
   switch (see Edge Virtual Bridging [6]) then the host is not required
   to perform the sFlow Packet Flow Sampling function. The combination
   of sFlow from the adjacent switch and sFlow performance metrics from
   the host provides a complete picture of network and server
   performance.

3. sFlow Datagram Extensions

   The SNMP Entity-MIB [7] can be used to describe the physical and
   logical containment hierarchy of host resources. Physical machines
   (PMs) can be modeled as physical entities, an already supported sFlow
   data source type.  Application services (ASs) and virtual machines
   (VMs) can be modeled as logical entities. Extending sFlow support for
   logical entities provides a means of exporting data related to



FINAL                           sFlow.org                       [Page 3]

FINAL                     sFlow Host Structures                July 2010


   application services and virtual machines.

   The sFlow MIB [1] identifies data sources by SNMP OID, so the only
   change needed is a comment indicating that a logical entity is a
   valid data source type:

        - entLogicalEntry.<L>
          An SFlowDataSource of this form refers to a logical entity
          within the agent and is called a 'logical-entity-based'
          dataSource.

   In addition, a mapping for logical entity data sources in the sFlow
   datagram needs to be specified:

        3 = entLogicalEntry

   These changes are backward compatible with existing sFlow agents. An
   sFlow collector must be able to ignore and skip over the MIB entries
   and data structures related to the logical data source type.

   However, since there is very little functional overlap between Host
   sFlow and existing switch based sFlow, sending Host sFlow to a
   collector that does not support the standard should be avoided. As
   Host sFlow becomes more common, it is likely that many sFlow
   analyzers will be extended to support the new structures in order to
   provide integrated network and system monitoring functionality.

   SNMP[8] is a standard management protocol for network equipment and
   sFlow monitoring of switches is often augmented by additional
   information obtained by SNMP (e.g. ifName, ifStack etc.). However,
   SNMP is much less frequently used in host monitoring. It is important
   that the Host sFlow structures define an internally consistent model
   of the host without depending on SNMP for critical information.
   However, when an SNMP agent is present on the host, the sFlow logical
   and physical entity data sources and their containment hierarchy must
   be consistent with data exported via SNMP.

   The host_parent structure is used to describe the containment
   hierarchy between application services, virtual machines and physical
   machines.

   The host_adapter structure provides the link between host performance
   statistics and sFlow implemented in network equipment. Identifying
   the MAC addresses associated with a physical or virtual network
   adapter allows traffic generated by that adapter to be identified on
   the network.

   Physical machine data sources must export counter_sample structures



FINAL                           sFlow.org                       [Page 4]

FINAL                     sFlow Host Structures                July 2010


   containing host performance statistics (host_cpu, host_memory,
   host_disk_io, host_net_io). Each counter_sample must also include
   host_adapter and host_descr structures.

   If the physical machine is hosting virtual machines then network
   connectivity to the virtual machines is typically provided by a
   virtual switching function. The virtual switch function connects one
   or more of the physical server's physical network adapters with
   virtual network adapters associated with each virtual machine. The
   virtual switch must export sFlow counter_sample and flow_sample
   structures, providing exactly the data that would be available from a
   physical switch. The data sources defined for a switch are typically
   ifIndex data sources and in the case of a virtual switch would
   represent each of the physical and virtual network adapters
   associated with the switch. Again, if an SNMP agent is running on the
   server, the ifIndex numbers exported via sFlow must correspond to the
   ifIndex numbers exported via SNMP.

   A data source associated with each of the virtual machines must
   export counter_sample structures containing performance metrics for
   that virtual machine (virt_cpu, virt_memory, virt_disk_io,
   virt_net_io). Each counter_sample must also include host_adapter,
   host_parent and host_descr structures.  Installing an sFlow agent on
   each virtual machine is not required. Implementing sFlow as part of
   the hypervisor functionality provides access to the virtual switch
   and performance counters for all the virtual machines.  However, if
   an sFlow agent is running on a virtual machine, it must export its
   own "physical server" statistics.

   Application services may export both counter_sample structures and
   flow_sample structures. When sampling at the application level,
   transactions do not have a simple correspondence with network
   packets. A single packet may contain more than one application level
   transaction, or an application level transaction may span many
   packets.  Application level sampling operates on the stream of
   application transactions, sampling a random selection of completed
   transactions and reporting on their attributes.

   For example, a data source monitoring a web server randomly samples
   1-in-N HTTP requests, capturing information about each completed
   request and exports the information as a flow_sample structure.  In
   addition, the data source maintains counters of request totals, types
   and errors and export these counters periodically using a
   counter_sample structure.

   The socket extended_flow structures provide the link between
   application level flow_sample structures and traffic measurements
   made using sFlow in network devices. Each application flow_sample



FINAL                           sFlow.org                       [Page 5]

FINAL                     sFlow Host Structures                July 2010


   must include a socket structure describing the socket on which the
   sampled transaction was received.

   This document creates a framework for exporting application service
   metrics. However, it does not specify application service
   counter_sample or flow_sample structures. Each application service
   will have its own metrics and these will be defined in separate
   documents.

   The goal of sFlow monitoring is to provide a consistent, network wide
   view of performance. In order to achieve this goal with system
   performance monitoring a common set of metrics needs to be defined
   that can be collected in multi-vendor, multi-OS environments.

   The Ganglia Monitoring System [9] has defined a basic set of system
   performance metrics and developed a portable library, libmetrics,
   that allows these metrics to be obtained on a wide variety of
   platforms. Making use of Ganglia's metric definitions to specify the
   sFlow performance counter structures simplifies the implementation of
   a Host sFlow agent, building on a mature and widely accepted set of
   metrics and tools.

   While the sFlow structure definitions are designed to permit an sFlow
   performance monitoring system to yield results that are compatible
   with Ganglia measurements, there are significant architectural
   differences between the two technologies that require some
   explanation.

   The Ganglia agent periodically polls systems counters and computes
   rates.  For example packet counters are polled and packet rates are
   exported (e.g. delta packets / delta time).  In the sFlow
   architecture the sFlow agent exports raw counter values and the sFlow
   analyzer computes deltas and rates.

   Just as Ganglia's libmetrics library defines a set of portable
   metrics for physical server performance monitoring, the libvert
   library [10] defines a standard set of metrics for virtual server
   performance monitoring.  Defining sFlow counter structures based on
   the libvert library simplifies the implementation of a Host sFlow
   agent, making use of a standard library that has been implemented on
   a wide variety of host operating systems and hypervisors. Exporting
   libvert metrics ensures consistency between sFlow performance
   monitoring and other tools making use of libvert.

   The Host sFlow agent [11] is an open source implementation of the
   Host sFlow specification that demonstrates the relationship between
   Host sFlow structures. A single physical entity data source
   periodically exports an sFlow counters_sample containing a



FINAL                           sFlow.org                       [Page 6]

FINAL                     sFlow Host Structures                July 2010


   host_descr, host_adapters, host_cpu, host_memory, host_disk_io and
   host_net_io structure. If virtual machines are present, the physical
   entity data source includes a virt_node structure in its set of
   counters. In addition, a logical entity data source corresponding to
   each virtual machine periodically exports an sFlow counters_sample
   containing a host_descr, host_adapters, host_parent, virt_cpu,
   virt_memory, virt_disk_io and virt_net_io structure.

   The following sFlow structures are defined to export performance and
   dependency information related to physical machines, virtual machines
   and applications:

/* Data structures for exporting Host statistics relating to
   logical and physical entities */

/* The machine_type enumeration may be expanded over time.
   Applications receiving sFlow must be prepared to receive
   host_descr structures with unknown machine_type values.

   The authoritative list of machine types will be maintained
   at www.sflow.org */

enum machine_type {
   unknown = 0,
   other   = 1,
   x86     = 2,
   x86_64  = 3,
   ia64    = 4,
   sparc   = 5,
   alpha   = 6,
   powerpc = 7,
   m68k    = 8,
   mips    = 9,
   arm     = 10,
   hppa    = 11,
   s390    = 12
}

/* The os_name enumeration may be expanded over time.
   Applications receiving sFlow must be prepared to receive
   host_descr structures with unknown machine_type values.

   The authoritative list of machine types will be maintained
   at www.sflow.org */

enum os_name {
   unknown   = 0,
   other     = 1,



FINAL                           sFlow.org                       [Page 7]

FINAL                     sFlow Host Structures                July 2010


   linux     = 2,
   windows   = 3,
   darwin    = 4,
   hpux      = 5,
   aix       = 6,
   dragonfly = 7,
   freebsd   = 8,
   netbsd    = 9,
   openbsd   = 10,
   osf       = 11,
   solaris   = 12
}

/* Physical or virtual host description */
/* opaque = counter_data; enterprise = 0; format = 2000 */
struct host_descr {
   string hostname<64>;       /* hostname, empty if unknown */
   opaque uuid<16>;           /* 16 bytes binary UUID, empty if unknown */
   machine_type machine_type; /* the processor family */
   os_name os_name;           /* Operating system */
   string os_release<32>;     /* e.g. 2.6.9-42.ELsmp,xp-sp3, empty if unknown */
}

/* Physical or virtual network adapter NIC/vNIC */
struct host_adapter {
   unsigned int ifIndex;     /* ifIndex associated with adapter
                                Must match ifIndex of vSwitch
                                port if vSwitch is exporting sFlow
                                0 = unknown */
   mac mac_address<>;        /* Adapter MAC address(es) */
}

/* Set of adapters associated with entity.
   A physical server will identify the physical network adapters
   associated with it and a virtual server will identify its virtual
   adapters. */
/* opaque = counter_data; enterprise = 0; format = 2001 */

struct host_adapters {
   adapter adapters<>;              /* adapter(s) associated with entity */
}

/* Define containment hierarchy between logical and physical
   entities. Only a single, strict containment tree is permitted,
   each entity must be contained within a single parent, but a parent
   may contain more than one child. The host_parent record is used
   by the child to identify its parent. Physical entities form the roots
   of the tree and do not send host_parent structures. */



FINAL                           sFlow.org                       [Page 8]

FINAL                     sFlow Host Structures                July 2010


/* opaque = counter_data; enterprise = 0; format = 2002 */

struct host_parent {
   unsigned int container_type;     /* sFlowDataSource type */
   unsigned int container_index;    /* sFlowDataSource index */
}

/* Extended socket information,
   Must be filled in for all application transactions associated with a network socket
   Omit if transaction associated with non-network IPC  */

/* IPv4 Socket */
/* opaque = flow_data; enterprise = 0; format = 2100 */
struct extended_socket_ipv4 {
   unsigned int protocol;     /* IP Protocol type
                                 (for example, TCP = 6, UDP = 17) */
   ip_v4 local_ip;            /* local IP address */
   ip_v4 remote_ip;           /* remote IP address */
   unsigned int local_port;   /* TCP/UDP local port number or equivalent */
   unsigned int remote_port;  /* TCP/UDP remote port number of equivalent */
}

/* IPv6 Socket */
/* opaque = flow_data; enterprise = 0; format = 2101 */
struct extended_socket_ipv6 {
   unsigned int protocol;     /* IP Protocol type
                                 (for example, TCP = 6, UDP = 17) */
   ip_v6 local_ip;            /* local IP address */
   ip_v6 remote_ip;           /* remote IP address */
   unsigned int local_port;   /* TCP/UDP local port number or equivalent */
   unsigned int remote_port;  /* TCP/UDP remote port number of equivalent */
}

/*
  Physical server performance metrics
*/

/* Physical Server CPU */
/* opaque = counter_data; enterprise = 0; format = 2003 */

struct host_cpu {
   float load_one;              /* 1 minute load avg., -1.0 = unknown */
   float load_five;             /* 5 minute load avg., -1.0 = unknown */
   float load_fifteen;          /* 15 minute load avg., -1.0 = unknown */
   unsigned int proc_run;       /* total number of running processes */
   unsigned int proc_total;     /* total number of processes */
   unsigned int cpu_num;        /* number of CPUs */
   unsigned int cpu_speed;      /* speed in MHz of CPU */



FINAL                           sFlow.org                       [Page 9]

FINAL                     sFlow Host Structures                July 2010


   unsigned int uptime;         /* seconds since last reboot */
   unsigned int cpu_user;       /* user time (ms) */
   unsigned int cpu_nice;       /* nice time (ms) */
   unsigned int cpu_system;     /* system time (ms) */
   unsigned int cpu_idle;       /* idle time (ms) */
   unsigned int cpu_wio;        /* time waiting for I/O to complete (ms) */
   unsigned int cpu_intr;       /* time servicing interrupts (ms) */
   unsigned int cpu_sintr;      /* time servicing soft interrupts (ms) */
   unsigned int interrupts;     /* interrupt count */
   unsigned int contexts;       /* context switch count */
}

/* Physical Server Memory */
/* opaque = counter_data; enterprise = 0; format = 2004 */

struct host_memory {
    unsigned hyper mem_total;   /* total bytes */
    unsigned hyper mem_free;    /* free bytes */
    unsigned hyper mem_shared;  /* shared bytes */
    unsigned hyper mem_buffers; /* buffers bytes */
    unsigned hyper mem_cached;  /* cached bytes */
    unsigned hyper swap_total;  /* swap total bytes */
    unsigned hyper swap_free;   /* swap free bytes */
    unsigned int page_in;       /* page in count */
    unsigned int page_out;      /* page out count */
    unsigned int swap_in;       /* swap in count */
    unsigned int swap_out;      /* swap out count */
}

/* Physical Server Disk I/O */
/* opaque = counter_data; enterprise = 0; format = 2005 */

struct host_disk_io {
   unsigned hyper disk_total;    /* total disk size in bytes */
   unsigned hyper disk_free;     /* total disk free in bytes */
   percentage  part_max_used;    /* utilization of most utilized partition */
   unsigned int reads;           /* reads issued */
   unsigned hyper bytes_read;    /* bytes read */
   unsigned int read_time;       /* read time (ms) */
   unsigned int writes;          /* writes completed */
   unsigned hyper bytes_written; /* bytes written */
   unsigned int write_time;      /* write time (ms) */
}

/* Physical Server Network I/O */
/* opaque = counter_data; enterprise = 0; format = 2006 */

struct host_net_io {



FINAL                           sFlow.org                      [Page 10]

FINAL                     sFlow Host Structures                July 2010


   unsigned hyper bytes_in;  /* total bytes in */
   unsigned int pkts_in;     /* total packets in */
   unsigned int errs_in;     /* total errors in */
   unsigned int drops_in;    /* total drops in */
   unsigned hyper bytes_out; /* total bytes out */
   unsigned int packets_out; /* total packets out */
   unsigned int errs_out;    /* total errors out */
   unsigned int drops_out;   /* total drops out */
}

/*
  Hypervisor and virtual machine performance metrics
*/

/* Virtual Node Statistics */
/* See libvirt, struct virtNodeInfo */
/* opaque = counter_data; enterprise = 0; format = 2100 */

struct virt_node {
   unsigned int mhz;           /* expected CPU frequency */
   unsigned int cpus;          /* the number of active CPUs */
   unsigned hyper memory;      /* memory size in bytes */
   unsigned hyper memory_free; /* unassigned memory in bytes */
   unsigned int num_domains;   /* number of active domains */
}

/* Virtual Domain CPU statistics */
/* See libvirt, struct virtDomainInfo */
/* opaque = counter_data; enterprise = 0; format = 2101 */

struct virt_cpu {
   unsigned int state;         /* virtDomainState */
   unsigned int cpuTime;       /* the CPU time used (ms) */
   unsigned int nrVirtCpu;     /* number of virtual CPUs for the domain */
}

/* Virtual Domain Memory statistics */
/* See libvirt, struct virtDomainInfo */
/* opaque = counter_data; enterprise = 0; format = 2102 */

struct virt_memory {
   unsigned hyper memory;      /* memory in bytes used by domain */
   unsigned hyper maxMemory;   /* memory in bytes allowed */
}

/* Virtual Domain Disk statistics */
/* See libvirt, struct virtDomainBlockInfo */
/* See libvirt, struct virtDomainBlockStatsStruct */



FINAL                           sFlow.org                      [Page 11]

FINAL                     sFlow Host Structures                July 2010


/* opaque = counter_data; enterprise = 0; format = 2103 */

struct virt_disk_io {
   unsigned hyper capacity;   /* logical size in bytes */
   unsigned hyper allocation; /* current allocation in bytes */
   unsigned hyper available;  /* remaining free bytes */
   unsigned int rd_req;       /* number of read requests */
   unsigend hyper rd_bytes;   /* number of read bytes */
   unsigned int wr_req;       /* number of write requests */
   unsigned hyper wr_bytes;   /* number of  written bytes */
   unsigned int errs;         /* read/write errors */
}

/* Virtual Domain Network statistics */
/* See libvirt, struct virtDomainInterfaceStatsStruct */
/* opaque = counter_data; enterprise = 0; format = 2104 */

struct virt_net_io {
   unsigned hyper rx_bytes;  /* total bytes received */
   unsigned int rx_packets;  /* total packets received */
   unsigned int rx_errs;     /* total receive errors */
   unsigned int rx_drop;     /* total receive drops */
   unsigned hyper tx_bytes;  /* total bytes transmitted */
   unsigned int tx_packets;  /* total packets transmitted */
   unsigned int tx_errs;     /* total transmit errors */
   unsigned int tx_drop;     /* total transmit drops */
}


4. References

[1]  Phaal, P. and Lavine, M., "sFlow Version 5",
     http://www.sflow.org/sflow_version_5.txt, July 2006

[2]  IETF, "RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1", June
     1999

[3]  IETF, "RFC 1813: NFS Version 3 Protocol Specification", June 1995

[4]  "Memcached", http://memcached.org/

[5]  "Hadoop", http://hadoop.apache.org/

[6]  IEEE, "802.1Qbg - Edge Virtual Bridging"

[7]  IETF, "RFC 2737: Entity MIB (Version 2)", December 1999





FINAL                           sFlow.org                      [Page 12]

FINAL                     sFlow Host Structures                July 2010


[8]  Case, J., Fedor, M., Schoffstall, M., and J. Davin, "Simple Network
     Management Protocol", STD 15, RFC 1157, May 1990.

[9]  "Ganglia Monitoring System", http://ganglia.sourceforge.net/

[10] "libvirt: virtualization API", http://libvirt.org/

[11] "Host sFlow", http://host-sflow.sourceforge.net/

5. Author's Address

   Peter Phaal
   InMon Corp.
   580 California Street, 5th Floor
   San Francisco, CA 94104

   Phone: (415) 283-3263
   EMail: peter.phaal@inmon.com

   Robert Jordan
   ImageMovers Digital
   9 Hamilton Landing
   Novato, CA 94949

   Phone: (415) 475-5800
   EMail: rjordan@imagemoversdigital.com

























FINAL                           sFlow.org                      [Page 13]

