Installing and Configuring Intel(r) Advanced Networking Services (ANS) on the Linux 2.2.x and 2.4.0 Kernels Contents: Overview & Environment Installation of Advanced Networking Services Configuring the Adapters and Teams Saving the Configuration across Reboots ) Removing Advanced Networking Services Modifying Advanced Networking Services Configurations Available Commands Example of Setting Up an AFT team Troubleshooting (also includes file distribution and conf.mod example) License Overview and Environment ======================= This package was tested on Red Hat* Linux* (Linux kernel 2.2.16, 2.2.18 and 2.4.0) on both UP and SMP Intel processor based PCs. Installation assumes you have all necessary kernels and build tools loaded. The package supports the following teaming modes: * AFT ( 10/100 and 1000) Adapter Fault Tolerance * ALB ( 10/100 or 1000) Adaptive Load Balancing (includes AFT) * FEC (10/100) or GEC (1000) Link Aggregation/FEC/GEC Descriptions of the teaming modes and their specific requirements are: - AFT (Adapter Fault Tolerance). Default mode. A secondary adapter in the team will automatically take over on any failure of the primary connection (cable, adapter or port). Only one card is active at a time. After a failover, if the connection to the primary adapter is restored, control passes automatically back to that primary adapter. May mix speed and duplex capabilities and settings. May use a hub or any switch. - ALB (Adaptive Load Balancing or asymmetrical link aggregation). A team of 2-8 adapters share the transmission load while the primary receives and transmits. Includes fault tolerance. Must match speed/duplex settings on all team members. May use any switch. - Link Aggregation (Cisco*'s Fast EtherChannel* Technology (FEC) or Intel Link Aggregation). A team of 2-8 10/100 adapters which simultaneously receive and transmit. Includes fault tolerance and load balancing. Must match speed/duplex settings on all team members Requires a switch that supports Intel Link Aggregation or Cisco's FEC. Spanning Tree Protocol must be turned off. Must match switch aggregation requirements. - GEC (Gigabit equivalent of FEC). Requires GEC compatible switch and PRO/1000 Adapters. Roles - For AFT and ALB, you may specify which adapters in a team you want to use as the Primary and Secondary adapters. The Primary will be the main adapter to be used in a team. For fault tolerance, the Primary is the only adapter used until a failure. For ALB, the Primary is the adapter that both transmits and receives. The Secondary is the preferred adapter (in a team consisting of more than 2 adapters) to fill the role of Primary adapter if it fails. If Priority is not specified, the adapter with the highest supported speed will be chosen. If all adapters support the same speed, the last one added will be the primary. These Advanced Networking Services only work with Intel adapters and each team requires at least one Server adapter. Linux is not supported on the Intel(R) PRO/100 Intelligent Server Adapter. IEEE 802 Virtual LANs (VLANs) are based on the addition of a tag to the packet header. All equipment on the network that will pass this traffic must be capable of handling the extra length. Settings on the adapter must match the VLAN settings on the switch. VLANs also add to load time and require additional memory resources. There is a limit of 64 VLANs per server. However VLANs may be numbered from 1-4096. Installation of the ADVANCED NETWORKING SERVICES Package ======================================================== Please install and compile the base driver package(s) per the readme included with those drivers. Verify that all interfaces are working and that you have the basedriver file(s) (e100.o, e1000.o) in lib/modules//net/. 1. Login as root. 2. Copy the iANS-x.x.x.tar.gz package file to a directory of your choice. 3. Open the package file by running 'tar xzf iANS-x.x.x.tar.gz' This will create iANS-x.x.x as a subdirectory under the current directory. 4. Enter the iANS-x.x.x directory. At this point, choose between the 2 separate methods for installation: 1. Manual (suggested for experienced users) 2. Scripted (suggested ONLY for Red Hat 6.x, SuSE* 6.4, Caldera* 2.2, and TurboLinux* Server 6.0 users) Manual Installation and Configuration ------------------------------------- 1. From the directory iANS-x.x.x run 'make' to compile the module. NOTE: SMP/UP and MODVERSIONS are determined by 'make'. If there is a mismatch between the configuration for the kernel source tree and the kernel itself, run 'make options" to list options available for overriding the source tree settings. These include SMP, MODVERSIONS, and NODEBUG (not listed). NODEBUG values are 0 (debugging is on), 1 (debugging is off). Default is 0. 2. Run 'make install'. (Default file locations may be changed by editing the paths at the top of Makefile - see Troubleshooting section.) There will not be any messages after make install. See the Troubleshooting section for a list of files and their default locations. 3. You must use Base Driver version 1.4.18 (or higher) for PRO/100 and/or Base Driver version 2.6.x (or higher) for PRO/1000 for ANS support. NOTE: Make sure no other base drivers are loaded (nor compiled into the kernel, for any of the Intel adapters. (e.g. eepro, eepro100, eexpress, e100,e1000). Be sure the configuration files have been edited to alias the new driver(s) and not any previous drivers. Please see the trouble- shooting section for an example file. 4. After the required base drivers are loaded, down any Intel adapters that are to be added to a team. (If they already have an IP address, run ifconfig ethx 0.0.0.0 to remove the IP number.) ifconfig - a - will list all current network devices ifconfig ethx down - will down adapters so ANS may be loaded Note: You should leave other vendor's Ethernet adapters configured as well as any Intel adapters that you will not adding to teams. This will ensure that network device name assignment remains consistent after reboot. 5. Load the Advanced Networking Services module by running 'insmod ians'. In the following command lines, optional items are placed in [ ]. All VLAN settings are shown as optional. 6. Create team(s); ianscfg -a -t [-M] [-V] Use the -M flag to specify the teaming mode (default AFT, others NONE | ALB | FEC | GEC). Use the -V flag to specify a VLAN . Use -M NONE if you wish VLANs on single adapters without link aggregation. Any other mode may be specified to use VLANs and link aggregation simultaneously. 7. Determine the teaming mode (AFT, ALB, FEC, GEC) for each team. ianscfg -C -t -M NOTE: team mode can be specified in either steps 6 or 7 8. Add Intel adapters to the team and specify role (priority) ianscfg -a -t -m [-p ] Priority may be None, Primary, or Secondary 9. Create virtual adapters. ianscfg -a -t -v [-i] vadapter_name may be anything except ethx as long as it ends with an instance suffix specify the -i options (vlan_id) only on VLAN based teams or adapters. 10. Activate the team ianscfg -c 11. Activate the virtual adapter Note: This binds the protocol to the VIRTUAL adapter. ifconfig netmask [broadcast ] 12. Check team status ianscfg -s Congratulations, the team is now active!! You should be able to send and receive packets via the virtual adapter. ---------------------------------------- Scripted Installation and Configuration ======================================== For ease of configuration, Intel is supplying interactive scripts - which are only supported on: Red Hat* (6.1, 6.2) SuSE* (6.4) Caldera* (2.2) TurboLinux* (6.0 server) ANS will set up 4 directories named Red Hat, SuSE, Caldera, and Turbo to match each of these distributions under iANS-x.x.x/install/ To use the install script, enter iANS-x.x.x/install/[dist_name] and type ./INSTALL NOTE: The module is created with support for the SMP/UP and MODVERSIONS features as defined by the configuration of the current kernel source tree (/usr/src/linux/). On some systems, there might be a mismatch between this configuration and the kernel image itself. If there is a mismatch, it will not be apparent until further in the process. Run 'INSTALL options' to see extra compilation switches that allow overriding the kernel source tree settings. Then follow the instructions displayed on the console. You will be asked some questions during the Process: 1. "Would you like the configuration utility to remove all network interfaces automatically?" The installation process must bring down the network interfaces related to Intel adapters, in order to unload the old Intel drivers and replace them with the new ones. Select Yes (default) removes all network interfaces (including other brands). Select No to choose for each interface. 2. "Enter team name,(Enter) to end." You must enter either a team name (often a number) or press Enter (CR) to quit adding teams (skips to Step 6). "Would you like to VLANs in team (name)" - Yes will create a VLAN-based team. "Select members for team (name)" will list all applicable interfaces and you need to type in the number of the adapter to add to the team. Select the role (none, primary or secondary). Repeat for each adapter (enter 0 when done.) 3. "Enter virtual adapter name for team (name)" can be anything but ethx and must end in an instance suffix. "Enter VLAN id for virtual adapter (name)" (for VLAN-based teams only) enter a number 1-4095 to specify the Virtual LAN. "Enter boot protocol (none, bootp, dhcp, or other)", If choose none will need to manually enter IP address and netmask. If choose bootp or DHCP must be able to reach another machine assigning the IP address and netmask. If choose Other must enter protocol name (such as IP) NOTE: This information will NOT be saved until you confirm your selection after configuring the team. 4. "Select Teaming Mode for Team (name) - None\AFT\ALB\FEC\GEC" - for descriptions see the first section of the readme. For Intel Link Aggregation use FEC or GEC. WARNING: Choosing None will still create a team! 5. "Are you sure?." Yes commits the team. Choosing No will make you start again at step3. After commit, you may make another team, or press Enter to proceed. 6. "Would you like to save this configuration and load it on boot? " Yes (default) - modifies init file. Choose which run-levels will have the ANS module automatically loaded. (Recommendation: select same level as the network boot - commonly 2,3,4, or 5) No - means the configuration will not be saved for boot. 7. "Would you like to start the network now?" Yes - activates the team No - should still show installation and configuration completed successfully. ANS will start at reboot if configuration was saved in step 4. ---------------------------------------------------------------- Saving ADVANCED NETWORKING SERVICES configuration across reboots =============================================================== NOTES: Do not use if setting up via Scripted configuration method. Sample lines are Red Hat 6.x specific - please modify for your distribution. Now that the Ethernet and ANS drivers are configured, follow the instructions below to configure the system to activate ANS the same way every time it boots. 1. Save the ANS configuration to a file by running: ianscfg -w [-f] (default is /etc/ians/ians.conf) 2. Make sure that the adapters that were selected as members of the team(s) are NOT configured on boot. To do so, change "ONBOOT=yes" to "ONBOOT=no" in every matching /etc/sysconfig/network-scripts/ifcfg-ethX file. 3. Set each virtual adapter to configure on boot by creating a config file /etc/sysconfig/network-scripts/ifcfg-, which should look something like this: DEVICE= BOOTPROTO="none" ONBOOT="yes" IPADDR="192.168.1.1" (sample) NETMASK="255.255.255.0" (sample) 4. Create an init script that will load the e100.o, e1000.o (if needed) and ians.o modules, and configure the ANS topology. You can use the sample script file by copying iANS-x.x.x/install/Red Hat/ans to /etc/rc.d/init.d/ans Configuration will be read from /etc/ians/ians.conf by default. Modify the script if you wish to use any other file. 5. Add the ANS script to the init process of the appropriate run levels. Run 'chkconfig --level ans on' where is the run levels you would like ANS to start in. NOTE: This script must be invoked BEFORE the system starts the network services. For example, if your network init script for run level 3 is /etc/rc.d/rc3.d/S10network, you should create a symbolic link /etc/rc.d/rc3.d/S09ans that points to /etc/rc.d/init.d/ans. ANS service is automatically stopped when the network service is stopped. ---------------------------------------------------------------------------- Removing ADVANCED NETWORKING SERVICES ===================================== There are several stages to removing Advanced Networking Services. * Bringing down all vadapter interfaces. * Removing the ANS configuration from future reboots. * Removing the ANS module * Removing the files from the system There are two methods for removing ANS: manual and scripted. Manual method of Removing ANS ----------------------------- 1. Login as root. 2. Run 'ifconfig -a' to see every virtual adapter configured in the system. 3. For each virtual adapter, run 'ifconfig down' 4. Remove the ians module by running 'rmmod ians' 5. In each virtual adapter file, either modify the file to contain the line ONBOOT="no" instead of ONBOOT="yes" or remove the file completely. 6. For each file for stand alone adapters, modify the file to contain the line ONBOOT="yes" instead of ONBOOT="no". 7. You may leave the Intel base driver or you may edit the modules config file (usually /etc/conf.modules or /etc/modules.conf) and restore its original form by removing the lines containing e100 or e1000 as the drivers that were added and restoring every line that was commented out. 8. Remove the ANS initialization script and all links to it. (e.g. /etc/rc.d/init.d/ans and /etc/rc.d/rc.3/S09ans). 9. From iANS-x.x.x directory, run 'make uninstall' to remove all the files Scripted Method (only those distributions covered by scripted install) ---------------------------------------------------------- 1. Login as root. 2. Enter the iANS-x.x.x/install/[distribution_name] directory. 3. Run 'UNINSTALL'. 4. The rest of the process is interactive. Follow the instructions displayed on the console. Prompts: Restore member adapters to original role (stand alone). Answer Yes for each adapter you wish to continue using standalone. This configuration will be made using Intel's e100(0).o driver although you may restore other drivers. Removal of the ANS module from memory will not down the adapters running in stand alone mode. The script will remove all files, scripts, and links created except the /etc/ians/ians.conf. Restarting the Network with the Original Configuration For the new settings to take, you will have to restart the network service either by rebooting the system or by running the network init script with the restart parameter. /etc/rc.d/init.d/network restart Modifying ADVANCED NETWORKING SERVICES Configuration ==================================================== ANS configuration and topology may be changed, with a few limitations. After commit, you may change priority and add or remove a team member (but not the last one). You may not: Change team mode. Remove a team or a virtual adapter. Remove the last member of an active team. Teams, members adapters and virtual adapters may be added on the fly while the service is running via the hot add and hot remove features (using the same set of commands). (Not to be confused with Hot PCI Add/Remove.) There are 2 methods of reconfiguring - Semi-automatic - good for changing mode and/or moving members between teams (can't create new teams plus all adapters must remain in teams). - Manual reconfiguration (see below) Semi-automatic Reconfiguration ------------------------------ 1. Manually modify the ANS topology description file (/etc/ians/ians.conf). 2. Call the ANS initialization script with the restart parameter /etc/rc.d/init.d/ans restart. 3. To reactivate, run the network service initialization script with the restart parameter. /etc/rc.d/init.d/network restart Manual Reconfiguration ----------------------- 1. Take down the virtual adapter ifconfig down 2. Remove the ians module rmmod ians 3. Create or modify the ians topology file to match the new desired topology (default location /etc/ians/ians.conf), or create script containing a set of calls to ianscfg that will generate the desired topology. 4. Load the ians module insmod ians 5. Run the script created in step 3, or run the command: ianscfg -r [-f] 6. Reset each created virtual adapter ifconfig netmask broadcast 7. Save the new configuration: ianscfg -w [-f] Create an interface configuration file for each new virtual adapter. * The files would normally reside in /etc/sysconfig/network-scripts/ * The name format should be ifcfg- * Sample file content: DEVICE= BOOTPROTO="none" ONBOOT="yes" IPADDR="192.168.1.1" NETMASK="255.255.255.0" 8. If any unnecessary virtual adapter configuration file is left, remove it or modify it. 9. For any adapter that is desired as stand alone connection, you may continue to use the Intel base driver. List of Available Commands: ============================= Create a Team: Create team(s); ianscfg -a -t [-M] [-V] Use the -M flag to specify the teaming mode (default AFT, others NONE | ALB | FEC | GEC). Use the -V flag to specify a VLAN . Set/Change team mode: (only before commit) ianscfg -C -t -M Add team members: ianscfg -a -t -m [-p Y] where Y can be none, primary or secondary, optional (Only existing Intel Ethernet devices may be added to a team) Add virtual adapter ianscfg -a -t -v [-i] NOTE: On non-VLAN teams, define ONLY one virtual adapter per team. On VLAN teams, the -i options (vlan_id) must be specified. The name of the virtual adapter can be any except ethx but must include instance number suffix. Activate (commit) the team: ianscfg -c After a team has been activated you can still add/remove members. NOTE: At this point the virtual adapter appears (ifconfig -a) in the adapters list but it is in the down state. Activate the virtual adapter: ifconfig netmask broadcast Get Status: ianscfg -s Get help on the ianscfg utility - Type 'ianscfg' To remove a member: ianscfg -d -m Note: You cannot remove the last member of a team. Don't specify the team here. To Disable the Virtual Adapter: ifconfig down To remove a team (only before commit): ianscfg -d -t To remove a virtual adapter: (only before commit) ianscfg -d -v To disable a virtual adapter: ifconfig down To Remove the ANS Module: (down all vadapters first) rmmod iANS To remove ANS files from your system: (after downing vadapters and resetting the configuration) make uninstall (from the iANS-x.x.x directory) To Save the current configuration in a file: (only after commit) ianscfg -w [-f ] (default is /etc/ians/ians.conf - editable text file) To automatically program a topology configuration from a file: ianscfg -r [-f ] (Again, the default file is /etc/ians/ians.conf) Turn off probes ianscfg -P -t -d ------------------------------------------------------------------- Example File for Setting up AFT mode with VLANs: (2 PRO/100 adapters) ===================================================================== insmod e100 insmod ians ianscfg -a -t team1 -M AFT -V ianscfg -at team1 -m eth0 -p primary ianscfg -at team1 -m eth1 -p secondary ianscfg -at team1 -v vadapt1 -i 10 ianscfg -at team1 -v vadapt2 -i 15 ianscfg -c team1 ianscfg -s ifconfig vadapt1 143.185.254.10 netmask 255.255.255.0 ifconfig vadapt2 143.185.254.10 netmask 255.255.255.0 ------------------------------------------------------------------- Troubleshooting 1. If an adapter has been running standalone with an assigned IP address you must configure the IP address to 0.0.0.0 before adding it to a team. ifconfig ethx {0.0.0.0}. 2. Always bind the protocol to the virtual adapter and NOT to the physical adapter for adapters that are in a team. (You still bind the protocol to the physical adapter for standalone adapters.) 3. If you do not remove references to other possible base drivers in the configurations, you risk multiple interfaces for adapters you wish to use in the team with an unpredictable outcome. 4. If you receive an error message "module not loaded" during configuration of the team, you need to start over. To do so type ./INSTALL veryclean all SMP=[0|1] MODVERSIONS=[0|1] choose 0 or 1 for SMP and MODVERSIONS to match the configuration of the current kernel tree. Cause: This will occur when the target module doesn't match the kernel version and install was done without parameters (found using ./INSTALL options) 5. If when adding the ANS script to the init process, you receive an error message : 'service ans does not support chkconfig' then create the symbolic link (to the ans file) manually for each run level desired with the command: ln -s /etc/rc.d/init.d/ans /etc/rc.d/rcX.d/S09ans where X is the number of the desired run level Structure of the ANS build tree * kernel dependant source files (src/*.[ch], src/lib/*.[ch]) * kernel independent precompiled object file (bin/iANS_core.o). * Makefiles. * Advanced Networking Services configuration utility (bin/ianscfg). * installation, uninstall and configuration scripts Distribution of files: By default, the components are distributed as follows: - /lib/modules/[KERNEL_VERSION]/net/ians.o - /usr/sbin/ianscfg - /usr/man/man1/ianscfg.1 conf.modules file example #alias eth0 eepro100 #alias eth1 eepro100 #alias eth2 eepro100 alias eth0 e100 alias eth1 e100 alias eth2 e100 alias parport_lowlevel parport_pc alias scsi_hostadapter aic7xxx ------------------------------------------------------------------ License ------- Copyright (c) 1999 - 2000, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistribution's of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistribution's in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.