×
Create a new article
Write your page title here:
We currently have 3,189 articles on s23. Type your article name above or create one of the articles listed here!



    s23
    3,189Articles

    nxge = nVidia Gigabit Ethernet

    Numbering[edit]

    Interfaces are numbered 0,1,2,3 with 3 being nearest the PCI slot.

    Driver[edit]

    In Solaris 10 U4 and above apparently, otherwise here or from http://www.sun.com/download/products.xml?id=45a593ce.

    On U3 or below you must update to Jumbo Kernel Patch 118833-36 or later, reboot immediately following the patch and before applying any more patches.

    For SPARC systems, change to the following directory:

    cd 10_GigabitEthernet/Solaris_10/sparc/Packages
    

    For sun4v systems, install the software packages by typing the following at the command line:

    /usr/sbin/pkgadd -d . SUNWnxge.v SUNWnxgem
    

    For sun4u systems, install the software packages by typing the following at the command line:

    /usr/sbin/pkgadd -d . SUNWnxge.u SUNWnxgem
    

    Docs[edit]

    http://docs.sun.com/source/820-0115-10/ (for Sun x8 Express Quad Gigabit Ethernet UTP Low Profile Adapter)

    Configuration[edit]

    Use /kernel/drv/nxge.conf. Here's an example

    #
    #---------------Link Configuration ----------------------
    #       The link parameters depend on the type of the card
    #       and the port.
    #       10 gigabit related parameters ( i.e adv_10gfdx_cap)
    #       apply only to 10gigabit ports.
    #       Half duplex is not supported on any NIU card.
    #
    #       adv-autoneg-cap
    #               Advertise auto-negotiation capability.
    #               default is 1
    # adv-autoneg-cap = 1;
    #
    #       adv_10gfdx_cap
    #               Advertise 10gbps Full duplex  capability.
    #               default is 1
    # adv_10gfdx_cap = 1;
    #
    #       adv_1000fdx_cap
    #               Advertise 1gbps Full duplex  capability.
    #               default is 1
    # adv_1000fdx_cap = 1;
    #
    #       adv_100fdx_cap
    #               Advertise 100mbps Full duplex  capability.
    #               default is 1
    # adv_100fdx_cap = 1;
    #
    #       adv_10fdx_cap
    #               Advertise 10mbps Full duplex  capability.
    #               default is 1
    # adv_10fdx_cap = 1;
    #
    #       adv_asmpause_cap
    #               Advertise Asymmetric pause capability.
    #               default is 0
    # adv_asmpause_cap = 0;
    #
    #       adv_pause_cap
    #               Advertise pause capability.
    #               default is 1
    # adv_pause_cap = 1;
    #
    #
    #------- Jumbo frame support ---------------------------------
    # To enable jumbo support for all nxge interfaces,
    # accept_jumbo = 1;
    #
    # To disable jumbo support for all nxge interfaces,
    # accept_jumbo = 0;
    #
    # Default is 0.  See the example at the end of this file for
    # enabling or disabling jumbo for a particular nxge interface.
    #
    #
    #------- Receive DMA Configuration ----------------------------
    #
    #  rxdma-intr-time
    #       Interrupts after this number of NIU hardware ticks have
    #       elapsed since the last packet was received.
    #       A value of zero means no time blanking (Default = 8).
    #
    # rxdma-intr-pkts
    #       Interrupt after this number of packets have arrived since
    #       the last packet was serviced. A value of zero indicates
    #       no packet blanking (Default = 20).
    #
    # Default Interrupt Blanking parameters.
    #
    # rxdma-intr-time = 8;
    # rxdma-intr-pkts = 20;
    #
    #
    #------- Classification and Load Distribution Configuration ------
    #
    # class-opt-****-***
    #       These variables define how each IP class is configured.
    #       Configuration options range from whether TCAM lookup ie
    #       is enabled to flow hash generation.
    #       This parameters also control how the flow template is
    #        constructed and how packet is distributed within RDC
    #       groups.
    #
    #       supported classes:
    #       class-opt-ipv4-tcp class-opt-ipv4-udp class-opt-ipv4-sctp
    #       class-opt-ipv4-ah class-opt-ipv6-tcp class-opt-ipv6-udp
    #       class-opt-ipv6-sctp class-opt-ipv6-ah
    #
    #       Configuration bits (The following bits will be decoded
    #       by the driver as hex format).
    #
    #       0010:           use MAC Port (for flow key)
    #       0020:           use L2DA (for flow key)
    #       0040:           use VLAN (for flow key)
    #       0080:           use proto (for flow key)
    #       0100:           use IP src addr (for flow key)
    #       0200:           use IP dest addr (for flow key)
    #       0400:           use Src Port (for flow key)
    #       0800:           use Dest Port (for flow key)
    #
    # class-opt-ipv4-tcp = fe0;
    #
    

    See the linked documentation for details on how to specify individual port details.

    You can view current interface speeds like this:

    root@host # dladm show-dev
    e1000g0         link: down      speed: 0     Mbps       duplex: half
    e1000g1         link: down      speed: 0     Mbps       duplex: half
    e1000g2         link: unknown   speed: 0     Mbps       duplex: unknown
    e1000g3         link: up        speed: 100   Mbps       duplex: full
    nxge0           link: down      speed: 0     Mbps       duplex: unknown
    nxge1           link: up        speed: 1000  Mbps       duplex: full
    nxge2           link: unknown   speed: 0     Mbps       duplex: unknown
    nxge3           link: unknown   speed: 0     Mbps       duplex: unknown
    


    Performance[edit]

    Add this line to /etc/system (use '8' on UltraSparc-T1 systems)

    • set ddi_msix_alloc_limit=4

    Add this line to a ndd start up script somewhere (set to 16 on UltraSparc-T1 systems)

    • ndd set ddi_msix_alloc_limit=4

    Firmware[edit]

    Updating the firmware on these cards is one scary-arse piece of work!

    You get the firmware and driver pack from http://www.sun.com/download/products.xml?id=45a593ce (this is the same line as above on this page). You need to unzip the HTML instructions from 10_GigabitEthernet/Firmware and copy them somewhere you can point a browser at (here). Follow the instructions VERY carefully!

    I've written a function to generate the firmware commands from the OS of a server as it's easy to get things wrong, escpeially if you have two or more nxge cards. However, I strongly recommend you verify it's output with what you'd expect before proceeding as I don't want to be held responsible for shafting an expensive server!

    nxge_fw_upgrade_cmds() {
     echo load-base dload NXGE_OBP.FC
     echo load-base 1 byte-load
     echo load net:,QGCEM3.12
     for i in 0 4 8 12 16; do
      test -e /dev/nxge$i || continue
      ifconfig nxge$i|nawk '/ether/{print $2,"set-mac-addr"}'|sed 's/:/ /g'
      echo "\" `ls -l /dev/nxge$i|awk '{print $NF}'|\
      sed 's|../devices||;s/:.*//'`\" begin-select-dev"
      echo load-base update
     done
     echo reset-all
    }
    
    Cookies help us deliver our services. By using our services, you agree to our use of cookies.
    Cookies help us deliver our services. By using our services, you agree to our use of cookies.