Storage teams often get blamed for problems that are actually network problems in disguise — a “slow SAN” that turns out to be a duplex mismatch, an “array outage” that turns out to be a flapping switch port. Over the years I have learned to treat network troubleshooting as a core storage administration skill, not something to just hand off to the network team and wait. This article walks through the most common networking problems that affect storage connectivity (iSCSI, NAS, and general IP-based storage traffic) and how to systematically diagnose them.
The Layered Troubleshooting Approach
Just like with Fibre Channel, I work through networking issues in layers, bottom-up:
1. Physical layer -> cabling, link lights, port status
2. Data link layer -> VLANs, MAC address tables, spanning tree
3. Network layer -> IP addressing, routing, subnetting
4. Transport layer -> TCP behavior, ports, MTU
5. Application layer -> iSCSI/NFS/SMB protocol-specific issues
Problem 1: Physical Connectivity and Link Issues
Symptoms: Interface down, intermittent connectivity, unexpected link speed.
# Linux - check interface status and negotiated speed/duplex
ethtool eth0
# Look specifically for:
# Link detected: yes/no
# Speed: 10000Mb/s
# Duplex: Full
# Cisco switch - check interface status
show interface GigabitEthernet0/1 status
show interface GigabitEthernet0/1 counters errors
Duplex mismatch is a classic, still-common cause of severe performance degradation (not total failure) — one side set to full duplex, the other to half, or one side on auto-negotiate paired with a hard-coded far end. This produces late collisions and CRC errors that manifest as bizarrely slow throughput without a link ever actually going down.
Problem 2: VLAN Misconfiguration
Symptoms: Host can ping the local gateway but not the storage array, or vice versa; intermittent connectivity that depends on which switch port a device is connected to.
# Cisco switch - verify VLAN assignment on a port
show interface GigabitEthernet0/5 switchport
# Verify VLAN exists and is active
show vlan brief
Checklist:
- Is the storage array’s port and the host’s port in the same VLAN (for a Layer 2 adjacent design), or is routing correctly configured between VLANs?
- For iSCSI environments using dedicated storage VLANs, is the host’s iSCSI initiator interface actually tagged/untagged correctly to match the switch port configuration (trunk vs. access port)?
- Is the VLAN allowed across all trunk links between the host’s switch and the array’s switch?
Problem 3: MTU / Jumbo Frame Mismatches
This is one of the most common — and most misdiagnosed — problems in iSCSI and NFS environments. Jumbo frames (MTU 9000) are commonly recommended for storage traffic to reduce CPU overhead and improve throughput, but a mismatch anywhere along the path causes fragmentation or silent packet drops.
# Linux - test MTU end to end with a specific packet size, no fragmentation allowed
ping -M do -s 8972 10.10.20.50
# 8972 + 28 bytes (ICMP/IP header) = 9000 total
# If this fails but a smaller size succeeds, MTU mismatch confirmed somewhere in the path
ping -M do -s 1472 10.10.20.50
# 1472 + 28 = 1500 (standard MTU) - if THIS also fails, it's not an MTU issue
Checklist for MTU consistency:
- Host NIC/vSwitch MTU setting
- Every physical switch port and trunk/uplink along the path
- Storage array’s network interface MTU
- Any router/L3 interface if traffic crosses subnets
A single device in the path left at the default 1500 MTU while everything else is set to 9000 causes silent performance problems or connection resets, and is a very common oversight during initial iSCSI SAN builds.
Problem 4: IP Addressing and Routing Issues
Symptoms: Cannot establish an iSCSI session, NFS mount hangs, intermittent connectivity between subnets.
# Basic connectivity test
ping 10.10.20.50
# Trace the path to identify where connectivity breaks
traceroute 10.10.20.50
# Verify routing table on the host
ip route show
For iSCSI specifically, a very common architecture mistake is running initiator and target traffic across a routed network without properly validating that both multipath sessions actually take physically diverse paths — if both “redundant” iSCSI sessions route through the same underlying switch or router, the redundancy is illusory.
Problem 5: TCP Performance Issues (Windowing, Retransmits)
Symptoms: Throughput far below expected line rate despite no obvious errors; performance that degrades over long-distance/high-latency links (like async replication over WAN).
# Capture and inspect TCP retransmission behavior
tcpdump -i eth0 -w capture.pcap host 10.10.20.50
# Analyze in Wireshark, filter: tcp.analysis.retransmission
# Check current TCP window scaling settings
sysctl net.ipv4.tcp_window_scaling
sysctl net.core.rmem_max
sysctl net.core.wmem_max
For high-latency, high-bandwidth links (common in replication between data centers), the bandwidth-delay product (BDP) determines the TCP window size needed to fully utilize available bandwidth:
BDP = Bandwidth x Round-Trip-Time
Example: 1 Gbps link, 40ms RTT
BDP = (1,000,000,000 bits/sec / 8) x 0.040 sec
= 125,000,000 bytes/sec x 0.040 sec
= 5,000,000 bytes ≈ 4.77 MB
If the TCP window is only set to the OS default (often 64KB-256KB),
throughput will be capped far below the 1Gbps link's actual capacity,
regardless of how clean the link is.
This calculation is the reason WAN-optimized replication appliances and tuned TCP stacks (window scaling enabled, larger buffer sizes) matter so much for async replication over distance — without proper window sizing, you will never achieve the link’s rated throughput no matter how good the physical circuit is.
Problem 6: Spanning Tree Issues Affecting Storage Network Stability
Symptoms: Brief, periodic connectivity drops, especially after adding new switches or making topology changes.
Spanning Tree Protocol (STP) topology changes can cause brief interruptions as the network recalculates loop-free paths. In storage networks, even sub-second interruptions can cause iSCSI session drops or NFS timeouts.
# Verify STP status and check for recent topology changes
show spanning-tree vlan 20
show spanning-tree detail | include Topology Change
# Enable PortFast on host-facing (non-switch-to-switch) ports to avoid
# unnecessary STP recalculation delay on those edge ports
interface GigabitEthernet0/5
spanning-tree portfast
Problem 7: DNS and Name Resolution Issues (NAS/SMB specific)
Symptoms: SMB shares intermittently inaccessible by hostname but accessible by IP; Active Directory-integrated NAS authentication failures.
# Verify forward and reverse DNS resolution
nslookup nasfiler01.corp.local
nslookup 10.10.30.20
# Both should resolve consistently and match; mismatched forward/reverse
# DNS is a very common cause of Kerberos authentication failures against
# AD-integrated NAS shares
Diagnostic Tool Reference Table
| Tool | Purpose | Layer |
|---|---|---|
ethtool | Interface speed/duplex/link status | Physical |
ping -M do -s | MTU path testing | Network |
traceroute/tracert | Path and routing verification | Network |
tcpdump/Wireshark | Packet-level capture and analysis | Transport/Application |
netstat/ss | Active connections, socket states | Transport |
iperf3 | Raw throughput testing between two endpoints | Transport |
nslookup/dig | DNS resolution verification | Application |
Using iperf3 to Isolate Storage vs. Network Performance Issues
A technique I use constantly: before blaming storage array performance, test raw network throughput between the host and the array’s network segment using iperf3, which removes the storage protocol entirely from the equation.
# On the array-side network segment (or a proxy host on the same subnet)
iperf3 -s
# On the host
iperf3 -c 10.10.20.50 -t 30 -P 4
# If iperf3 throughput is far below expected, the problem is network-layer,
# not storage-layer, and you have just saved yourself hours of chasing
# the wrong team
Real-World Enterprise Scenario
A common scenario: after a data center network refresh, an NFS-backed VMware datastore starts reporting intermittent latency spikes and occasional APD (All Paths Down) events. The methodical approach:
- Check physical layer — link status clean, no errors
- Check MTU consistency — found: new spine switches were provisioned with default 1500 MTU while the old switches and NAS heads were configured for 9000
- Result: jumbo frames were being silently fragmented or dropped at the new switches, causing retransmissions and latency spikes under load
- Fix: standardize MTU 9000 across the entire path, verify with
ping -M do -s 8972 - Confirm resolution with sustained iperf3 throughput test matching expected line rate
This exact class of issue — a single device in the path with a mismatched MTU after a partial hardware refresh — is one of the most common real-world storage networking problems I have personally diagnosed.
Preventive Practices That Reduce Troubleshooting Load
A good chunk of the network troubleshooting I have done over the years could have been avoided with a bit of upfront discipline. A few practices that consistently pay off:
- Document expected MTU, VLAN, and speed/duplex settings for every storage network segment, and periodically audit actual configuration against that documentation — configuration drift after unrelated network changes is one of the most common causes of “it used to work” tickets.
- Baseline normal throughput and latency with
iperf3and switch counters during a known-healthy period, so you have something concrete to compare against during an incident instead of guessing what “normal” looks like. - Validate true physical path diversity for redundant links during initial design, not just logical redundancy — trace actual cable runs and switch chassis to confirm a single point of failure does not exist.
- Include network validation in every storage change window, even changes that seem storage-only, since array firmware updates and reconfigurations can sometimes reset network-facing settings (like MTU or LACP configuration) unexpectedly.
- Keep a change log correlated with performance baselines, so that when a performance regression appears, you can quickly check whether it lines up with a recent network or storage change.
Coordinating With the Network Team
Storage and network teams sometimes operate in silos, which slows down troubleshooting exactly when speed matters most. A few coordination habits that help:
- Agree in advance on a shared set of diagnostic commands and expected outputs for storage network segments, so both teams are looking at the same data during a joint incident call
- Establish a fast escalation path directly to network engineering for storage-adjacent network issues, rather than routing every ticket through a general help desk queue
- Include the network team in storage architecture reviews, especially for MTU, VLAN, and redundancy design decisions, since decisions made storage-side (like enabling jumbo frames) have direct network-side configuration implications
Common Mistakes
- Blaming the storage array first without ruling out the network path with basic tools like
ping,traceroute, andiperf3. - Assuming jumbo frames are configured everywhere just because they were configured on the array and one switch — always verify end to end.
- Not validating true path diversity for multipathed iSCSI, ending up with “redundant” paths that share a single point of failure.
- Ignoring TCP window/buffer tuning for long-distance replication links, then wrongly concluding the circuit itself is underperforming.
- Skipping DNS/reverse-DNS validation when troubleshooting AD-integrated NAS access issues, which is a very common but easily overlooked root cause.
FAQs
Q: How do I tell if a storage performance issue is network-related or array-related? Run an iperf3 throughput test on the same physical path, bypassing the storage protocol entirely. If raw network throughput is already below expectations, the problem is upstream of the array.
Q: Do I need jumbo frames for iSCSI/NFS to work at all? No — standard 1500 MTU works fine functionally. Jumbo frames (9000 MTU) primarily reduce CPU overhead and modestly improve throughput; the risk is entirely in inconsistent configuration across the path, not in using jumbo frames themselves.
Q: What is the most common root cause of “SAN is slow” tickets that turn out to be network issues? In my experience: MTU mismatches and duplex mismatches are the two most common, followed by inadequate TCP window sizing on long-distance replication links.
Summary
Most storage-adjacent networking problems fall into a fairly small, well-known set of categories: physical layer issues, VLAN misconfiguration, MTU mismatches, routing/addressing problems, and TCP tuning gaps on high-latency links. Working through the OSI layers methodically, and using simple tools like ping -M do -s, traceroute, and iperf3 to isolate network performance from storage array performance, will resolve the overwhelming majority of “mystery” storage connectivity and performance issues far faster than guessing.
References
- Cisco — Nexus and Catalyst switch troubleshooting guides, cisco.com
- VMware — Networking and iSCSI Storage documentation, docs.vmware.com
- Microsoft — SMB troubleshooting and DNS/Kerberos guides, learn.microsoft.com
- SNIA — iSCSI and networked storage technical documents, snia.org
- Red Hat — Network performance tuning guide, access.redhat.com/documentation
