Yes, there are mature IP over FC implementations, BUT it HIGHLY depends on the HBA (FC card) and drivers in most cases. Emulex, Qlogic, LSI and ATTO seem to be the major manufacturers, alas, I have neither ATTO nor LSI cards to test/play with. I'll be happy to test one or more of them out if someone sends one or two my way. ;) Windows 2k, XP, 2K3, (TODO: test with 2K8, XP64, Win 7 32/64) Emulex LP800 and up work with a little configuration, (you may need a config util) although you want to look closely at the speed displayed by the network adapter. it is in MBps NOT Mbps (106 megaBYTES/sec, not megaBITS, and is actually 106.25MBps) QLogic QLA2200 and up work with the correct driver, although the speed display seems to be halved IIRC. QLA2100 adapters do NOT support IP in Windows. Solaris (8 and up): Supposedly supports IP over any HBA with Solaris drivers, but I can only confirm QLA2200. As of Solaris 8, there were specific QLA2xxx patches, haven't checked on Sol 10 yet. Look at the FCIP driver (man page) for more info. Linux: I have absolutely no clue why, but as of 2.6.21 it looks like there is NO kernel support for IP over FC. In fact, every drievr changelog I've skimmed over has bits and pieces of 'removed IP code' stuff. From what I've read so far, there has been an actual effort to remove IP over FC from linux kernel code. I've searched the web, and the kernel source code (grep -R etc) and haven't found support at this point. Emulex claims to offer support with their driver, but changelogs seem to indicate it's been removed. I only have one emulex card and it's in a windows box for now, so I haven't verified this. Qlogic offers drivers with IP support for the 23xx (and up?) cards, BUT they've dropped support in newer versions of the driver (check the changelog in the newest version for "remove IP", also note that the driver will run a 2200 but NOT IP over a 2200) In Linux, the idea seems to be that 'it doesn't make sense to run IP over your storage network' and so it's simply not supported. A rude shock to say the least, considering how flexible the kernel is in general. I guess if you already have a 5km cable run, you should just run another for your IP since 'IP and storage traffic over the same cable doesn't make sense'. Sigh. Dunno about BSD, IRIX has one card by a manufacturer no longer around that supports IP/FC with the OEM driver. No clue whether MacOS of any version supports IP/FC, a quick search doesn't seem to show aything. Ok, as for actually setting it up, that's pretty simple. In Windows, it's just another Local Area Connection, however, you will probably want to set up a static IP since it's pretty danged unlikey you have an FC router with DHCP compatibility. ;> You can use a windows server or a dhcp server program on 'regular' windows if you like. In linux (pre 2.6.??), it's usually fc0 (1,2, etc) and again, configured pretty much the same way as an eth0. Other OS's i don't have enough experience with IP/FC to say for sure. As with GigE. you will REALLY want to set up a MUCH larger MTU. The largest MTU you want to use is 65,280. For file transfers over a good connection, this is nice. For interactive connections (games, chats, internet etc) you'll wnat domething a lot smaller, say around 7936 or so as a starting point. (IPv6 can have variable length headers, so I'm leaving a little extra room in the MTU size for that) Gotchas: You've probably got a 'regular' PCI slot. 32 bit, 33MHz, which is 133MBytes per second. A 1Gbps (entry level basically) FC card can push 106Mbytes per second. This leaves only 27Mbytes/sec for EVERYTHING else on that PCI bus. The most commonly used PCI device is...your hard drive. So, one of the two is going to be slowed down. I simplify a bit, but them's the facts. ;> Of course, enough PCI-Express lanes will take care of all this handily, but you will need enough motherboard/cpu/ram to support the speeds. One day soon I hope to try out an 'enthusiast class' motherboard and 2Gb or even 4Gb FC PCI-Express controllers. (Where DID I put that lottery ticket?) Ran across a slightly older qlogic forum posting about IP over FC and SAN, so, tried a couple of things out, pulled together some research, added a femto-percent of load to googles' servers and: UDPATE 07/07/2009: Since I seem to have the high ranking in Google for IP over FC I guess I'm the 'ranking' expert. Sigh. I'm not truly an FC guru by any means, but I can at least answer this threads main questions. Quick info: FC is really nice for SAN's for several reasons. Separated networks. Your streaming super-3d CAD app over FC won't affect Joe down the hall printing to an ethernet network printer. Much more mature (thus, theoretically stable and/or reliable) support for failover and multipathing, and so on, although this is not confined to IP but is a function of FC traditionally being 'Enterprise Class' FC connection handling vs ethernet 'switching' or collision detection schemes. If you have multiple clients hitting an GigE switch at speed, you quickly learn that most do NOT allow full speed to all ports at one time. I've seen several 'business class' 24 port Gbit ethernet switches staurate with 4 clients pumping data, which kind of leaves the other 20 clients out in the ether. FC (again, since it's 'enterprise class') hardware tends to push the limits a lot harder, there IS no congestion. FC does not have congestion issues, has gauranteed data delivery (with notifiation on the FC level of lost 'packets/frames', and uses 'credit based' flow control thus preventing any one (or more) devices from hogging the bandwidth. Heck, ethernet has to give up 20_BYTES_ per frame by spec (look up interframe gap and preamble if you're skeptical) and that 20 bytes per frame is sucked off of your Gigabit per second speed. Cheaper per Gbit/sec. GigE is pretty cheap, but if you need 2Gbit bandwidth, you have to trunk. That means your switch has to handle trunking correctly AND not saturate the switch itself. As for 10Gb, well, for the price of a 10Gbit ethernet card, you can get a dual channel 4GB FC card. Now figure in the interconnect (switch/hub/crossover) and then compare prices. (heck, i t is possible to have 127 FC devices in a loop without any switch, hub or other interconnect... not good for reliability, but if your 10GigE ethernet switch goes down in the middle of the night, you're stuck till a replacement is available, with FC you could, as a last resort, 'wire around' your FC switch) Now add in the MTU size. Ok, it gets a wee bit interesting here, but you can have a TCP/IP (maybe other protocols???) MTU of 64K that gets mapped into 32 ~2k FC frames (aka a 'sequence'). FC actually allows a data payload to be mapped over 64 FC frames, so ~128K, but TCP/IP v4 itself can only handle 64k size. You'd need to move to TCP/IP v6 and have a correct implementation of TCP/IP jumbograms to be able to completely utilize the 128K max 'MTU' on FC. What does this mean in the real world? It means your maximum effective TCP/IP MTU is 64K rather than GigE's ~9k. Larger MTU = less overhead = more actual data transferred per packet. In other words, it's faster, and it's less load on the server(s) since they process 64Kbytes per interrupt instead of 9. Not only that, since the FC layer is taking care of MTU to Frame conversion, a packet that needs 2k only uses 2k on the wire, while still allowing 64k packets to flow. Ethernet is fixed size. With 9k ethernet frames, sending a Ctrl-C to your server takes...9K. :) Interactive sessions on a congested GigE jumbo-frame network can remind you of 10Mbit ethernet. Did I mention that FC has much lower latencies then GigE or 10GigE? Very short drawbacks to FC: Fewer 'experienced' techs, hard to explain to PHB's that an 8Gig FC will (very probably) be faster than 10Gig Ethernet, and...you will probably need a server to 'translate' from ethernet clients to FC Storage (although, search FC over Ethernet sometime) Such server will need a bit of RAM to buffer everything, and enough I/O speed available to keep up with all the gig and multiple gig connections. Well, that's my (not-so)quick and dirty overview. My original research and testing (including how to actually set up IP over FC on a couple of OS's) is here: http://noteablecomputers.com/ipfc.txt Of course, I'll be adding the content of this post in to that page soon. END UPDATE 07/07/2009 UPDATE 2 07/07/2009 Note that 'NonBlocking' GigE switches are a lot more common now, but you better be 100% sure the switch you get IS actually nonblocking on all ports simultaneously, I've actually seen cheesy switches advertise nonblocking only to discover in the manual that nonblocking is limited to a certain number of ports at a time. Amazing how you can define anything to mean nothing, no? As for the 'translating server', yes, you will want multiple GigE conenctions to take advantage of the 2x or 4x or even 8x Gigabit FC speeds. Trunking doesn't really help you there, your client computers are probably not up to keeping up with dual or quad trunked connections to the server. Although I'm not 100% absolutely sure, I'd venture a strong guess that you'd want multiple IP's along with GigE ports on the ethernet side of the server to fully utilize the FC side. Oh, and please do remember that (ethernet) VLANs up the overhead on ethernet by 4 bytes, thus losing you more effective speed, even though the actual packet length is extended. Of course, larger packets on ethernet mean more chance of congestion, but hey, what can ya do? END UPDATE 2 07/07/2009 Feel free to email me if you want more detail or if you think I might be able to help you out a bit, I'm not an expert, but I have learned a bit. I'm also very interested in new or corrected information. I do have a nice little 2GBit FC SAN at home here, alas, my poor old 9G & 18G FC drives suck so much power to run _and_ cool, I can only justify running the arrays in the winter. :) Thanks paul at note able computers dot com