Experimenting with network topology and throughput.

[agent_kith]

I am running a new router in my home now. A Celeron 3865U mini PC with 6 ports. I'm not sure if a Celeron is capable of switching gigabit speeds, so tested it out with the iPerf benchmark tool.

So what iperf is doing is really testing the maximum transfer speed (upload or download) of your network. You need a computer running iPerf in server mode (in my case the router), and then run clients at various computers to test the speed.

This is the results of running iPerf in one of the development machines.

Code:

Client connecting to 10.x.x.x, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 10.x.x.x port 42884 connected with 10.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.15 GBytes   987 Mbits/sec

987 Mbits, not too bad. Good to know I'm wrong. Celeron is capable of maxing out a gigabit network.

Next, I try this from my NAS.

Code:

[  3] local 10.x.x.x port 33607 connected with 10.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1014 MBytes   851 Mbits/sec

Not as quick, but still acceptable. Kind of expected because this NAS is running on HP N54L, and I believe it's using Marvel  NIC. This network card is slower than Intel (and consume more power).

And this is the throughput from my primary music player (running Snakeoil of course):

Code:

[  3] local 10.x.x.x port 58196 connected with 10.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   625 MBytes   524 Mbits/sec

524 Mbit/s. This is expected because I am using a PCI network card, PCI maxes out at 533 Mbit/s. So I'm already very close to the limit. Besides USB is 480 Mbits/s so this is OK.

Finally, I run this on the Raspberry Pi 3... And what an appaling result!!

Code:

[  3] local 10.x.x.x port 35116 connected with 10.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   338 MBytes   284 Mbits/sec

284 Mbits/s! Did I got that right?

A quick google couldn't tell me anything conclusive. But it does appear network is a bottle neck with these Pis. As good as they are, there are restrictions when it's a SoC.

USB 2.0 max out at 480 Mbits/s  anyway, so I reckon even at this speed the Pi should be able to handle high res or DSD relatively well. But there's not much room for error here.

ODroids, Banana Pis can sustain higher throughputs. They are also more expensive.

Anyway, why the new router? I am finally going to bite the bullet and seggerate everything in my network into it's own VLAN. Not entirely sure what that'd do, so watch this space.

[agent_kith]

So before we begin, let's introduce the hardware:

This is the new router:
   
Purchased off Aliexpress. It took about a month to deliver to Perth.

Here's the unit, plus:

• 2x 8GB of 2133 MHz RAM puchased locally
  
• Reused an old 500 GB HDD I have lying around somewhere
  

   

Here's a look of the insides. This is the base of the unit. Behind this board sits the CPU, and that's attached to the top of the case, where all the heat sink fins are. The picture doesn't give you a scale of how small and compact this unit is. The machining is done to millimeter perfection! There is no gap for me to pry the bottom cover off. I have to remove one of the sides so I can remove the top.
   

And here it is with RAM and HDD fitted. This should give you a scale of how compact this unit unit is.
   

Last but not least, my network is now broken up into:
   

So what is a VLAN? It stands for Virtual Local Area Network. In a network switch, all computers have the capability to talk to any other computer on this same switch. A virtual LAN is like inception, it creates another switch inside this switch (hence the word virtual), and computers can only talk to another computer if they are on the same virtual switch.

So in the above example, computers in "DEV" can only see other computers in "DEV". If there is a need to talk to computers outside the virtual switch, they need to go through the router.

In this setup, I have broken down the network communications logically, this in turn prevents a lot of unnecessary suprious network traffic on the network. The theory is, the overheads of VLAN will outweigh the constant traffic noise that is bombaring all the ports on the network. 

So if I design this right, the "Music" VLAN will be where my music server and players are. It can still communicate with any computer on the other VLAN, but it will no longer see the unnecessary broadcast messages (This is the noise I am referring to).

For this to work, the router has to be extrememly fast, and be capable of receiving and forwardinig packets at a moment's notice. Hence the choice of the Minisys router. It has 6 network ports. 1 is connected to the Internet, 4 goes back to the switch (This is called LAGG).

Don't worry if you don't understand any of the above.

The unit is powered off a 12 V SMPS. Not ideal for sound quality, so it'd be something I'd look at addressing in the future.

For folks who are interested in specs, here's the router specifications:
   

There are faster versions - fanless i5 and even an i7 in a different case. But a router is designed to run 24x7. Figures a Celeron will be better for the environment.

My only worry was it is incapable of switching at the maximum speed. For now that seems to be OK. I doubt the Celeron can do full 6 ports at 1 gbps, but that is not a requirement for me.

My quick tests have showed me, not all network chips are built the same. Some chips are just faster than the others. The fastest chips so far are still made from Intel.

I don't  use any audiophile network cables in my setup at all. But in the future if I get a chance to try them out, will be interested to see the iPerf results .. My guess is they'd be the same in terms of throughput.

[agent_kith]

Been busy lately so havn't had the time to sit down for a proper audition.. But did managed to sneak in a couple hours last night and on Sunday...

I havn't totally isolated the NAS with my CAP yet, but removed enough of the other machines off to their own VLANs and thought that should make a good difference.

So what's my first impressions? Not so good unfortuantely..



Not entirely sure what is happening, just do not feel that engaged with the music with this new setup now. The music is boring and lifeless...

This is weird because technically the packets are transferred local to the switch, and not to the router. So everything should be the same as before - with the exception of less spurious network traffic and more VLANs..

More VLANs. Maybe I should stick to just 2 VLANs - one for music, and one for everything else, instead of trying to be too clever and seperate out everything. :shrug: D'oh.

Will continue to try different things over the next few days and see how it goes.

[agent_kith]

Not entirely sure what is happening, just do not feel that engaged with the music with this new setup now. The music is boring and lifeless...

Man, I should really keep this thread up to date! It's been > a year since the last update. For historical reasons, the solution to fix the above "boring" sound is to ground the case of this tiny PC. So all I did is to attach a thin lean wire. Tap one end to this router, and the other end to my development machine (which is grounded). Problem solved.

Second tweak that works is for me to start using Jumbo Frames. See this  blog article when I turned off Jumbo Frames and found the music was perceptally worse. Long story short the network card I was using on my NAS is causing random reboots. This isn't completely fixed yet, I'll need to buy a new NIC to fix this permanently.

Anyhoo the routing software running on this tiny PC is something called pfSense. Over the year I have experimented with various ways of connecting things up (see here for a quick review). And right now I've finalised on the L3 topology. For a long time this setup is running fine. Then I stupidly tried to change a setting on the router. I can't remember what that was but it brought down the whole network. Luckily pfSense has this great feature - undo configuration change. So all I need to do is to roll-back to a previous configuration.
 

Or so I thought! Everything seems to work, but for some stupid reason Jumbo frames no longer cross VLANs (sometimes it works, and sometimes it doesn't).... Honestly NFI what the problem could be. I am pretty sure if I factory reset the settings back to default, and re-apply the configuration everything will work as before. But I really shouldn't need to do that! All these little problems I constantly get with pfSense, this is the last straw!

So come next year, I'm gonna ditch pfSense and go with Untangle. Good thing is all I need to do is to swap out the HDD to install Untangle. And I can always swap the old pfSense HDD back to get the old system back.

So many things to do, so little time .

[Bromf]

So many things to do, so little time

+1 !


Hope you have a wonderful and relaxing break in Honkers and Taiwan this festive season... you deserve it !!

[agent_kith]

Here's a second experiment. What happens if I want to send something with a payload that's greater than 9000 bytes, but the receiving party cannot accept this number? For example with Raspberry Pis, the maximum allowable MTU is 1500.

Now the router will be smart enough to break it up into smaller packets and send it to the receiving party. The split up packets will be marked so that it can be re-assembled into it's original packets.

Like so, I am trying to send a payload of 9001 bytes to a machine with a MTU of 9000. If I have the "do not fragment" bit set, this original 9001 bytes cannot be split up and must be sent as is, or be rejected. But if I don't have this bit set, the payload can be split into 2. In Microsoft Windows, the "do not fragment" flag is -f. So with this on, trying to send 9001 bytes will fail, and succeed if off. Exactly as designed when I'm using Untangle as the router:
 

Frustratingly, I ran the same experiments before in pfSense, and it worked! Now this behaviour is intermittent and I have ny idea why. One thing that bugs me most is the use of "double negatives" in pfSense configuration. Turning something on (tick in a checkbox) means I'm actually turning something off!

There are features in pfSense that I will miss (HAProxy and ACME). But thanks to the generous support by you guys, I now have a Ryzen server with more grunt and RAM. I can now now create a new virtual machine to run HAProxy and ACME. In fact, I can even run the Untangle firewall as a virtual machine, keeping things even simpler!

I don't expect a lot of people to push their music setup to this extent. But if you are interested, remember to stay tune to the blogs to find out how I do it.. It'll also serve as a written record for me to refer back when things are broken again.

[agent_kith]

Typically, now that I have my home network finally singing sweet I'm made aware of other products. This time it's Mikrotik. And boy do they make sure interesting products, specs wise. Have a look of 'em here: Link. I mean Layer 3 switch, for $209!

Granted I'm guessing given the CPU it's running on, if I turn on L3 capability, the speed is probably not going to be great. But hey now that I'm running my router as a virtual machine, it's using a very fast CPU with very fast RAM. This VM router is going to beat most hardware L3 switches I think (when I turn off inspection and IDS).

Passive 8 SFP+ switch, for $269!

I'm seriously considering ditching my Unifi stack and go with Mikrotik. Sure the Unifi UI is impressive, but it wears out really quickly. there are just some features that is sorely lacking in Unifi, that I doubt they'll ever deliver as promised. No L3 routing, only 1 mode of LACP, thereby killing my ability to fully trunk connections to max out speed.

Hey MikroTik, if you managed to see this somehow and want to sponsor this project with some hardware, give me a shout out! :)

[shaitan667]

Interesting read! I work in IT and am always messing with the network at home.
My current setup is a bit rubbish -
ISP (Belong) supplied Sagemcom NBN(FTTN) in bridged mode to a Draytek 2830vn Router doing the actual work. Apple Time Capsule 2tb (now minus the hard drive - turns out it still works without the drive) is the wireless AP, all routing disabled.
Then a Netgear DGN3500 running OpenWRT is configured as a wireless bridge - the Snakeoil Server is connected to this via ethernet in the stereo cabinet.

I've been really strongly considering building a router. Either from second hand PC equipment (my preferred option for home servers and the like), or going the route you did with a box from China.

The last decent device I had was a Fortigate 100d (got it for free from a past employer) and loved the interface.

How has your overall experience with pfSense been? I've looking into Opnsense as well - think I'm leaning towards that over pfSense.

Alternatively, there are a heap of wireless routers on AliExpress that support OpenWRT that look half decent.

[agent_kith]

How has your overall experience with pfSense been? I've looking into Opnsense as well - think I'm leaning towards that over pfSense.

OPNsense is a fork of pfSense i believe. And from what I gather, is the less dodgy version (i.e. more open source). I havn't tried OPNsense myself, so can only speak for pfSense. With the latter, I have been using it for a long time. And it was good, it just works, and it works well. Routing speed is good, as I only have suricata working on the WAN side.

My only complaints is somethings things break, and when it breaks, restoring configuration doesn't quite work. You have to factory reset, and then restart from the beginning.

If you are interested in IPv6 (esp IPv6-PD), as far as I can tell they have the full stack working throughout. To this day not many can claim that. The current FW I'm using (Meraki and Untangle) don't support the correct IPv6 implementation, or at all.

Alternatively, there are a heap of wireless routers on AliExpress that support OpenWRT that look half decent.

For the most part, the switch is more important. You have to make sure this switches at wire speed, i.e. with Jumbo frames set to 9000, you need to be able to pass through around 987 Mbps or better. I've only tried four ports (not all of them). Routing wise is unnecessary in this context. Having said that, I have tried those tiny Mikrotik routers, while not fast, they look pretty feature rich. Give them a shot and let me know how it goes.

[agent_kith]

Why did I drop pfSense and go with Untangle? Well, the main reason really is because I have broken jumbo frames routing on pfSense and couldn't get this feature back working again.

Second reason, it gives me more insight into my home network that I couldn't really get before in pfSense.

Sure pfSense has a app called ntopng that can do a subset of what Untangle can do, but it's not integrated like Untangle is. The Untangle offers more features, and is more powerful.

As an example, looking at the Untangle Dashboard I noticed the majority of traffic that crossed my network is an application called NFS. For the uninitiated, NFS is a file sharing protocol (allows me to share files over the network).

This chart breakdown tells me NFS took up >80% of the network traffic over the past 24 hours. This is not good if this NFS comes out from network interface of my NAS (Media). I need to keep this network interface as quiet as possible because this is the network my music PC is also on.

It's important that the NAS do not talk to my music PC and other machines at the same time. The reason is because I'm using Jumbo Frames.
 

 
Hovering the mouse over that green area above, it tells me that is 25 GB of NFS data. This graph is a plot of the past 24 hours, so that's > 1 GB an hour!

On the same dashboard is a widget that shows me the traffic that enters/leaves the network. This is just an example as the problem is already fixed after I took these screen caps. Looking at the widget then, I can confirm the NFS is indeed coming out from my NAS VLAN, and into the Development VLAN (where all my Snakeoil development machines sits on).
 

Knowing this, I now go into the 'Sessions' section, and list all the network connections on the DEV VLAN (You can also just list the connections on the Media VLAN). From the list I identified the machine that causes all the NFS network activity (again this is just an example, not the actual screen cap of the problem):
 

The problem machine is responsible for storing all the source code of Snakeoil. In here I also know the development machine is constantly transferring files in and out of the NAS at a rate of 250 - 500 Kb/s in and out). So the time it took me to notice there's an issue and isolate the problem took less than 30 seconds.

Here's the breakdown of traffic in the past hour since the problem is fixed, no more NFS traffic hogging the network.
 

So the majority of the traffic is now something called UDP. Extrapolate that hourly data to 24 hours means it'll be around 4 GB of data per 24 hours. The traffic is significantly less than before. And looking at the network layout widget, there's way less data entering and exiting the media interface now.

The words above in bold is important. As the above is a problem where data crosses VLANs (i.e. coming in and out from different network).

Traffic that's internal inside the Media VLAN will not be shown here, and for all we know, could be as busy as hell. You'll need a different tool to see if the Media VLAN is busy. I've always used the packet capture feature on the Meraki equipment for this. But you can also do something similar on Untangle:
 

So the above output (blanked out) basically tells me the Media VLAN indeed is relatively free from 'unnecessary' traffic.

All is good again. Now will this bring any enhances to audio? At the end of the day, I am now assured my network is running in peak form. That (re)assurance gives me peace of mind, which in turn means I'll be able to enjoy music better. You can't lose!