WAN Bonding with ROOter and OpenMPTCProuter

How To Tutorials related to Routers and Firmware
Post Reply
User avatar
bnhf2
Posts: 40
Joined: Sun Aug 05, 2018 10:03 am
Has thanked: 0
Been thanked: 24 times

WAN Bonding with ROOter and OpenMPTCProuter

Post by bnhf2 » Tue Apr 14, 2020 5:25 pm

Here's another post I did earlier this year on the ROOter forum, which is now a bit difficult to find. I did this post in late January, and I've since moved from the Raspberry Pi 4B to a Qotom x86 (i3) industrial PC with 4x gigabit Ethernet ports, 2x USB3 and 2x USB2 ports. OpenMPTCProuter needs an Internet appliance with some horsepower, but it has proved amazing for true WAN bonding -- there's little else like it! With my low end VPS (1vCore, 1GB RAM, 25GB vSSD), I'm frequently topping out at about 235Mbps on the D/L. I'll bring a 2vCore VPS online sometime soon.

I'm happy to say after a number of failed attempts at getting WAN bonding working properly — I've finally got it functioning the way I want. Like many other projects, it's easy to get off track without a little guidance, and OpenMPTCProuter is a bit thin on tutorials. Here's how I did it, just in case there are others that have WAN bonding on their "to do" list.

OpenMPTCProuter can be found here:

https://www.openmptcprouter.com/

Download whatever version will work with that extra bit of kit you have laying around and get it up-and-running. It's OpenWRT based, so this should be familiar territory for everyone here. I used a Raspberry Pi 4 with 4GB of RAM:

https://www.openmptcprouter.com/download

ext4 images are preferred over squashfs, when both are offered.

Before you start configuring your OpenMPTCProuter device, you'll want to get your VPS (Virtual Private Server) running, as WAN bonding always depends on having a server on the other end with a very high speed Internet connection. You can bond up to 8 connections according to the developer, and those need to be reconstituted into a single connection (with a public IP!) on the server end.

The developer has written scripts for specific versions of Debian and Ubuntu, so be sure to use a supported Linux when you setup your VPS. I'm using a Vultr Cloud Computer VPS with 1GB of RAM, a 25GB SSD and a single CPU core. For Linux, I went with Debian 10 x64 for US$5/mo. For that half a sawbuck, you get your very own virtual server and public IP address:

https://www.vultr.com/

Once your server is running, you can SSH (PuTTY works great for Windows users) into it and execute the script as shown in the wiki:

https://github.com/Ysurac/openmptcprout ... te-the-VPS

Once the script is done running be sure to grab /root/openmptcprouter_config.txt as this contains keys and other useful information you'll need when setting up your WAN bonding router. WinSCP is perfect for connecting to your virtual server (before you reboot it!) to grab the file (use SCP protocol just like with ROOter). You can also keep a copy of the PuTTY output from running the script, as the information you need is at the end of the script execution too. Note that the script changes your SSH port to 65222.

Now that your VPS is online, it's time to get your router configured. OpenMPTCProuter uses 192.168.100.1 as an IP address. Set your default admin password just like in ROOter and then proceed to Network – Interfaces to setup WAN1 and WAN2. In my case, I already have both of my cellular WAN devices on a single Ethernet network, each using its own VLAN ID, as part of the "network of gateways" approach I've posted about previously.

https://github.com/Ysurac/openmptcprout ... er-install

I won't cover the specifics of that again here other than to say one of my WAN routers is on VLAN3 with an IP address of 192.168.3.1 and the other is on VLAN4 with an IP address of 192.168.4.1. So when setting up WAN1 and WAN2 in OpenMPTCProuter I used physical "custom interface(s)" of eth1.3 and eth1.4 with static addresses of 192.168.3.30 and 192.168.4.40 respectively. My LAN is on eth0.

https://github.com/Ysurac/openmptcprout ... figuration

After that navigate to System – OpenMPTCProuter – Settings Wizard and enter the IP address of your VPS server, along with "server key" (which auto-loads all the other keys you'll need). From there double-check that WAN1 and WAN2 have the appropriate IP addresses for your downstream ROOter routers. If these routers are on a common switch like mine you'll want to either use VLANs or OpenMPTCP router also supports something called "macvlan" which should do the same thing. If the various WAN routers that you'd like to bond are all physically separate (individual Ethernet or USB cables to your WAN bonding device, then no VLAN or macvlan is needed).

After that go to System – OpenMPTCProuter – Status, and you should see a webpage that looks something like this (drum roll please):
screenshot-192.168.100.1-2020.01.png
If not, the first thing to try is to reboot your WAN bonding router, as there are a number of moving parts here including a VPN.

In my case I'm bonding a Sprint connection using one of my bnhf outdoor directional ROOter setups (RBM33G routerboard, em7565 modem and a pair of directional weBoost wide band antennas cross-polarized) which has been typically giving me about 120Mbps down and 10Mbps up, and an AT&T outdoor omni-directional ROOter setup (RBM33G routerboard, em7565 modem and a pair of weBoost 4G-OTR omni-directional antennas) which has typically been giving me about 40Mbps down and about the same up. The results of the new, bonded, VPN protected connection, complete with a public IP (and port forwarding) are exactly what I had hoped for, with the bonded connection giving me Speedtest.net results of 165Mbps down and 45Mbps up:
screenshot-www.speedtest.net-2020.01.png
screenshot-www.speedtest.net-2020.01.png (45.29 KiB) Viewed 9560 times
In the next week or so, I'll add another cellular provider (Verizon) and a 50Mbps+ hardwire connection I have available at the moment, to see how additional WAN bonding scales on the RPi4.

User avatar
bnhf2
Posts: 40
Joined: Sun Aug 05, 2018 10:03 am
Has thanked: 0
Been thanked: 24 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by bnhf2 » Tue Apr 14, 2020 5:41 pm

And here's the follow-on post I did, about a month later, after moving to the x86 router box:

If you missed my original post on this amazing upgrade to your ROOter setup (assuming you have at least two decent sources of Internet available), you might want to have a look at this post first whrl.pl/RfZ7Tq

What's got me enthused today, is just how fluidly OpenMPTCProuter is working on an x86 platform, combined with a trio of ROOter routers running on outdoor RBM33G setups! One of the ROOter boxes is even linked using a gigabit wireless bridge, rather than Ethernet. All of the ROOter boxes are living on the WAN side of my primary router in what I've been calling a Network of Gateways.

Here's the 4-port "firewall appliance" I'm using from Qotom via Amazon:

https://www.amazon.com/gp/product/B074XNYHL4/

In addition to some nice hardware specs for a router, I'll also give these guys a big thumbs-up for having it set to boot off a USB drive out-of-the box. There's a 16GB SSD inside this unit, and maybe I'll "flash it" at some point, but it's really convenient to be able to write an image to a USB3 flash drive, and boot from that — perfect for those of us that like to be able to reconfigure our setups with a minimum of fuss.

This unit is working with software VLANs too, which is great given all the different subnets commingling on the WAN side of my primary router. By running VLANs, I'm also able to access each of my ROOter routers by IP, which I was not able to do using macvlan on my previous Raspberry Pi 4 OMR box. My interface setup looks like this:
screenshot-192.168.100.1-2020.02.21-14_35_23.png
On first boot of OMR, the unit was configured with 1 LAN and 3 WAN ports, each on its own physical ethernet adapter — these units don't have internal switches. The LAN port is eth0, so be sure to connect there to access LuCI. Since I have a VLAN interface setup and available for each of my ROOter boxes, I reconfigured the 3 WAN ports to use eth1.3, eth1.4 and eth1.6 which represent VLAN tags of 3, 4 and 6 respectively. This is done by entering a "custom" physical interface, as you're setting the appropriate static addresses for your ROOter routers.

The only other thing I needed to do was to put in my OMR server key and VPS IP address, as described in my previous post. The end result, after a few moments for everything to connect-up looks like this:
screenshot-192.168.100.1-2020.02.21-14_30_53.png
And of course, the first order of business on the new Multipath TCP setup is to run a Speedtest:
screenshot-www.speedtest.net-2020.02.21-14_34_08.png
screenshot-www.speedtest.net-2020.02.21-14_34_08.png (43.78 KiB) Viewed 9558 times
A word of caution here, if your resulting speeds are anything like mine, you can use a LOT of data in a hurry running Speedtest.net on a setup like this. 3 tests and I had blown through a gigabyte of data! Also, some providers block multipath, the most notable one for me so far is Verizon. Not sure if they do it everywhere, but they are near me. AT&T and Sprint have been no problem, and they're the fastest anyway. :-) Also, I get limited value out of my 3rd WAN as it's the same provider, on the same tower as WAN1. It's nice to spread my data usage across all of these SIMs I'm paying for anyway. Speeds are great, and it's a very responsive and robust connection.

User avatar
terryjett
Posts: 325
Joined: Tue Sep 24, 2019 10:42 pm
Location: East Texas Piney Woods
Has thanked: 83 times
Been thanked: 72 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by terryjett » Wed Apr 15, 2020 1:31 am

Man, that is cool stuff and a lot on information to digest. Thanks for sharing with us and I am intrigued... Just do not have the time now to try this, but one day soon, I have got to!
people without a sense of humor should avoid me at all cost...

mezmerizz0454
Posts: 26
Joined: Wed Jan 29, 2020 5:56 am
Has thanked: 7 times
Been thanked: 3 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by mezmerizz0454 » Wed Apr 15, 2020 1:46 pm

i have read this 3 times i think it might sink in after 2 more times ,,and it is very interesting.
I'm pretty sure there are more than a few people here who carry the Old School Captain Crunch Whistle to make long distance phone calls :) keep the gold coming

User avatar
BillA
Posts: 922
Joined: Sun Dec 01, 2019 6:46 pm
Location: USA
Has thanked: 154 times
Been thanked: 191 times
Contact:

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by BillA » Sun Apr 19, 2020 3:15 am

Definitely an interesting project, however it may be a little too complex for the average user.
Having to run a virtual server in some remote data center incurs an extra monthly cost, and it's also an additional hop point for the data to take, including a potential security hole/snooping. The less hops the better, more reliable, lower latency, and more secure.

I have found a simple and inexpensive device for channel bonding using TP-Link's Safestream Multi WAN Router.
While it's sold as a 4 port load balancer, you can enable channel bonding by disabling "Application Optimized Routing" under Advanced > Load Balance > Configuration, to bond the bandwidth from multiple WAN inputs (up to 4 sources).

100Mbps/$60 - https://www.amazon.com/SafeStream-Broad ... B002T4D3L8
1,000Mbps/$160 - https://www.amazon.com/SafeStream-TL-ER ... B00A8NWU70

The basic connection setup looks like this:
Multiple Mobile/cable/DSL source's LAN port > into the WAN ports on the TP-Link Safestream then from LAN port > into the WAN port of a regular router/access point

There are a couple of ways to use the TP-Link box in order to bond multiple mobile connections, or cable/DSL in any combination. On all the internet source devices DHCP must be turned off since we don't want it to assign IP's.
1. The cheapest route (about $40 + the cost of them modem) is by connecting a mobile modem in an external USB enclosure to a GLI MT300N box to convert the modem's USB port into an ethernet port, connected to the TP-Link box.
2. For just a little more (about $50 + the cost of them modem), a more elegant all-in-one solution is by using an inexpensive WE826 router box with an internal modem connected to the TP-Link box. Course you'll need multiple WE826 boxes for multiple mobile modem connections, or in combination with a cable/DSL modem.

Technically any OpenWRT router could bind multiple internet sources as long as it has enough processing power, memory, and someone writes the necessary code for it. But for now the OpenMPTCProuter and the TP-Link box are the only viable solutions for true channel-bonding.

User avatar
bnhf2
Posts: 40
Joined: Sun Aug 05, 2018 10:03 am
Has thanked: 0
Been thanked: 24 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by bnhf2 » Sun Apr 19, 2020 10:08 am

There are definitely plenty of good applications for the TP-Link multi-WAN, fail-over, load-balancing router. However, what that box does and what OpenMPTCProuter does are not the same thing. OMR is true link aggregation, i.e. if you have two Internet connections where each speed tests at 50 Mbps, the resulting combination of the two using OMR should be around 90Mbps (or better). Devices like the TP-Link box, can do a good job of providing additional bandwidth, but will never be any faster than the fastest attached Internet connection.

As it turns out there's lots of upside to having a full-time Virtual Private Server on the other end of an OMR setup. First, OMR uses a VPN between the two as part of its magic -- so users get the advantages of a full-time VPN, without the usual performance penalty. Second, and this is a big plus for many cellular Internet users -- you now have a real public IP, complete with port-forwarding! OMR also supports having a Pi-hole (ad blocking DNS server) running on the VPS, which eliminates the need for anything ad blocking related to run on your LAN.

I agree with you that OMR is not for the average user, though I will say the developer has made it relatively straightforward to get up-and-running. VPS VMs are amazingly cheap these days (I think I pay $5/mo for mine), and there's little setup required beyond running a script on the VPS. There's processing power required on the OMR side (more than the average router), but something as simple and cheap as a Raspberry Pi is supported.

User avatar
BillA
Posts: 922
Joined: Sun Dec 01, 2019 6:46 pm
Location: USA
Has thanked: 154 times
Been thanked: 191 times
Contact:

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by BillA » Sun Apr 19, 2020 10:28 am

bnhf2 wrote:
Sun Apr 19, 2020 10:08 am
There are definitely plenty of good applications for the TP-Link multi-WAN, fail-over, load-balancing router. However, what that box does and what OpenMPTCProuter does are not the same thing. OMR is true link aggregation, i.e. if you have two Internet connections where each speed tests at 50 Mbps, the resulting combination of the two using OMR should be around 90Mbps (or better). Devices like the TP-Link box, can do a good job of providing additional bandwidth, but will never be any faster than the fastest attached Internet connection.

As it turns out there's lots of upside to having a full-time Virtual Private Server on the other end of an OMR setup. First, OMR uses a VPN between the two as part of its magic -- so users get the advantages of a full-time VPN, without the usual performance penalty. Second, and this is a big plus for many cellular Internet users -- you now have a real public IP, complete with port-forwarding! OMR also supports having a Pi-hole (ad blocking DNS server) running on the VPS, which eliminates the need for anything ad blocking related to run on your LAN.

I agree with you that OMR is not for the average user, though I will say the developer has made it relatively straightforward to get up-and-running. VPS VMs are amazingly cheap these days (I think I pay $5/mo for mine), and there's little setup required beyond running a script on the VPS. There's processing power required on the OMR side (more than the average router), but something as simple and cheap as a Raspberry Pi is supported.

I love the concept, and I understand that it's more than just a load-balancing router which is mainly useful on multi-segment downloads. Having OMR baked into the standard GoldenOrb release, would make it that much more popular.
However I'm still not sold on the idea of having to run a remote VPN server to combine the packets. If packet reassembly could all be done locally in the mobile router like on hardware based load-balancing/aggregating routers, then it would be a clear winner. Unfortunately 99% of non-technical users would be lost on the server setup, let alone the client/server settings. You and I might think it's easy, but having dealt with non-technical users for decades, trust me it ain't, just being realistic. But it would make a great business setting it up for them. ;)

User avatar
bnhf2
Posts: 40
Joined: Sun Aug 05, 2018 10:03 am
Has thanked: 0
Been thanked: 24 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by bnhf2 » Sun Apr 19, 2020 12:39 pm

BillA wrote:
Sun Apr 19, 2020 10:28 am
Unfortunately 99% of non-technical users would be lost on the server setup, let alone the client/server settings
My apologies if I gave the impression that this was an "anyone can do it" sort of post. There's little documentation on OMR, and no step-by-step guides for deploying it, so for anyone who feels they would need something like that to make it happen -- probably best to look elsewhere for a solution.

Let me try to break down the skills that would be needed:

For setting up the VPS, anyone that's ever created a virtual machine using a tool like VirtualBox, VMware or Parallels will have no problem understanding the concepts. "Know to work" cloud platforms, and virtual machine specifications are listed on the OMR website.

Once the VM is setup, a tool like PuTTY is required to be able to login in and run the couple of commands required to set it up. The script the OMR developer has written to do all the real work is fantastic, and is the same script that one runs to update the VPS when needed.

There are only a few types of client router devices supported, but they include x86 boxes (you can use anything from an Intel NUC to an industrial firewall appliance) and the Raspberry Pi. For these devices, one needs to be able to flash an SD card or USB thumb drive using a tool like Win32DiskImager or Etcher. There are a couple of consumer router type devices supported too, which would need to flashed using the same procedures as OpenWRT, ROOter, DD-WRT, Gargoyle or any other open source router OS.

In between your OpenMPTCProuter and the VPS will be your Internet gateways, which can be pretty much anything. DSL, cable modems, cellular hotspots, phones, WiFi as WAN links, ROOter routers -- whatever. And here's where things get interesting, as you can combine up to 8 of them (I've had as many as 5), and get near linear speed aggregation. For the user with better-than-average skills, in Internet-challenged areas, this can mean putting together a bunch of modest connections (say 4 10Mbps links) and ending up with a single 35Mbps virtual link, that won't miss a beat if one gateway goes offline.

So you're absolutely right, OMR is not built for the masses. But for those comfortable setting up and using an open source router OS like ROOter -- or building a cellular router solution using individual components such as those available through LTE Fix, I don't think it would be much of a stretch. And for the power user, that wants a public IP to host a webserver, or make the content on their Plex Media Sever available to friends and family, or host an MQTT broker to unlock some Internet of Things magic -- there's almost nothing available like it.

56387f
Posts: 47
Joined: Mon Feb 24, 2020 10:26 pm
Has thanked: 3 times
Been thanked: 18 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by 56387f » Sun Apr 26, 2020 11:06 pm

bnhf2 wrote:
Sun Apr 19, 2020 12:39 pm
So you're absolutely right, OMR is not built for the masses. But for those comfortable setting up and using an open source router OS like ROOter -- or building a cellular router solution using individual components such as those available through LTE Fix, I don't think it would be much of a stretch. And for the power user, that wants a public IP to host a webserver, or make the content on their Plex Media Sever available to friends and family, or host an MQTT broker to unlock some Internet of Things magic -- there's almost nothing available like it.
In your last post you summarized all things in a very nice way. Yes, I can see how having high bandwidth internet connection can be a certain advantage in case of hosting a web server because in this case "the more the better". I have dealt with load balancing/ failover configurations and a couple of times was thinking about multiple WAN aggregation scenarios but somehow every time in the end I simply could not justify the cost & complexity to achieve a decent result. Instead, every time I just switched to a faster/better service and eventually (170 Down/150 up FIOS) reached the level when I stopped caring about the speed - if it becomes 300/250 or 3000/2500 it will not change a thing for me.
So I wonder now how nowadays a power user who does not host a web server could potentially benefit for WAN bonding?

User avatar
bnhf2
Posts: 40
Joined: Sun Aug 05, 2018 10:03 am
Has thanked: 0
Been thanked: 24 times

Re: WAN Bonding with ROOter and OpenMPTCProuter

Post by bnhf2 » Tue Apr 28, 2020 10:12 am

56387f wrote:
Sun Apr 26, 2020 11:06 pm
I have dealt with load balancing/ failover configurations and a couple of times was thinking about multiple WAN aggregation scenarios but somehow every time in the end I simply could not justify the cost & complexity to achieve a decent result. Instead, every time I just switched to a faster/better service and eventually (170 Down/150 up FIOS) reached the level when I stopped caring about the speed - if it becomes 300/250 or 3000/2500 it will not change a thing for me.
The equation changes of course for those with good residential Internet service available, as one just doesn't have the same issues and challenges as for those who depend on cellular, satellite or rural DSL Internet. I probably should have mentioned this in my original post, though it may have been implied by this being on the "LTE Hacks" forum and my use of ROOter (a cellular specific version of OpenWRT).

In my case, as a full-time RVer for the last decade, my Internet service is far from predictable, and changes every time we re-locate our motorhome. As such, OpenMPTCProuter has really been a dream-come-true, in terms of being able to better utilize the multiple cellular plans I pay for (Sprint, AT&T and Verizon ), of always having a public IP (which I would otherwise only get from Sprint) and a full-time VPN as a bonus. But it's the link speed aggregation that really makes it unique, and different from other ways of utilizing multiple gateways.

OMR is one of those projects that for those who understand what it does, have multiple Internet connections available, and are stuck behind carrier-grade NAT, can be game changing. More often than not though, it gets confused with basic load balancing and failover/failback solutions. Part of the reason for this I believe is that there are very few similar products. Peplink has their original SpeedFusion product (which is quite expensive, so not much used by individuals), there's Speedify (which has never shown me much speedwise), and not much else.

Post Reply