Latency Issues? (Solved: QMI > MBIM)

Related to Modems, Hotspots, Dongles ONLY
Post Reply
Falias
Posts: 13
Joined: Sat May 25, 2019 12:01 am

Latency Issues? (Solved: QMI > MBIM)

Post by Falias » Sat Jun 15, 2019 9:12 pm

Hi all -

I have a wg3526 set up with an em7565 on an unlimited verizon prepaid sim. The modem gets around the same signal strength and upload/download speeds (~15-20/75-90) as my android phone sitting next to it, but the latency is significantly higher with the modem... They are both connecting to the same tower. (They both switch between bands quite often, but this issue happens over all of the ids.)

I tried matching the phone's default ttl of 52 (and also 53, 64, and 117) with no effect. (I can see that it's working on the ping response, but it doesn't decrease the latency.)

Any ideas?
Thanks

From the router:
327 packets transmitted, 327 packets received, 0% packet loss
round-trip min/avg/max = 43.217/65.820/98.320 ms

From the phone's verizon connection (running concurrently):
30/40/175

(Amusingly this short run had the two closer than the one I did overnight last night... that one was like 36 average v 71. This run still shows it though - the minimum ping with the router is higher than the cell phone's average.)

xdavidx
Posts: 244
Joined: Tue May 28, 2019 4:04 pm

Re: Verizon Latency Issues?

Post by xdavidx » Sun Jun 16, 2019 12:06 am

Just to understand the test conditions... When you are testing with the router, what is connected to it and how is it connected?

When you are testing with the phone, are you running something on the phone, or are you connecting another device to the phone? If so, what is connected and how?

Falias
Posts: 13
Joined: Sat May 25, 2019 12:01 am

Re: Verizon Latency Issues?

Post by Falias » Sun Jun 16, 2019 1:43 am

The router has a computer attached by ethernet cable which I'm ssh'ing from - the actual ping is being run on the router itself though.

Nothing is connected to the phone; the ping is being run in Termux.
https://play.google.com/store/apps/deta ... com.termux

Thanks

xdavidx
Posts: 244
Joined: Tue May 28, 2019 4:04 pm

Re: Verizon Latency Issues?

Post by xdavidx » Sun Jun 16, 2019 8:28 am

Well, that certainly reduces the variables.

Thoughts/ideas:

- Try running traceroute on each connection to see if the packets are taking the same path (well, the same IP path, there could still be a different path under IP, I guess).

- It is possible that connection state could be handled differently, with the router modem making and breaking connections for each transmission.

- It is possible the router modem is going down to a pre LTE connection for small amounts of data. There are commands to force it to only use LTE as a test.

- It is possible that something else is sending data from the PC (or wifi, if other devices are attached to the wifi radios in the router) when you are pinging. You can use windows firewall to exclude everything except your ssh program. It is a bit of a pain to set up. Or you could disconnect the ethernet and try wifi from the phone, just as a test, even though wifi itself might add latency. Just keep the phone right next to the router. The phone has background transmission possibilities too.

- How many pings are you doing with each?

- Maybe specify the packet size on each to be sure it is the same.

serverside
Posts: 64
Joined: Wed Jul 11, 2018 3:02 pm

Re: Verizon Latency Issues?

Post by serverside » Sun Jun 16, 2019 2:20 pm

HAH I thought I was the only one. Something is going on with Verizon. 9+ months with your basically same setup, prepaid unlimited Verizon. Used to get 60-90ms ping on discord and games. Now this last week, it's now hitting minimum 150ms ping in discord and 200ms in games. I've reset the modem contacted the reseller to reset the data, I've checked different antennas etc. If I connect via my phone it does a good bit better.

I think something is going on with Verizon, what region or state are you in? I'm in mid TN.

This is super frustrating as I work remote and this causes a lot of latency issues with my job.

I'm thinking about doing a NEMA box build and moving it to the top of my property or attaching it to a tall tree, but I don't know how much this will help.

This wouldn't help anything would it?
https://ltehacks.com/viewtopic.php?f=15&t=330

Falias
Posts: 13
Joined: Sat May 25, 2019 12:01 am

Re: Verizon Latency Issues?

Post by Falias » Sun Jun 16, 2019 8:46 pm

Thanks for the ideas...

1. Will check tracerts tonight and update.

2. I had noticed that they both dropped ca without a connection, so I tried with a background download at ~2mbps. It had no effect... but that was also downloading from a different server than I was pinging. I'm not really sure what a good way to isolate/test this would be - any thoughts (even on the same server it would be tracked as a different connection...)?

3. I have both selrat=6 and bands restricted to lte. (I've actually watched the bands/tower ids on both the phone and router while doing the pings... They aren't always on the same one at the same time, but they both swap between the same 3 quite often - during the test window even.)

4. The computer is running linux; I don't have any internet related operations in the background except for a connectivity check and ntp. (As an aside: I was absolutely floored when I had to use windows 10 at one point and it was not only downloading in the background, but that download was invisible to both the task manager and resmon... not having any control over your computer is an alarming norm.)

5. The one that I quoted was only a few hundred, I ran one overnight when I first found the problem though... so that would have been ~30,000.

6. I had also run the command from the phone with wifi on and off... so the ping program/parameters were identical... the results were consistent with the other tests - going through the wifi, the latency was around double.

@serverside
I'm disheartened that you're having the issue too... that makes it less likely that it's something fixable... though maybe it's temporary if you've just started having it. (Hopefully!)
I'm in central VA...
I have decent reception, so I don't think that's it.
No, I tried the TTL changes - it has no effect for me.

Thanks again

xdavidx
Posts: 244
Joined: Tue May 28, 2019 4:04 pm

Re: Verizon Latency Issues?

Post by xdavidx » Sun Jun 16, 2019 9:58 pm

Falias wrote:
Sun Jun 16, 2019 8:46 pm
2. I had noticed that they both dropped ca without a connection, so I tried with a background download at ~2mbps. It had no effect... but that was also downloading from a different server than I was pinging. I'm not really sure what a good way to isolate/test this would be - any thoughts (even on the same server it would be tracked as a different connection...)?
Yeah, they'll drop the CA, but should keep the connection to the primary band. However, trying with and without, from a testing perspective was worthwhile, so nothing wrong with you doing that. I would think that pinging on the primary band, without CA, would reduce variables. If the phone is somehow maintaining CA the whole time and splitting the ping "load" across bands, I suppose that could lead to a difference in the phone vs the router.

Do you have some way to enable/disable CA and/or lock/block bands on the phone?
Falias wrote:
Sun Jun 16, 2019 8:46 pm
3. I have both selrat=6 and bands restricted to lte. (I've actually watched the bands/tower ids on both the phone and router while doing the pings... They aren't always on the same one at the same time, but they both swap between the same 3 quite often - during the test window even.)
If the tower ids change, then that would be a reasonably big difference, not only because the data is flowing through a different path, but the characteristics of the connection between one tower and another could add to latency. And switching back and forth in the middle would obviously require breaking and making connections, hence adding to latency.

This is my best guess for what might be going on, but it is only a guess. The phone might jump around between towers less often. Additionally, if there is interference between the towers, maybe the phone handles retransmissions differently than the router.

As for the bands, maybe lock to a single band to reduce that to a single variable. That assumes you can lock it on the phone too. But even if you can't, you can see if any single band on the router gives you better latency than the others.
Falias wrote:
Sun Jun 16, 2019 8:46 pm
4. The computer is running linux; I don't have any internet related operations in the background except for a connectivity check and ntp. (As an aside: I was absolutely floored when I had to use windows 10 at one point and it was not only downloading in the background, but that download was invisible to both the task manager and resmon... not having any control over your computer is an alarming norm.)
Linux helps with that variable. I agree that the lack of control is horrible. It took me a good bit of time to figure out how to use the firewall to disable everything, except what I wanted enabled, and how to switch back and forth easily.

Hmm...I seem to see the Windows Update network activity in task manager and resource monitor on my machines, both in Windows 7 and Windows 10. Not sure why it wouldn't show on your end, but it is Windows, of course. :lol:

How did you see the data activity?
Falias wrote:
Sun Jun 16, 2019 8:46 pm
5. The one that I quoted was only a few hundred, I ran one overnight when I first found the problem though... so that would have been ~30,000.
Should be enough. :lol: Just to be positive, maybe run them again on each system to be sure it is consistently slower on the router.

And try running both devices at the same time and in a mutually exclusive way, just in case there is some sort of interference going on. Longshot.
Falias wrote:
Sun Jun 16, 2019 8:46 pm
6. I had also run the command from the phone with wifi on and off... so the ping program/parameters were identical... the results were consistent with the other tests - going through the wifi, the latency was around double.
In the other stats you supplied, average was about 66 ms vs 40 ms. Did the wifi increase that difference even more? I would have thought a couple milliseconds, but not 10's of ms.

Just as a test, you could try pinging on the linux machine through the router and you could try setting up the phone as a hotspot and pinging from another device through that connection. It doesn't seem like it is ping program related, but maybe something will jump out.

You could also try setting the ping send buffer size to ensure it is consistent and you can try increasing that to see if it has more of an effect on one device versus the other. Again, I'm just brainstorming.

If it is something on the Verizon side of things, the question would be why does it only affect the router and not the phone? If there is some sort of throttling going on (tied to the sim in the router), I guess it could affect latency too. You said the TTL changes didn't have an effect, however, and your speeds are great.

Not sure if your phone sim works in the router and vice versa, but that is a possible test.

It is only a guess, but my best guess is that the connections jumping are leading to differences. That doesn't really explain the mininum latency being different between the two. It is hard to know if that min was a fluke for the phone and the max was a fluke for the phone and it is running 99% in the 40 ms range and that the router is also mainly in the 40 ms range, but had a lot of 90+ ms samples, due to jumping around between towers, and that skewed the average.

Maybe one handles interference better than the other.

A directional antenna is another thing that could be used to test. Of course, that's an expensive test. Alternatively, you could try bringing the router to a single tower and see what that shows for latency of the router and phone each having a non-changing connection to a tower and a very strong signal and less interference. You'd have to run it off an inverter in the car.

serverside
Posts: 64
Joined: Wed Jul 11, 2018 3:02 pm

Re: Verizon Latency Issues?

Post by serverside » Mon Jun 17, 2019 8:42 am

Falias wrote:
Sun Jun 16, 2019 8:46 pm

@serverside
I'm disheartened that you're having the issue too... that makes it less likely that it's something fixable... though maybe it's temporary if you've just started having it. (Hopefully!)
I'm in central VA...
I have decent reception, so I don't think that's it.
No, I tried the TTL changes - it has no effect for me.

Thanks again
Hmm, with us being in different states, I doubt it's Verizon. What I'm worried about is that the bands we are connected to are now congested. What I noticed yesterday was that when I forced the connection to band 4, which is super weak, it connected and I had great ping, but my upload/download was trash. I use the same service for my phone, and today I was thinking of putting my phones Sim into the router to make sure that I'm not being throttled. Being de-prioritized shouldn't have an effect on latency, but just speed for U/D, so I don't think it's this. If it is, I'll cancel this account and spin up a new one on the same plan.

I also have 2 yagi's coming vs my Proxicast 2x2 panel, which I think because it's a panel isn't able to get a strong enough signal. I am thinking of setting them up at 45 degrees and seeing if I can get CA connected. As CA connects randomly but is 99% of the time no connected.

Also, how do you do the TTL changes? I wanted to try this but I couldn't find on my version of GoldenOrb in the Network -> Firewall, any TTL settings.

Falias
Posts: 13
Joined: Sat May 25, 2019 12:01 am

Re: Verizon Latency Issues?

Post by Falias » Mon Jun 17, 2019 12:35 pm

Okay... multiple problems with trying to check the tracerts... nothing insurmountable though.

First, on the modem, all ip4 tracerts show a total of 1 hop... I assume verizon is 4>6 natting it immediately and then I just see nothing. While that doesn't happen on the phone, it doesn't appear to be the issue (the latency issues happen with ipv6 too).
(An interesting note here: they are both getting a public 100.x.x.x ip4, but the phone is in a /27 whereas the router is in a /32... as far as I'm aware, there's no point to having a subnet with a single host since it wouldn't be able to talk to anything... which I guess is why it's natted immediately... seems odd.)

The two connections do use different routes for 2001:4860:4860::8888, so I found a server that was in the common path early on at 2001:10:10::1 ... which shows the same pattern. (Path is router/phone > gw > hop that doesn't respond > 2001:10:10::1)

It also turns out the ping6 command in android (at least on my phone) uses a restricted syscall... so can't ping v6 addresses with the phone unless I root... which is annoying. I'm using the hotspot for the ping tests below - it makes the connection significantly worse, but even so, it's still clearly faster than the router's connection.

Histograms:
Figure_2.png
Figure_1.png
These were run concurrently from the computer with both the router and phone on LTE13 without CA (I bandlocked the router and disabled CA, and then I just watched the phone and ended the ping command as soon as it changed.) I ran the pings every 0.2 seconds to get more samples before the phone switched... I ended up with ~2300 samples on each over ~7.5 minutes.

Like I said, using the phone through the hotspot makes it far worse - running on the phone directly gives a much more normal distribution... just a few in 40-45 and almost none above that.

Other clarification: I'm always connected to the same physical tower on both, there are just different cellids for the different bands. I'm in the mountains and I actually wasn't able to connect to other towers even when trying with the directional antenna.

Falias
Posts: 13
Joined: Sat May 25, 2019 12:01 am

Re: Verizon Latency Issues?

Post by Falias » Mon Jun 17, 2019 12:49 pm

serverside wrote:
Mon Jun 17, 2019 8:42 am
Also, how do you do the TTL changes? I wanted to try this but I couldn't find on my version of GoldenOrb in the Network -> Firewall, any TTL settings.
That tab just adds the following two rules to "Custom Rules" (make sure it's the correct interface)
iptables -t mangle -I POSTROUTING -o wwan0 -j TTL --ttl-set 52
iptables -t mangle -I PREROUTING -i wwan0 -j TTL --ttl-set 52

Verizon CA (at least for me) is only active during during a download/upload... so you may actually be using it without realizing it.

I would recommend not canceling the plan... it isn't offered anymore; we only have it grandfathered as long as the line stays active.

In regards to you trying a new antenna; these are my stats (while getting 101/31 on a speed test):
LTE band: B66 LTE bw: 20 MHz
LTE SSC1 state:INACTIVE LTE SSC1 band: B13
LTE SSC1 bw : 10 MHz
PCC RxM RSSI: -79 PCC RxM RSRP: -100
PCC RxD RSSI: -80 PCC RxD RSRP: -102
SCC1 RxM RSSI: -65 SCC1 RxM RSRP: -84
SCC1 RxD RSSI: -65 SCC1 RxD RSRP: -84
RSRQ (dB): -7.3
SINR (dB): 20.0
So I don't think that getting a better antenna would resolve it.

Post Reply
  • Similar Topics
    Replies
    Views
    Last post