cancel
Showing results for 
Search instead for 
Did you mean: 

Poor data transfer rate in general, streaming issues in particular

Chris
Legend
Posts: 17,724
Thanks: 600
Fixes: 169
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Hi Mike,
The VMBU tool on your account was showing different usage and times due to a workaround we'd tried to fix the underlying issues you're reporting. We removed the restrictions from happening on your account this month to allow us to monitor this issue and ensure that you didn't get restricted which may have clouded the issue. That should correct itself shortly.
We've continued looking into this and there aren't any rate limits on your account that would cause the streaming issues you are reporting. The only rate limit that may be in place for this is undetected CDN (akamai etc) traffic, and that is currently set at 4Mb so is well above the speeds you're seeing.
It's very odd that you're seeing Gold 2 traffic as the only things in there are the likes of rapidshare.
If you could provide a full wireshark (starting before you start streaming), that'll help us see where the stream is being established and pin down the issues.
Former Plusnet Staff member. Posts after 31st Jan 2020 are not on behalf of Plusnet.
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

Might have been a good idea for someone to have mentioned in advance that things were generally being tinkered with and consequently VMU would be U/S for the time being and all that Wink  Things have apparently changed again today and VMU data doesn't seem to make much sense at all now.  I never normally look at VMU (only when there's some problem or other related to usage) so time to resume never using it I think !
An ethereal capture of iplayer streaming data being tagged $60 on 13th May is available from HERE although I've no idea what the programme actually was now and it's probably no longer available anyway.  It was simply another random selection from the iplayer home page at the time just to grab some more data for a quicky looky see. Note that the problem started part way through playback, perhaps immediately after the intro or something, or perhaps just randomly part way through the programme itself (as appears to have happened on other occasions) when iplayer presumably decided there was insufficient bandwidth for reliable playback using RTMP so switched to HTTP or something. I really can't be sure and in any case, your guess as to why iplayer switches streams is probably just as good as mine anyway.  
There are just too many variables, too many unknowns and absolutely no consistency whatsoever meaning that trying to do anything even remotely close to a sensible investigation is nigh on impossible.  All I have is lots of chunks of data, a few with pretty obvious issues such as this one but virtually all looking fairly 'normal' regardless of whether they actually played sensibly or otherwise.  
Exactly the same problem can also be seen during one of the iplayer diagnostic tests conducted earlier on the same day. It perhaps explains why some of the individual diagnostic tests were seen to fail intermittently whilst the others ran and the results looked almost normal. An ethereal capture of an iplayer diagnostic test containing a large block of data being tagged $60 along with some random BBC website accesses also being tagged  $60 is available from HERE  
There have been no particularly obvious problems with iplayer streaming so far this week but then again, I think it's been streaming RTMP exclusively.  I don't think I've noticed any HTTP streams at all.  I suspect that HTTP is used for low quality low bandwidth streams and it's probably a combination of factors that results in problems being experienced.  An unreasonably low BW restriction in general triggers iplayer to stream HTTP and this HTTP data stream is then sometimes being restricted even more than everything else is being restricted due to incorrect data classification issues perhaps ?  I dunno, but I do know that pulling the plug and walking away is beginning to seem like by far the best solution for absolutely everyone concerned !!
PLEASE NOTE that the time stamps in the captures may well look more than a bit odd - they certainly do when I look at them here but the old copy of ethereal I have handy has always shown strange things wrt timing and I have no idea why.


B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Thanks for the details.
We can see within the capture what is being said, though we are also struggling with the timestamps it is producing. There is no logic to what it is being produced.
Whilst we are ignoring it for the most part and using the usable portions, there is a small nagging feeling that if this isn't populated right, what else may not be correct? We are not saying it is useless (yet), but would you mind exploring some more why that is happening so we can discard the nagging feeling Wink
I note you have a Draytek. Simply for confirmation, can you confirm the model you have and what physical setup you have please? (diagrams would be nice Wink ). Also can we confirm if you have any WAN load balancing, failover and or using anything other than the factory default QoS rules in the unit (which are disabled).
In the interim, we are working to go and replicate that, but are without one of our important toys this morning to do that. We will get back once we have it.
Many thanks.
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

The timestamps have always displayed as weird, it's been like that for, I dunno, probably the best part of 10 years !  Every capture will be much the same.  In fact every recent capture I currently have *IS* pretty much exactly the same as are various ancient ones that I still have hanging around from god knows when.  A shiny new capture made today will still be displayed with weird timestamps and will just contain different data and so on that's all.  I can't seem to tell which version it is but the exe is dated Dec 2002 so not exactly the most up-to-date version and all that  Roll_eyes
The data is always apparently in the right order (i.e. packets appear to be quite obviously in the order they were received) but the stated timestamps are often just plain crazy. I have never been able to get to the bottom of why and have long since given up thinking about it TBH. I have no ideas at all other than perhaps because it's just a bug-ridden antique version of ethereal.  I'll try to grab a later version sometime to try in future (providing that it will actually still run on Ye Olde Laptop that is) just to see if it still displays timestamps in much the same way or perhaps provides something rather more sensible.
However, no matter how crazy the timestamps allegedly are, it doesn't alter in any way the fact that packets are clearly arriving with a seemingly incorrect and completely unexplained DSF.  Data from some (but by no means all) servers that obviously should be tagged $80 is quite clearly being tagged $60 on some (but by no means all) occasions.  There's no getting away from that fact IMHO, nothing can influence DSF in any way and nor would it even try to.  To even vaguely consider otherwise is *seriously* clutching at virtually non-existent straws I think !
I can well understand putting ??'s on the data quality in general though as indeed I have done myself many times in the past but I have yet to find that captures provide anything other than what they should be doing apart from the displayed dodgy timestamp. I've always just assumed that it was a display or cosmetic type issue rather than something 'real' so to speak but it could equally be something influenced by my router, switch, hub etc.  I don't have a problem accepting that if it's true it's just that I have absolutely no idea where that particular bit of data comes from, how it's established or how it could be influenced by equipment various along the way. My guess would be that it's something done (very badly !) by ethereal when sniffing the packet off the wires rather than some dodgy data that is actual present within the packet on the wires. Happy to proven wrong or whatever as I'd dearly love to know why it's been doing it and consistently doing it ever since day 1.  My pure guess  would be that it's a maths and/or general overflow error somewhere.
What I can say though is with this particular problem, I can also assure you that a problem seen in practice (i.e. vid not playing sensibly or diagnostic failing badly) correlated to the apparent incorrect DSF in the streaming data and this is further supported by a Netmeter graph showing an almost negligible data transfer rate for the iplayer stream at the time.  The Netmeter graph posted on 14th May (near the top of page 2 of this thread) was grabbed around the same time as the ethereal streaming data capture and is almost without doubt the very same data. It was all done just after 2300 hours on 13th May. The Netmeter graph was produced and captured by the PC receiving and trying to display the iplayer stream. The ethereal capture was done by a laptop tee'd into the ethernet connection using a hub. I'm not running ethereal on the same PC that is trying to display an iplayer stream. It simply wouldn't be able to cope sensibly with that.
Yes it's a Draytek.  Yes it's very ancient, a bit cranky at times and well past it's prime in general ... just like me really  Grin  But equally, it's been 100% reliable for almost a decade and almost certainly cannot influence strategic discrete bits of data within packets such as the DSF for certain packets of data from certain sources and only at certain times. That's just not likely to happen is it even if there were a fundamental bug of some kind ?  There's absolutely nothing clever going on at this end either. It's just a bog standard vanilla router from years ago.  No load balancing, no fancy QoS options or anything even remotely similar.  In fact most allegedly clever options and facilities are turned off in any case - I don't want (or need) 'clever', I just want simple and reliable that's all. The firewall is about the only thing that's heavily used and that's really only relevant to unsolicited incoming data and port mapping etc. Nothing there should affect routine data requested by a local PC and there's certainly nothing that I'm even vaguely aware of to tweak things like DSF.  It's just not that level of router, all those kinda features didn't arrive until several models down the line.  If the iplayer problem returns with a vengeance again then I do have alternative routers to slip in on a temporary basis to see what effect (if any) that has - probably none whatsoever needless to say - but doing that is not going to be without several issues of it's own because the other routers aren't set up appropriately and aren't 100% compatible with a whole load of existing systems and monitoring. It's not something I can use long-term without a major amount of hassle and loads of additional work but it is something I can (and will) try for a short period of time if/when the problem is at it's worst and at least somewhat more consistently bad and much less infuriatingly very intermittent !

PS: I'm going to delete those capture files now because the ast time I posted similar links to massive files on here, way too many silly people DL'd them way too many times and very nearly got my webspace suspended.


B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Hi Mikeb,
Thanks for the reply, though I have just deleted a rather long winded item and had to start from scratch, as you have thankfully answered some important points I almost looked silly for repeating Wink
If you could attempt a upgrade, that would be handy as it would ensure this isn't an issue. The good news is, it is unlikely the router, switch and so on causing the timestamps. PCap packet captures are recommended to record the timestamp in seconds since EPOCH, then the microseconds as well (which should never equal or be greater than a whole second) within the packet header detail. It states it should be from the capture time of the frame, the frame itself doesn't contain the timestamp.
So this is going to be a software glitch.
I'm not inclined to completely rule out a software glitch for other possible issues in the capture, but the more we read the data, the less likely we see it being an issue (mainly given the pattern of the problem and the fact that it captures it in the way we would expect it in a non-glitchy capture).
We have not been able to replicate what you are seeing at all. We have configured another account to be the same your own one and performed captures, yet they all come out as 0x80 and none of the sessions are 0x60.
Without replicating this live, it's really issue for us to look at the switch and gateways and see what detection category was used for that session. Then in turn look at that detection routine and work out why/how it got applied to the captured data.
This is one of those circumstances that we have to see the whole thing at the same time to get around it.
We have however spotted a common denominator that may be why you are seeing this (which will help us progress to the rest of the above). However the verification is not present in the output and there is a suggestion it would have been present before the capture started.
There is some hope though. I see in the filename of one of your captures that you have marked is as "filtered". Can I assume that you have only saved a filtered portion of your capture and had more captured? Do you by any chance still have the RAW capture at all?
If so, can you run the following display filter on it and save the filtered output?

ip.addr == 88.221.0.0/16

The pattern we are seeing is that the packets match these IPs which fall in our Akamai Content Delivery Network filters for streaming. Akamai also provide a number of others services, including on these IPs.
One train of thought is that there was another connection to the same IPs or other IPs in this range that are correctly defined as 0x60. Further packets are then marked as 0x60 (but not in all cases) which may or may not be the correct process.
Seeing all data for those IPs may help this.
If you run that display filter on "iplayer problem 13-05-2012.pcap", you will see frames 39 & 40, with reset acknowledgements for previous sessions to two different IPs in the same range. There are no previous data frames from that IP so we cannot see if they were 0x60 or 0x80.
Another supporting item is that frame 387 has the same remote destination port as frames 39 & 40 and is marked 0x60. However frame 391 has different remote destination ports and is 0x80.
So we need to see if we can narrow down what previous application was using a session to the Akamai CDN.
Do you use iPlayer desktop download client at all for anything? Is this running?
Do you use any other download applications that may use a CDN like Akamai?
If possible, do you have a number I can call on this evening so we can look at this further, as whilst the back and forth is natural, the gaps between are clearly the frustrating point. If you can PM me details that will be handy as I may not be able to pickup your account details later.
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

Quote
There is some hope though. I see in the filename of one of your captures that you have marked is as "filtered". Can I assume that you have only saved a filtered portion of your capture and had more captured? Do you by any chance still have the RAW capture at all?

Did I ? There was no intentional filtering of any kind in either capture. They should both be complete captures either starting just before heading to the relevant webpage and hitting the play/start button or at the very least from just before hitting the play/start button if the webpage was already loaded or cached. The captures were then terminated manually after a minute or 2 depending on what was actually going on. Anything too much longer than that and things start to get more than a bit out of hand on Ye Olde Laptop !  It's quite possible that frames were dropped because ethereal couldn't keep up but other than that it should be all there and intact.
Perhaps I've screwed up somewhere when generating the (now deleted) zipfiles or you misread the file names ?  The diagnostic test data capture file should have been "iplayer diagnostic peak time rate limited.cap" meaning that it was the standard iplayer diagnostic test (probably in full) and conducted during peak time when traffic was apparently being unreasonably rate-limited. The streaming data capture file should have been "iplayer problem 13-05-2012.cap" meaning it was the first minute or two of some random iplayer programme that was streamed on 13th May and seen to be not playing properly.  File creation dates are a reasonable indication of when the captures were made as I always try to remember to save them ASAP but it is quite possible there could be a delay if I get side-tracked by something !
Quote
Do you use iPlayer desktop download client at all for anything? Is this running?
Do you use any other download applications that may use a CDN like Akamai?

No and not that I'm aware of I'm afraid.  There shouldn't have been anything else significant going on at the same time as the captures, at least nothing intentional anyway.  There could possibly have been some background update going on I suppose but TBH I try really hard to permanently disable any such things and I don't willingly give anything the right to make connections and DL stuff behind my back no matter what !  I'll have a good look at the data in question a bit later and whilst I don't have ethereal captures from immediately before those you already have, I do have comprehensive access logs available so may be able to work out more about what it was and what was going on from there. My guess would be that it was just something else that happened to be on the same iplayer page or perhaps more likely, another browser tab had some other content being displayed just before I did the capture. It probably represents a data connection for a photo on the active iplayer page, a photo or short video clip for some other inactive iplayer page I may also have had in the browser or perhaps an advert of some kind etc. on a completely unrelated webpage.
Quote
If possible, do you have a number I can call on this evening so we can look at this further, as whilst the back and forth is natural, the gaps between are clearly the frustrating point. If you can PM me details that will be handy as I may not be able to pickup your account details later.

Ummm, not tonight ( Josaphine Tongue ) unless you're still up and raring to go right now anyway !  

Thoughts for the day: I think that what you're sort-of suggesting might be going on here is quite honestly an almost perfect example of why I have always had such a major problem with the way PN selectively target specific high bandwidth sources for often unreasonable throttling.
PN presumably have no official and definitive information whatsoever relating to which websites use which 3rd parties to host their data let alone sufficient intimate information about the various specific websites and/or the data content itself to be able to establish exactly what data rate is actually required for everything to work exactly as intended. PN presumably haven't really got the faintest idea what the data is that is being served by any one specific server at any particular point in time.  I do not know whether PN effectively keep lists of IPs that they consider sources of "streaming data" and "routine data" etc. or whether decisions are made on-the-fly based on some form of real-time inspection of the data itself or how it was requested but either way, PN generally appear to potentially be routinely making wild assumptions about someone else's data and then judging it's relative importance to the world. There is absolutely no reason whatsoever why any one specific server IP cannot be the source of both "routine data" and "streaming data" at much the same point in time on much the same place on a website. Indeed, these kind of server IP's appear to change unbelievably frequently at the best of times. For instance, a lookup for an Akamai server having a TTL of more than a few minutes seems to be somewhat rare whenever I've bothered to check and all that.
Consider this as a possible explanation of what's going on:
PN have arbitrarily decided that media content in the form of still images or short video clips on a mostly descriptive webpage are "routine data" whereas complete program video data is "streaming data". It is then their intention that the former will be given a low priority and often be heavily limited whereas the latter will be given a much greater priority with no significant limiting at all. In order to achieve their aims they think they have established various IPs providing the two different types of media and/or they think they can reliably detect the two types of media on-the-fly so they've programmed their traffic management systems accordingly. The reality of course is that the website owner and/or data host has 100% control over these IPs along with the data being served. The source for "routine data" and "streaming data" can easily be swapped around willy-nilly as and when they feel like it and PN will never know for sure that it's happened let alone be able to respond in a timely manner to any such changes. PN has absolutely no control whatsoever over what is happening in real time.  Similarly, both types of data could quite legitimately end up being supplied from exactly the same IP at exactly the same time.  And all of this can (and often does it seem) change every few minutes. Everything is 100% outside of PN's control and yet PN always appear to be making some pretty wild assumptions and taking some often drastic action when the reality appears to be that they don't really have any idea of the effect that they're actually having. They're just trying to save bandwidth (and no doubt money) by hitting "easy targets" without having to be too open about exactly who they're hitting, when or why and all of this happens as a matter of routine without there being any reasonable degree of certainty that they're not fundamentally breaking things somewhere. Only when a sufficient number of complaints start flooding in is a problem of some kind suspected.
Isn't that perhaps exactly what's going here ?  The same Akamai server or block of servers just so happened to randomly end up serving "routine data" and "streaming data" at much the same point in time for much the same website and all of this activity is 100% out of PN's control of course.  The BBC and Akamai have every right to do what they like when they like ... PN has very little right (IMHO) to arbitrarily decide which data is important and which data is not or indeed to ultimately be able to effectively stop a service from working completely after making wild assumptions that are, in some specific instances, quite obviously incorrect. The system may work just fine for a significant part of the time in perhaps most instances but things can so easily and quickly change leading to a service randomly stopping working completely due to an incorrect data type classification. Such changes may be temporary, quite specific and very short lived or fairly widespread and of a much more permanent nature, only time along with the number of complaints received will flag this up.  Just think about the various 4oD issues not so long ago. Data classification and other problems regularly screwing things up and it was eventually confirmed and resolved yet apparently there was an almost complete lack of reports coming in from other customers for several days/weeks.  Intermittent problems in general are usually always an absolute bl**dy nightmare but this is an order of magnitude worse I think due to the sheer number of variables and unknowns involved.  

PS: It was apparently ethereal V0.9.8 with WinPCAP V2.3 from checking my program archives and both are indeed 2002 vintage at best. The good news is that updating WinPCAP to V3.1 (the last version that I can use) appears to have resolved the timestamp issues for new captures but obviously can't do much about any existing ones. I didn't get very far at all trying to update ethereal so gave up for now and will continue to live with the occasional random crankiness of this antique yet amazingly useful  beast !!  I'll do a couple of captures and post links a bit later just so that you can check things out.  They almost certainly won't show a major DSF problem of course although they should generally be much the same in content to the two captures that you already have but with somewhat more sensible timestamps ... hopefully so anyway !

PPS: 88.221.84.xxx is *very* common occurrence in my logs for 13th May and is also being accessed on a quite regular basis almost all day every day.  It seems to often be an Ad Server of one form or another based on some of the adjacent access. It seems closely related with accesses to many sites that I visit on a very regular basis ... including the BBC iplayer page ! In fact it's *very* prevalent on the iplayer home page just now. 88.221.94.xxx (as was seen in the ethereal captures) seems to occur primarily around iplayer accesses.  Thing is, PN have no control over what these IPs serve and you should really expect IPs to change now and again or perhaps even regularly as well as being used by many quite different websites for many quite different things. Trying to work out which data you can easily get away with throttling without anyone really noticing and which you definitely can't was never going to be even remotely easy long-term  Grin

ethereal test capture #1: iplayer diagnostic test 26-05-2012.zip
ethereal test capture #2: iplayer streaming test 26-05-2012.zip
Have a look at the last but 1 packet in the 2nd test file above ... it's from one of our 'favourite' servers [88.221.84.34] and appears to have DSF = $00, a bit odd perhaps when all previous relevant packets appeared to be $80 as expected ?


B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

Right then, some more relevant info to answer your earlier questions and this should hopefully help to pin down exactly what was going on here. Firstly, I had a good old trawl through the access logs for more info on the connection(s) being queried from the ethereal streaming data capture on 13/05/2012.
Connections to local ports 1201 (RTMP) and 1202, 1203 & 1204 (HTTP) from 88.221.94.135 were opened consecutively and all appear to be related to something on a BBC website. All other adjacent accesses also appear to be in some way related to something BBC anyway and at a reasonable look through IPs various, I can't see any other sites being accessed at around the same time.  Interestingly though, I can't actually see any record in the logs of these connections actually being closed either.  Perhaps even more interesting though, they also appear to have been opened around 1 hour prior to the more relevant connections detailed in the ethereal streaming data capture for 13/05/2012. Here's an extract from my access logs confirming this:


[iplayer diagnostic test]
2012-05-13 21:55:57        Local2.Info        192.168.1.1        May 13 20:56:42 vigor-router-adsl: Local User: 192.168.1.111:1200 -> 212.58.225.190:80 (TCP)Web
...
2012-05-13 21:56:37        Local2.Info        192.168.1.1        May 13 20:57:23 vigor-router-adsl: Local User: 192.168.1.111:1201 -> 88.221.94.135:1935 (TCP)
2012-05-13 21:56:37        Local2.Info        192.168.1.1        May 13 20:57:23 vigor-router-adsl: Local User: 192.168.1.111:1202 -> 88.221.94.135:80 (TCP)Web
2012-05-13 21:56:37        Local2.Info        192.168.1.1        May 13 20:57:23 vigor-router-adsl: Local User: 192.168.1.111:1203 -> 88.221.94.135:80 (TCP)Web
2012-05-13 21:56:37        Local2.Info        192.168.1.1        May 13 20:57:23 vigor-router-adsl: Local User: 192.168.1.111:1204 -> 88.221.94.135:80 (TCP)Web
...
2012-05-13 21:56:39        Local2.Info        192.168.1.1        May 13 20:57:25 vigor-router-adsl: Local User: 192.168.1.111:1200 -> 212.58.225.190:80 (TCP) close connection
2012-05-13 21:56:51        Local2.Info        192.168.1.1        May 13 20:57:37 vigor-router-adsl: Local User: 192.168.1.111:1205 -> 4.26.228.126:80 (TCP)Web
2012-05-13 21:56:51        Local2.Info        192.168.1.1        May 13 20:57:37 vigor-router-adsl: Local User: 192.168.1.111:1206 -> 8.27.0.254:1935 (TCP)
2012-05-13 21:56:51        Local2.Info        192.168.1.1        May 13 20:57:37 vigor-router-adsl: Local User: 192.168.1.111:1207 -> 209.84.7.254:80 (TCP)Web
2012-05-13 21:56:51        Local2.Info        192.168.1.1        May 13 20:57:37 vigor-router-adsl: Local User: 192.168.1.111:1208 -> 209.84.7.151:80 (TCP)Web
2012-05-13 21:56:51        Local2.Info        192.168.1.1        May 13 20:57:37 vigor-router-adsl: Local User: 192.168.1.111:1205 -> 4.26.228.126:80 (TCP)Web
...
[iplayer streaming data test]
2012-05-13 22:57:45        Local2.Info        192.168.1.1        May 13 21:58:30 vigor-router-adsl: Local User: 192.168.1.111:1361 -> 212.58.246.90:80 (TCP)Web
2012-05-13 22:57:45        Local2.Info        192.168.1.1        May 13 21:58:30 vigor-router-adsl: Local User: 192.168.1.111:1362 -> 87.248.213.117:1935 (TCP)
2012-05-13 22:57:45        Local2.Info        192.168.1.1        May 13 21:58:30 vigor-router-adsl: Local User: 192.168.1.111:1363 -> 95.140.227.94:80 (TCP)Web
2012-05-13 22:57:45        Local2.Info        192.168.1.1        May 13 21:58:30 vigor-router-adsl: Local User: 192.168.1.111:1364 -> 95.140.227.49:80 (TCP)Web
2012-05-13 22:57:45        Local2.Info        192.168.1.1        May 13 21:58:30 vigor-router-adsl: Local User: 192.168.1.111:1365 -> 95.140.227.49:80 (TCP)Web
2012-05-13 22:57:46        Local2.Info        192.168.1.1        May 13 21:58:31 vigor-router-adsl: Local User: 192.168.1.111:1361 -> 212.58.246.90:80 (TCP) close connection
2012-05-13 22:57:49        Local2.Info        192.168.1.1        May 13 21:58:34 vigor-router-adsl: Local User: 192.168.1.111:1366 -> 212.58.246.90:80 (TCP)Web
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:34 vigor-router-adsl: Local User: 192.168.1.111:1367 -> 212.58.227.138:80 (TCP)Web
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:34 vigor-router-adsl: Local User: 192.168.1.111:1368 -> 88.221.94.166:80 (TCP)Web
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:35 vigor-router-adsl: Local User: 192.168.1.111:1369 -> 88.221.94.166:1935 (TCP)
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:35 vigor-router-adsl: Local User: 192.168.1.111:1370 -> 88.221.94.167:80 (TCP)Web
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:35 vigor-router-adsl: Local User: 192.168.1.111:1371 -> 88.221.94.167:80 (TCP)Web
2012-05-13 22:57:50        Local2.Info        192.168.1.1        May 13 21:58:35 vigor-router-adsl: Local User: 192.168.1.111:1366 -> 212.58.246.90:80 (TCP) close connection
2012-05-13 22:57:56        Local2.Info        192.168.1.1        May 13 21:58:41 vigor-router-adsl: Local User: 192.168.1.111:1367 -> 212.58.227.138:80 (TCP) close connection
2012-05-13 22:58:00        Local2.Info        192.168.1.1        May 13 21:58:45 vigor-router-adsl: Local User: 192.168.1.111:1372 -> 212.58.227.138:80 (TCP)Web
2012-05-13 22:58:05        Local2.Info        192.168.1.1        May 13 21:58:50 vigor-router-adsl: Local User: 192.168.1.111:1372 -> 212.58.227.138:80 (TCP) close connection
2012-05-13 22:58:10        Local2.Info        192.168.1.1        May 13 21:58:55 vigor-router-adsl: Local User: 192.168.1.111:1373 -> 212.58.227.138:80 (TCP)Web


However, having now just looked around for any other useful data that I may have had lying around, I can see exactly what was going here.  Fortunately so can you as well because you do actually have all the relevant ethereal data to hand for around 2200 hours as well as for around 2300 hours !
The BBC/iplayer traffic at around 2200 hours - i.e. that to local ports 1200(ish) - was primarily an iplayer diagnostic test that was conducted around 1 hour prior to the iplayer streaming data test that involved traffic to local ports 1300(ish).  There was certainly very little going on in the way of any other traffic of any other kind in between these times or indeed for most of the evening.  My comments posted in this thread several hours later state that iplayer wasn't working sensibly all evening and general web access was also pretty dire all evening as well which explains precisely why there was nothing much going on other than the occasional test of one form or another. You already have the ethereal capture for both of these tests in addition to a typical netmeter graph and speed test results etc. at around the same times.  My service was virtually unusable all evening on 13/05/2012 as opposed to just being inexplicably bad.
The "iplayer diagnostic peak time rate limited" ethereal capture clearly shows the ports you are questioning being used for iplayer related traffic and being tagged $60. This capture also contains a block of iplayer data for one of the individual tests being incorrectly tagged as $60 as well of course.
The "iplayer problem 13-05-2012" ethereal capture is the streaming data test conducted mostly out of morbid curiosity around 1 hour later. This capture also clearly shows the streaming data being incorrectly tagged as $60 as well.  Needless to say the vid wouldn't play at all with such a major restriction in transfer rate.
My post made several hours later in this thread (HERE) contains a netmeter graph showing typical transfer rates for web and iplayer data from sometime around 2300 hours.  The iplayer streaming data is most likely to be the exactly same iplayer streaming data for which you have the ethereal capture. Data transfer rates are clearly quite poor in general but the iplayer data stream in particular is obviously completely wrong and totally unusable.  The speed test results also included in this post show general rate-limiting in place for the entire peak-time period.
Please note, however, that I think it would be very wrong to jump to a conclusion that iplayer streaming data tagged with $60 and a connection apparently rate-limited to 2Mb/s were both symptoms of exactly the same thing and are therefore directly related. Rate-limiting had clearly been seen on many days but incorrectly tagged iplayer data had not been obvious at the same time. There is definitely no indication that it had been consistently present at the same time that's for sure. In fact there is actually various evidence of iplayer streaming data being correctly tagged $80 although still being problematic of course due to rate-limiting and the related issues that caused of course.
I suspect (but have no hard evidence to support it) that iplayer data was being intermittently tagged $60 at various times (not necessarily just during peak-time or when rate-limiting in general was in place either) but it most definitely wasn't happening all the time.  The 13/05/2012 was quite clearly a *very* bad day, significantly worse than all other days, but intermittent problems had clearly been present to a generally speaking lesser extent for absolutely ages. It's very obvious when iplayer data is just "not quite right" from a quick look at netmeter even when it's sort-of playing reasonably well despite apparent bandwidth issues. In general though, there's always a massive difference in the graph when traffic is being incorrectly tagged. There's always a significant difference when iplayer traffic is just not quite right as well. Same thing with 4oD et al. It's usually pretty obvious to see when something just isn't as it should be even if it's not possible to see why.
I think it's important to note that all the file dates/times appear to stack up almost perfectly with the dates/times in the access logs. There are 3 different PCs and several different applications and lots of my very own fingers involved here ! They can't all be wrong in some major way but still come up with much the same date/time stamps for much the same data.  The ethereal timestamps may well be completely screwed up but it's pretty clear exactly when the captures were made and exactly what was going on at the time. It's just the timing of individual packets that is somewhat less than correct or in any way clear in ethereal although that could perhaps be established if it would help in some way. It would certainly be possible to establish when each of the various local ports were opened and subsequently closed if nothing else. It would also be possible to confirm gateway and all relevant connection stats at the time as well of course.
I've not seen any incorrectly tagged data at all recently.  In fact the only time that I have been able to find data tagged as $60 was by visiting the rapidshare site itself.  Even data for the home page of the site is tagged $60 never mind any random files being served by rapidshare. I've just not seen $60 data at any other time or place whenever I've been out looking for it. I've also not been seeing HTTP iplayer data much if at all recently either. Virtually everything is RTMP and is coming in at (or at least reasonably close to) the maximum available rate. There is absolutely no evidence or even any reasonable suspicion to suggest that iplayer RTMP data from anywhere was ever being screwed up in a major and identifiable way. Such data might not have been streaming particularly well or even playing that well at the time either but there was never any proof of something definately very wrong ... just lots of indications of something not quite right !  It has always been HTTP data and HTTP data from Akamai servers in particular that has been exceptionally problematic in a variety of ways, including being tagged $60 rather than $80.
With virtually no HTTP streaming data being seen, it is highly unlikely that any significant problems will be found or at least no useful evidence will be obtained even if whatever was going on to cause the problems being experienced is still going on today.  Does anyone have any suggestions for how I can somehow force iplayer to stream in HTTP ? I've tried a few things in an attempt to persuade it to use HTTP but all I seem to achieve is stopping it from working at all or it continues to stream in RTMP.  Clicking on "lower bandwidth" doesn't seem to do much if anything but then again it does say lower bandwidth IF available and all that.
As I said a while back, I'm kinda convinced that it was a combination of things causing the problems and I suspect that it was all being kicked off by low bandwidth in general. Low bandwidth probably leads to HTTP data being used and relatively poor quality playback.  Some HTTP data from some sources was then intermittently being tagged incorrectly leading to even lower bandwidth, even more problems with playback and ultimately resulted in iplayer giving up completely. The one thing that's very apparent the whole time is unexplained bandwidth restrictions various but most often, no real consistency either.  


B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

Here's another interesting ethereal capture (with dodgy timestamps unfortunately) grabbed at around 2030 hours on 11/05/2012.  Just like the capture you have for 13/05/2012, this one also appears to contain a certain amount of HTTP data via Akamai that is being tagged with $60 rather than $80. However, it doesn't appear to be the streaming data itself, it looks like something else on the same page, probably a still image or something.  The streaming data appears to be HTTP data via Limelight and that is correctly tagged with $80.  
The simple fact that I bothered to save the capture and name it as I did tells me that iplayer wasn't working very well if at all sensibly at the time. A post in this thread several hours later also confirms that as does the netmeter graph and iplayer diagnostics included in that post. The netmeter graph and iplayer diagnostics result (duplicated below for convenience) show what the iplayer streaming data rate actually looked like at some point in time less than 45 minutes after the ethereal capture. The netmeter graph might be the same data as in the ethereal capture but I can't be sure and it's probably unlikely as the graph was saved quite some time after the capture rather than almost immediately after. It's perhaps much more likely to be a continuation of the data showing the point where iplayer gave up trying to stream it due to bandwidth issues.  However, I would suggest that it's quite reasonable to assume that the netmeter graph is a fair indication of how things were looking for most of the time around when the capture was done. Other data I have available indicates that streaming data was (presumably) being watched between ~2030 hours until shortly after 2100 hours.  This would suggest that the ethereal capture is the very start of this programme and the netmeter graph was grabbed to show the point when iplayer hiccupped and subsequently ground to a complete halt with an error message.
Apart from the generally low and somewhat inconsistent data rate for the majority of the netmeter graph, note the break in the stream and in addition, a short burst of data at a ridiculously low rate in the middle of the portion after the stream attempted to restart and before iplayer presumably gave up completely.  This is almost exactly how incorrectly tagged data streams generally look in netmeter i.e. nothing like the expected 4Mb/s I think you mentioned elsewhere but at least an order of magnitude below that.   Has someone somewhere managed to get their bits and bites confused so the limits are a factor of 10 out in reality ?   Finding that a programme started playing pretty badly although just about watchable and then deteriorated or had significant breaks in it until ultimately giving up was a fairly typical experience around the time these tests were done.


Here's the relevant excerpt from the access log to help validate timing etc:

2012-05-11 20:32:26        Local2.Info        192.168.1.1        May 11 19:32:34 vigor-router-adsl: Local User: 192.168.1.111:1144 -> 212.58.246.94:80 (TCP)Web
2012-05-11 20:32:26        Local2.Info        192.168.1.1        May 11 19:32:34 vigor-router-adsl: Local User: 192.168.1.111:1145 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:32:26        Local2.Info        192.168.1.1        May 11 19:32:34 vigor-router-adsl: Local User: 192.168.1.111:1146 -> 95.140.226.147:80 (TCP)Web
2012-05-11 20:32:26        Local2.Info        192.168.1.1        May 11 19:32:34 vigor-router-adsl: Local User: 192.168.1.111:1147 -> 95.140.226.156:1935 (TCP)
2012-05-11 20:32:27        Local2.Info        192.168.1.1        May 11 19:32:34 vigor-router-adsl: Local User: 192.168.1.111:1148 -> 95.140.226.147:80 (TCP)Web
2012-05-11 20:32:27        Local2.Info        192.168.1.1        May 11 19:32:35 vigor-router-adsl: Local User: 192.168.1.111:1144 -> 212.58.246.94:80 (TCP) close connection
2012-05-11 20:32:31        Local2.Info        192.168.1.1        May 11 19:32:39 vigor-router-adsl: Local User: 192.168.1.111:1145 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:32:37        Local2.Info        192.168.1.1        May 11 19:32:45 vigor-router-adsl: Local User: 192.168.1.111:1149 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:32:40        Local2.Info        192.168.1.1        May 11 19:32:48 vigor-router-adsl: Local User: 192.168.1.111:1149 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:32:47        Local2.Info        192.168.1.1        May 11 19:32:55 vigor-router-adsl: Local User: 192.168.1.111:1150 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:32:47        Local2.Info        192.168.1.1        May 11 19:32:55 vigor-router-adsl: Local User: 192.168.1.111:1151 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:32:51        Local2.Info        192.168.1.1        May 11 19:33:00 vigor-router-adsl: Local User: 192.168.1.111:1151 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:32:54        Local2.Info        192.168.1.1        May 11 19:33:03 vigor-router-adsl: Local User: 192.168.1.111:1150 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:32:56        Local2.Info        192.168.1.1        May 11 19:33:05 vigor-router-adsl: Local User: 192.168.1.111:1152 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:33:00        Local2.Info        192.168.1.1        May 11 19:33:09 vigor-router-adsl: Local User: 192.168.1.111:1152 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:33:06        Local2.Info        192.168.1.1        May 11 19:33:15 vigor-router-adsl: Local User: 192.168.1.111:1153 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:33:12        Local2.Info        192.168.1.1        May 11 19:33:21 vigor-router-adsl: Local User: 192.168.1.111:1153 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:33:16        Local2.Info        192.168.1.1        May 11 19:33:25 vigor-router-adsl: Local User: 192.168.1.111:1154 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:33:21        Local2.Info        192.168.1.1        May 11 19:33:30 vigor-router-adsl: Local User: 192.168.1.111:1154 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:33:26        Local2.Info        192.168.1.1        May 11 19:33:35 vigor-router-adsl: Local User: 192.168.1.111:1155 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:33:30        Local2.Info        192.168.1.1        May 11 19:33:39 vigor-router-adsl: Local User: 192.168.1.111:1155 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:33:32        Local2.Info        192.168.1.1        May 11 19:33:41 vigor-router-adsl: Local User: 192.168.1.111:1156 -> 212.58.246.85:80 (TCP)Web
2012-05-11 20:33:33        Local2.Info        192.168.1.1        May 11 19:33:41 vigor-router-adsl: Local User: 192.168.1.111:1157 -> 88.221.84.80:80 (TCP)Web
2012-05-11 20:33:38        Local2.Info        192.168.1.1        May 11 19:33:47 vigor-router-adsl: Local User: 192.168.1.111:1156 -> 212.58.246.85:80 (TCP) close connection
2012-05-11 20:33:39        Local2.Info        192.168.1.1        May 11 19:33:48 vigor-router-adsl: Local User: 192.168.1.111:1125 -> 92.123.154.32:80 (TCP) close connection
2012-05-11 20:33:42        Local2.Info        192.168.1.1        May 11 19:33:51 vigor-router-adsl: Local User: 192.168.1.111:1106 -> 88.221.84.75:80 (TCP) close connection
2012-05-11 20:33:42        Local2.Info        192.168.1.1        May 11 19:33:51 vigor-router-adsl: Local User: 192.168.1.111:1107 -> 88.221.84.75:80 (TCP) close connection
2012-05-11 20:33:55        Local2.Info        192.168.1.1        May 11 19:34:04 vigor-router-adsl: Local User: 192.168.1.111:1112 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:33:58        Local2.Info        192.168.1.1        May 11 19:34:06 vigor-router-adsl: Local User: 192.168.1.111:1113 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:33:58        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1108 -> 88.221.84.75:80 (TCP) close connection
2012-05-11 20:33:58        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1115 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:33:59        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1118 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:33:59        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1105 -> 88.221.84.75:80 (TCP) close connection
2012-05-11 20:33:59        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1117 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:33:59        Local2.Info        192.168.1.1        May 11 19:34:07 vigor-router-adsl: Local User: 192.168.1.111:1119 -> 88.221.84.73:80 (TCP) close connection
2012-05-11 20:34:02        Local2.Info        192.168.1.1        May 11 19:34:11 vigor-router-adsl: Local User: 192.168.1.111:1104 -> 88.221.84.75:80 (TCP) close connection
2012-05-11 20:34:26        Local2.Info        192.168.1.1        May 11 19:34:35 vigor-router-adsl: Local User: 192.168.1.111:1158 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:34:30        Local2.Info        192.168.1.1        May 11 19:34:39 vigor-router-adsl: Local User: 192.168.1.111:1158 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:35:25        Local2.Info        192.168.1.1        May 11 19:35:35 vigor-router-adsl: Local User: 192.168.1.111:1159 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:35:25        Local2.Info        192.168.1.1        May 11 19:35:35 vigor-router-adsl: Local User: 192.168.1.111:1160 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:35:27        Local2.Info        192.168.1.1        May 11 19:35:37 vigor-router-adsl: Local User: 192.168.1.111:1157 -> 88.221.84.80:80 (TCP) close connection
2012-05-11 20:35:30        Local2.Info        192.168.1.1        May 11 19:35:39 vigor-router-adsl: Local User: 192.168.1.111:1160 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:35:33        Local2.Info        192.168.1.1        May 11 19:35:43 vigor-router-adsl: Local User: 192.168.1.111:1159 -> 212.58.227.137:80 (TCP) close connection
2012-05-11 20:36:25        Local2.Info        192.168.1.1        May 11 19:36:35 vigor-router-adsl: Local User: 192.168.1.111:1161 -> 212.58.227.137:80 (TCP)Web
2012-05-11 20:36:30        Local2.Info        192.168.1.1        May 11 19:36:39 vigor-router-adsl: Local User: 192.168.1.111:1161 -> 212.58.227.137:80 (TCP) close connection



B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Thanks for the update,
Good to hear the timestamp issue looks to be resolved. A quick check shows that version did have a known issue. So not so uncomfortable with the older version of Ethernet any more, though the risk of dropped frames given the age of the machine does still exist, but I am not too concerned as you would expect Ethereal/Wireshark to complain on its analysis of the flow (mainly through a mass of different coloured packets).
Quote from: mikeb
Have a look at the last but 1 packet in the 2nd test file above ... it's from one of our 'favourite' servers [88.221.84.34] and appears to have DSF = $00, a bit odd perhaps when all previous relevant packets appeared to be $80 as expected ?

What you are seeing is not the cause of the problem, but we are going to raise a query with our vendors to get this clarified.
Established sessions take resources, so dead sessions must be timed out. So when we perform analysis of the packet on the platform, it first looks to see if it is an established flow within the system and uses this to mark the packet accordingly. When a session times out, the established flow does not exist, so it performs analysis of the packet from scratch again.
The default for that specific packet you highlighted should be the exact same result for the SYN/ACK for the same TCP stream. The SYN/ACK was 0x80, thus the RST should be too. It is noted that the RST is a poorly formed packet though, which could be a factor. Regardless, we are going to get it analysed some more.
Quote from: mikeb
Has someone somewhere managed to get their bits and bites confused so the limits are a factor of 10 out in reality ?

Yes, no, maybe and everything in-between.
The situation here is that something is going on and at this time, only appears to be impacting yourself. One of the major issues is that each capture seen, whilst confirming the issue, does not enough enough data prior to your flow to see what happened before the 0x60 classification. This is am important point.
Next, is the issue of replication. We have been unable to do so, which is really hampering efforts. I'm not ruling something out related to the setup there, but there is clearly enough determined in this thread to suggest this is nothing to do with that. The fact it does only impact you means we are also reliant on your captures and I am afraid to say, these are going to need to continue until they contain enough data prior to a 0x60 stream to determine the problem.
A pattern will emerge, it's a case of when rather than if.
I think you have jumped to a lot of conspiracy theorist conclusions in your response. You feel that we may possibly be classifying this as general data either on purpose or other suggestive deliberate ways.
I'm going to go out on a limb here and say, yes, you are correct.
Lets face it, we make no bones we perform traffic management, which is a fully automated process. However we are not deliberately performing management in a way that is causing your problem. Something is going wrong here, either on the platform or is some other capacity.
The fact we are in this thread trying to locate the cause should be evident of that. All the captures confirm an issue, but is not the cause of your problems in every case (as many streams are correctly classified).
It would be ideal if possible please if we can stick to the facts. The screenshots of the download meters are really not helping, neither is the conjecture. If we can keep this to the captures showing the problem, that would be ideal, though by all means analyse them yourself and let us know what you are seeing (a second set of eyes always helps).
mikeb
Rising Star
Posts: 463
Thanks: 15
Registered: ‎10-06-2007

Re: Poor data transfer rate in general, streaming issues in particular

So what exactly do you want me to do here ? ... because running ethereal 24/7 so that you have the desired bucketful of data from whatever was going on way before some random and intermittent event occurs just isn't ever going to happen !  I'm quite sure that you can understand fully that it's totally impractical.  It's plenty enough of a PITA to have to have a 2nd system up and running then remember to start ethereal manually every time I think that I might be about to do something that might possibly result in something strange happening.  
The two most relevant captures you have were started immediately prior to doing something that might have resulted in something strange happening. Something strange did in fact happen and was therefore faithfully recorded.  But of course there were also many more  occasions when nothing identifiably strange happened although something still wasn't quite right. Whilst what happened immediately before hitting the old capture button might well be very interesting or perhaps even highly relevant and all that, realistically, the data is never going to be readily available.  I mean just how much data before some random and intermittent event of interest are you expecting here ? 1 second, 10 seconds, a minute ... an hour !  The two captures you have from 13/05/2012 were done around 1 hour apart yet there are still things of major interest in the first capture continuing on into the second.  This seems to imply that had the capture been running continuously for the entire 1 hour period then this still wouldn't have been good enough and you'd probably still have wanted data for some undefined period of time before the first capture.  Obtaining monster length captures is impractical as is trying to analyse them or even get them to you. So what EXACTLY is it that you need/want here ?  Tell me exactly what you want and I'll certainly do my very best to achieve it if I possibly can.  But please keep it practical and realistic as well as bear very much in mind that I'm a paying punter not a paid member of staff !  I fully appreciate that only I can provide the necessary data to document exactly what's happening right here right now ... but there's only so much that I can sensibly do in between the shedload of stuff that I either want to do, have to do or should be doing Smiley
I've not been seeing anything particularly odd happening since the middle of May now but then again you (and BT I suspect) have tweaked no end of things since then.  The profile problem, the usage recording problem, the whatever was causing the unexpected transfer rate issues and/or routine packet loss issues have all apparently been sorted in one way or another ... and not by doing anything at this end of the wires either.  You've also apparently fiddled with the usage recording and/or monitoring systems in PN Towers as well and I'm not sure if that's back to as it should be or not. In short, way too many things have apparently changed and so much is now quite different to what it was when the problem was at it's worst.  
Things that were clearly happening on an intermittent but quite regular if not almost predictable basis (and had been for ages) no longer seem to be. For instance, I'm no longer seeing any appreciable amount of HTTP iplayer traffic and it was almost certainly HTTP traffic in particular that was being seriously affected by whatever was going wrong.  I'm no longer seeing 3rd party (and Akamai in particular) hosted data on sites such as BBC and eBay et al fairly obviously being treated somewhat differently in some way to what is quite likely to be very similar routine data on other very much less popular sites. I'm no longer seeing general browsing being quite so sluggish for no apparent reason either at various times. In addition, I only ever use iplayer on average for 30 minutes maximum a day during peak time so that's the one and only opportunity I have to try and capture anything odd that might happen with iplayer each day. Apart from anything else, using iplayer any more frequently than that will stuff my BW allowance completely in next to no time. Any further streaming required is always off-peak where no particular problems have ever been experienced, or at least none on anything even remotely close to a regular basis anyway. With iplayer now behaving fundamentally differently to the way it was before and with other no doubt contributory factors now addressed, the chances of actually capturing something going a bit wrong do seem to be somewhat remote now and capturing what went on for some undefined period of time prior to something that might go a bit wrong seems to be somewhat unrealistic into the bargain.
Soooooo ... ball back in your court I think.  Put everything back to exactly as it should be for my specific A/C (i.e. remove any tweaks, fiddles, fudges and anything else you might have done along the way) so that everything is 100% back to 'standard' and then tell me exactly what it is you want me do in an attempt to confirm whether all is now well or otherwise along with how/when to do it. There's absolutely no point either of us wasting our time b*ggering around here if we don't know what it is we're trying to achieve and/or the test conditions are perhaps not truly  representative of real life either. If the problem has been fixed somewhere along the way then that's good. If it's simply gone away of it's own accord or been masked by something else so it's just much less visible or much less likely to occur  then that's obviously not so good. But only you guys know what's been going on behind the scenes and what (if anything) might have contributed to what appears to be a significant  improvement since the beginning of the month.
I'd still like to know if anyone has any suggestions for how to persuade iplayer to stream HTTP on a regular basis rather than primarily streaming in RTMP.


B T Plusnet, a bit kinda like P T Barnum ...

... but quite often appears to feature more clowns Tongue
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Quote from: mikeb
So what exactly do you want me to do here ? ... because running ethereal 24/7 so that you have the desired bucketful of data from whatever was going on way before some random and intermittent event occurs just isn't ever going to happen !  I'm quite sure that you can understand fully that it's totally impractical.  It's plenty enough of a PITA to have to have a 2nd system up and running then remember to start ethereal manually every time I think that I might be about to do something that might possibly result in something strange happening.

Unfortunately I am unfamiliar with the what options are available through the version of Ethereal you have. However running it 24/7 is perfectly feasible, without any significant impact to you (though I do refer to an average system rather than the older one you have acknowledged).
I've attached a screenshot to the post. The following options can be used to allow almost continuous capture without significant resource of storage needs.
Under the file section, browse and give a filename to store in, then set the following after doing that.
* Select "Use multiple files"
* Select the first "Next file every" and set it to a reasonable value, such as 5MB
* Select "Ring buffer with" and set this to something between 7 and 10
Depending what MB value you stated and the ring buffer, you would only store a limited volume and allow it to capture continuously. I've used this to captures multiple days worth of data under a very low memory footprint on a machine (though still suspect it might have been slightly better that the machine you are using).
Quote from: mikeb
The two most relevant captures you have were started immediately prior to doing something that might have resulted in something strange happening. Something strange did in fact happen and was therefore faithfully recorded.

The problem with said captures, is they are still too late. The start of the incorrect 0x60 flags are almost always very early in the stream. Ok, they are not frame one, but are still very soon. The problem we are facing and information we believe is before the captures, potentially several hundred to thousands of frames before. More accurately, we may need to see anything from 1 to 10 minutes worth of data from possible previous flows to the same IP range.
One specific thing we ideally need is the SYN packet onwards for previous flows.
We absolutely recognise this is a hard task and would do this for you, but again, we are having difficulty in doing so.
I've not finished with your post yet and will get back to the rest of it. I'm off to a meeting but though I would get these important points in first.
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Phil,
You seem sure that this is a problem only affecting Mike. I'd suggest it's far more likely that only Mike has the knowledge and determination to get to the bottom of what's happening and that others have experienced the same and found that if they just stop it and start it again it's all back to as it should be so the issue either never gets reported and/or recognised.
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Morning all,
This has not been dropped. Looking into this took a little more time than hopped and had to jump back to a few things. Fortunately (or unfortunately depending how you look at it), we knew this wasn't something that was going away and could come back to it.
Jelv,
I agree in part. There is a chance this not the only case of this, though all others so far have not shown what is quite easy to replicate in this instance.
That isn't to say they are not the same problem, but most others are not indicated or have other reasons so far.
We are still having difficulty trying to replicate this ourselves and feel this is something quite unique to what MikeB is doing (not strictly unique to the hardware, but things like viewing habits and what else hi system is accessing).
All we can really say here is that we are going to be in this for the long haul on trying to do this and in reality, I am hoping we can replicate it here. Getting to that point is frustrating for all though.
ejs
Aspiring Hero
Posts: 5,442
Thanks: 631
Fixes: 25
Registered: ‎10-06-2010

Re: Poor data transfer rate in general, streaming issues in particular

Quote from: mikeb
I'd still like to know if anyone has any suggestions for how to persuade iplayer to stream HTTP on a regular basis rather than primarily streaming in RTMP.

Maybe configure your firewall to block outbound connections to TCP port 1935 - then it might fallback to something else.
Is this 33MB capture any use?
http://ejs1920.users.sourceforge.net/bbc-streaming-20120531.pcap
It starts from opening the BBC home page in newly launched SeaMonkey. From what I saw in it, every packet from 88.221.*.* had "Differentiated Services Field: 0x60 (DSCP 0x18: Class Selector 3; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))". There's a live HTTP iplayer stream towards the end.
prichardson
Grafter
Posts: 1,503
Thanks: 1
Registered: ‎05-04-2007

Re: Poor data transfer rate in general, streaming issues in particular

Thanks ejs,
We are going to review that shortly, which is going to take some time. Your notes suggest that it is hitting and seeing the symptoms we are looking for though.