# Northern Rock and online banking capacity



## askew70 (15 Sep 2007)

BraveInca said:


> - When the system is operating normally it asks for a _different_ pair of characters each time. The fact that it is not doing this is a symptom of it being overloaded IMO.



No it's not, it is a symptom of it being badly implemented. Every time you connect to the server, to login, it has to issue you with a prompt for your details - this obviously takes processing power so if you get a prompt then the server had the grunt to get this far in the process. It also has to select two characters to prompt you for, and it has to record which character positions those are to match against your full password later - this is the step at which it should nominate which two random characters which you are expected to supply, and the fact that it doesn't select random characters every times means that the selection process is not really random at all, which equates to weaker security because the system is badly designed and/or implemented and not because the server(s) doesn't have the resources to function.



BraveInca said:


> - The bank cannot be expected to design computer systems that have the capacity to handle a run on the bank - it just wouldn't make any sense, and there are sound business reasons why you wouldn't want your computer systems to be able to smoothly handle a run!


 

Of course a bank can be expect to design systems that handle the numbers of customers it has. Anything less than that amounts to poor and shoddy service. NR have a system which is badly implemented either because the people they contracted to build it don't know what they are doing, or because NR are not willing to put up enough money to pay for a real and proper system (i.e. well designed and with adequate hardware and software to allow it to function well).

As an example of a system that works well, 24 hours a day, for many millions of people, look at Google. When was the last time that you got a timed out connection when you used Google's search facility. And that involves hefty database lookups which require significant horsepower (and/or an extremely well designed system). The Google system is so good, because if it weren't no-one would use it and Google would cease to exist. The NR system being bad, relatively speaking, reflects badly on NR.


----------



## BraveInca (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> Of course a bank can be expect to design systems that handle the numbers of customers it has. Anything less than that amounts to poor and shoddy service. NR have a system which is badly implemented either because the people they contracted to build it don't know what they are doing, or because NR are not willing to put up enough money to pay for a real and proper system (i.e. well designed and with adequate hardware and software to allow it to function well).
> 
> As an example of a system that works well, 24 hours a day, for many millions of people, look at Google. When was the last time that you got a timed out connection when you used Google's search facility. And that involves hefty database lookups which require significant horsepower (and/or an extremely well designed system). The Google system is so good, because if it weren't no-one would use it and Google would cease to exist. The NR system being bad, relatively speaking, reflects badly on NR.


 
I guess you don't have much experience of IT systems. 

All IT systems are designed to handle expected volumes and peaks within a certain set of parameters. It would be ludicrous to pay for a system that could handle 75% of customers logging in at once when your normal volume is, say, 0.05 % of customers logging in at the same time.

Google is designed to handle Google's expected volumes, NR is designed to handle NRs expected volumes.

Ever heard of a Denial of Service attack? Where a hacker simulates many simultaneous http requests to bring down a website? This has happened in the past to Yahoo and Paddypower, and is the equivilant of what is happening to NR at the moment.

Lastly, even if it wasn't massively expensive to allow 100% of customers to withdraw at the same time, no sane bank would implement a system that could do that.


----------



## GeneralZod (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*

I'd like to know if the design of the system allows the same number of successful transactions to be completed when it's under high load. If it's poorly designed it could actually be processing fewer transactions than when it isn't under high load. This could account for the long periods during peak load when nobody reported getting logged in and completing a transaction. This explanation doesn't need a conspiracy theory to account for the reduced transaction processing rate.

As Askew70 said a well designed system like google's (where programmers are given the time and respect necessary to do the job right) reacts well under load by not locking out all users. Any well designed system also uses dimensioning rules for expected processing requirement rates so that costs may be balanced with benefits.


----------



## askew70 (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



BraveInca said:


> I guess you don't have much experience of IT systems.



Oh, only about 15 years or so. Will that do, at all?



BraveInca said:


> All IT systems are designed to handle expected volumes and peaks within a certain set of parameters. It would be ludicrous to pay for a system that could handle 75% of customers logging in at once when your normal volume is, say, 0.05 % of customers logging in at the same time.



So, it is ludicrous to pay for a system that can accommodate even the majority (let alone all) of your customers at the same time? Someone better tell places like Tesco, for example, that they should reduce the number of cash registers in their stores, as Tesco seem to be under the impression that lack of resources in that area will result in their customers choosing to go elsewhere. 

I wonder if you would be so forgiving of NR if you were a customer that dealt with them at a branch with one desk to service all customers, regardless of whether the number of customers was 1 or 1,000?



BraveInca said:


> Google is designed to handle Google's expected volumes, NR is designed to handle NRs expected volumes.



I take it then that if you are a customer of NR then you have always found their web service to be very speedy and responsive? Posts on this thread, and my own personal experience, suggest you would be in the minority if so.



BraveInca said:


> Ever heard of a Denial of Service attack? Where a hacker simulates many simultaneous http requests to bring down a website? This has happened in the past to Yahoo and Paddypower, and is the equivilant of what is happening to NR at the moment.



Ah yes, Denial of Service attacks. Yup, I know about those. Typically though, a provider of a service would design their system so that their own customers trying to avail of the service does not cause a Denial of Service. Usually the cause of a Denial of Service is expected to be a malicious or careless third party, not attempted valid use of the service as that would generally be considered shooting yourself in the foot.

Incidentally, defending against a Denial of Service attack is typically a lot more feasible than helping a poorly resourced/implemented system to limp along under load.



BraveInca said:


> Lastly, even if it wasn't massively expensive to allow 100% of customers to withdraw at the same time, no sane bank would implement a system that could do that.



To follow that line of thinking, if they had branches on the street, would NR be entitled to lock the doors and close up shop for the day if the queue of customers built up to more than, say, 50?


----------



## huskerdu (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*

I can understand your anger askew70, but if you can find a bank with an online facility with the bandwidth that you  require, I suggest that you bank with them, I am not so naive as to think that I am going to find one. 

In answer to your last question, banks have been doing this for years. 
I am not excusing it, just realistic. In the days when all retail banking had to  be done in the branch, lots of branches would just close the doors early
because the queues inside were too long to be dealt with that day. 

Different technology, same service.


----------



## BraveInca (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> Oh, only about 15 years or so. Will that do, at all?
> 
> 
> 
> So, it is ludicrous to pay for a system that can accommodate even the majority (let alone all) of your customers at the same time? Someone better tell places like Tesco, for example, that they should reduce the number of cash registers in their stores, as Tesco seem to be under the impression that lack of resources in that area will result in their customers choosing to go elsewhere.


 

Wow, you're really missing the point. 

To follow your analogy, Tesco have, say, 1 million customers in Ireland. If there was news of impending food shortages and all one million customers decided that they needed to buy a month's food right now, would Tesco be able to cope?

Should they build capacity for 1 million customers in case this unlikely contingency should arise? 

No, of course not. No business does. They build capacity for _expected volumes_ plus reasonable contingency. This is what Google does, what Tesco does, what everyone does. If every user of the internet hit Google right now then the site would go down. 

If your expected peak capacity is .05%, then you might build in contingency to be able to handle peaks of 100 times that (i.e. 5%).

Anyway, my reason for posting was simply to point out that there is a reasonable explanation why their systems would be unresponsive right now. Not trying to get under anyone's skin


----------



## askew70 (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



huskerdu said:


> I can understand your anger askew70, but if you can find a bank with an online facility with the bandwidth that you  require, I suggest that you bank with them, I am not so naive as to think that I am going to find one.
> 
> In answer to your last question, banks have been doing this for years.
> I am not excusing it, just realistic. In the days when all retail banking had to  be done in the branch, lots of branches would just close the doors early
> ...



I'm not angry, what would be the point of that. However, I am disappointed that a bank which sells a service based entirely on the premise that your account is available online at any time of the day, can't deliver on this. 

The technology to provide a reliable and robust web-based service is not new, and Google is just one very well know example of that (and the load on the Google service is many many times greater than that on NR's web service). Unfortunately, as with many things, the true quality of the service only becomes apparent in times of quasi-crisis (i.e. in this case that manifests itself as heavy load on the server(s)), and the NR web service is proving itself to be extremely poor. They simply haven't invested enough in this core aspect of their service, be that an investment of time, money, and/or quality control.

Would the online service of other banks fare just as badly in the same kind of situation? Maybe. But the more important question is should customers of such a service be willing to accept poor quality of service and the answer to that must be no. A poor quality service is a poor quality service regardless of whether it is face to face in a branch, or on-line.

Unfortunately, the issues with NR's service seems to be tainting peoples' view of online banking services generally, if some of the post in this thread are anything to go by. That is a shame because NR's service, to me, falls short of even attaining the middle ground of the quality range and is therefore not a good yardstick by which to measure similar services by other parties.


----------



## askew70 (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



BraveInca said:


> If your expected peak capacity is .05%, then you might build in contingency to be able to handle peaks of 100 times that (i.e. 5%).
> 
> Anyway, my reason for posting was simply to point out that there is a reasonable explanation why their systems would be unresponsive right now. Not trying to get under anyone's skin



If you build a system capable of handling only 5% of your customers, yet continue to advertise to all existing and potential customers that this service will be available 24 hours a day, 7 days a week, then you should be penalised for false advertising.

And yes, there is a reasonable explanation for why their systems would be unresponsive right now - their system was simply not designed to handle the load. This happens with a lot of websites, but you don't expect it to happen with a website whose service is so valuable and important to so many people. 

To take another example, as you don't seem to like my Tesco analogy: If a plumber installed a toilet in your house that simply failed to operate after it reached a limit of 5 flushes in any one day, then I suspect you'd be a wee bit upset. You can choose to be selective about which services you expect quality from, but I prefer to be more consistent.


----------



## BraveInca (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> To take another example, as you don't seem to like my Tesco analogy: If a plumber installed a toilet in your house that simply failed to operate after it reached a limit of 5 flushes in any one day, then I suspect you'd be a wee bit upset. You can choose to be selective about which services you expect quality from, but I prefer to be more consistent.


 
I can have a crack at that one too  

If everyone in your house had a bad case of the runs and needed to use the toilet at exactly the same time, but they couldn't, would you blame the plumber? Would you pay for and maintain individual toilets for each person in case this contingency should arise? Most people don't because they are happy to have enough toilet capacity for normal demand.

Okay, enough!  I agree that its frustrating, but so far they haven't failed my service expectations. However, the situation continues for much longer then my opinion will certainly change. I'm willing to give them the benefit of the doubt for a couple of days in the current circumstances.


----------



## askew70 (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



BraveInca said:


> I can have a crack at that one too
> 
> If everyone in your house had a bad case of the runs and needed to use the toilet at exactly the same time, but they couldn't, would you blame the plumber? Would you pay for and maintain individual toilets for each person in case this contingency should arise? Most people don't because they are happy to have enough toilet capacity for normal demand.
> 
> Okay, enough!  I agree that its frustrating, but so far they haven't failed my service expectations. However, the situation continues for much longer then my opinion will certainly change. I'm willing to give them the benefit of the doubt for a couple of days in the current circumstances.



Service expectations should be based on what the service claims to offer. NR offer 24x7 access, and use this as a strong (and perhaps even the only) selling point for their service, but when tested they can't provide this. I am surprised that anyone considers this reasonable, at any time. It is not beyond the reach of a company with sizeable funds available to build a robust system to cater for hundreds of thousands, and perhaps millions, of customers, but it requires the commitment of the company offering that service, and NR's service has demonstrated their lack of commitment in that regards.

I managed to finally login a few minutes ago, for the first time in the last 42 hours. That, to me, falls very far short of a decent service.


----------



## Nemesis (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> Service expectations should be based on what the service claims to offer. NR offer 24x7 access, and use this as a strong (and perhaps even the only) selling point for their service, but when tested they can't provide this. I am surprised that anyone considers this reasonable, at any time. It is not beyond the reach of a company with sizeable funds available to build a robust system to cater for hundreds of thousands, and perhaps millions, of customers, but it requires the commitment of the company offering that service, and NR's service has demonstrated their lack of commitment in that regards.
> 
> I managed to finally login a few minutes ago, for the first time in the last 42 hours. That, to me, falls very far short of a decent service.



Building the kind of capacity to cope with an extremely rare event such as this would be a ludicrous waste of money. It's unfair to expect Northern Rock to have such capacity available, and it isn't the least bit surprising that the system is next to impossible to access in the midst of this present panic. I have always felt though that the NR online system was rather sluggish under normal conditions and certainly this is something that could have been improved. But I regard that as a separate issue from what's going on right now. Even if normal service was more responsive, it's hard to see how it would make much difference under the kind of demand it's facing at present.

This is getting annoying now. I intended to withdraw some money for the week after next and it looks like I'm gonna have to waste time and endure frustration battling with the online system sooner rather than later if I'm gonna be sure the money's out when I want it. It really is extraordinary the level of panic this has produced. I can understand if all someone's life savings is in NR, or they're above the protection limits and want to move some of the money for peace of mind, but there are definitely people overreacting to all of this and making matters worse for everyone else.


----------



## z109 (15 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



GeneralZod said:


> I'd like to know if the design of the system allows the same number of successful transactions to be completed when it's under high load. If it's poorly designed it could actually be processing fewer transactions than when it isn't under high load. This could account for the long periods during peak load when nobody reported getting logged in and completing a transaction. This explanation doesn't need a conspiracy theory to account for the reduced transaction processing rate.
> 
> As Askew70 said a well designed system like google's (where programmers are given the time and respect necessary to do the job right) reacts well under load by not locking out all users. Any well designed system also uses dimensioning rules for expected processing requirement rates so that costs may be balanced with benefits.


On-Line Transaction Processing (OLTP) is a complex business. As the good general points out, many systems operate poorly when stressed. As an example of this with NR, I found last night that I was getting timeouts (the service unavailable message) after I had signed into my account. I found that (using firefox), if I reloaded the frame, I could get back to where I was (i.e. still signed in) as I posted earlier.

I would agree that, in particular, their security servers are undersized and look like serious bottlenecks at high volumes. It also appears that they are using multiple security servers that are not correctly scaled in size (if you get past the first one, you should be able to get past the second one easily and you should then be able to conduct transactions on your account without further timeouts). In other words, it looks like there is no capacity lock on the system - it doesn't shutdown access to new traffic when volumes are high, it just craps on everyone (whether trying to enter the system or already on it).

I don't believe that this is either intentional in design or a response to the current situation, I think the system is just badly designed from the outset.

Google is not a comparable operation, IMO, as it does not require security authentication. A better example would be if amazon had a massive sale on. I doubt if it would cope any better. As other posters have pointed out, ticketmaster have struggled even when they know there is exceptional demand coming for a particular event. Contrast this with the airline industry who have been running OLTP systems for the last 40 odd years and see that difference when Ryanair have a sale. Experience counts and the banks are novices at this.


----------



## askew70 (16 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Nemesis said:


> Building the kind of capacity to cope with an extremely rare event such as this would be a ludicrous waste of money. It's unfair to expect Northern Rock to have such capacity available, and it isn't the least bit surprising that the system is next to impossible to access in the midst of this present panic. I have always felt though that the NR online system was rather sluggish under normal conditions and certainly this is something that could have been improved. But I regard that as a separate issue from what's going on right now. Even if normal service was more responsive, it's hard to see how it would make much difference under the kind of demand it's facing at present.



You are making the same large assumptions that several others seem to be making. 

For one thing you are assuming that building a system to cater for heavy load is extremely expensive. It can be, but you really only start to hit expensive solutions once you have exhausted the potential of what you already have. From what you say, you seem to be assuming that NR's system is bursting at the seams because the existing hardware and software is incapable of coping, and therefore more investment is required. Another possibility however is that the hardware and software they have is not the issue (yet) because the basic architecture of the solution is at fault. I have seen at least one appallingly designed web-based solution implemented by a financial institution where they threw money at the hardware and software, but the basic design was always going to result in failure of the service before any single machine even broke a sweat. In that case they didn't need more money to make a better solution, just more thought.

For another thing, you are assuming that massive numbers of NR customers are hitting the web service at the same time. That may be the case, but none of us know that for sure. It could well be that the system is so bad that it flounders even if 20 people access it simultaneously. In any case, any system which is well designed should "fail" gracefully, rather than providing the illusion of working until your connection eventually times out after several minutes.

Ultimately, as with most things, customers will end up with the service they deserve. If people are willing to accept an online service of the standard of NR's, without complaint, then there is no reason for any NR competitors to go to the trouble of building a better system. If enough people deem the recent performance of the NR online service as perfectly reasonable, then we are all doomed to crap online banking for some time to come.


----------



## askew70 (16 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



yoganmahew said:


> Google is not a comparable operation, IMO, as it does not require security authentication. A better example would be if amazon had a massive sale on. I doubt if it would cope any better. As other posters have pointed out, ticketmaster have struggled even when they know there is exceptional demand coming for a particular event. Contrast this with the airline industry who have been running OLTP systems for the last 40 odd years and see that difference when Ryanair have a sale. Experience counts and the banks are novices at this.



The process of accessing Google is certainly different in that it doesn't involve authentication (although Google Mail is a heavily used service that I have known to be problematic on only a handful of occasions and even then only for a minute or two at most). Having said that, the Google service does tailor itself based on the results of some checks against your IP address, for example, so it does carry out some processing before it presents you with the search page. Also, the Google service does involve running searches on massive databases, and authentication can be seen as essentially just a database search for a user's details (followed by a comparison against what the user supplied).

However, implementation details aside, the main similarity between Google and NR's online service is that the success/survival of one arm of each company depends entirely on providing an online service that will entice customers in rather than scare them away. Google practically built their service from scratch to produce a very impressive solution, whereas NR seem to have thrown a few things together without much consideration for how it would fare under pressure (which many companies are equally guilty of, of course). Hopefully NR, and other banks, will learn from this recent clear demonstration of this mistaken approach. By comparison, Google have always known that their service being unavailable to a lot of people for several days in a row will lose them customers permanently, so they specifically designed their system to minimise this possibility, and very effectively too.


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



GeneralZod said:


> Has anyone that made a withdrawal request on Saturday actually seen the money leave their account yet? My transactions are still showing up as pending. They are normal "3 - 4 working day" withdrawals but I'd expect to see them processed by now. Normally my money spends a few days in cyberspace/NR's account on the way out. Do they manually process all of these transactions?



I made a withdrawal on Saturday, which today shows as having been processed. Mind you, the withdrawal still appears in my list of pending transactions, but as I have no faith left in their online service (as distinct from no faith left in the bank itself), this doesn't unduly worry me.

In fact, due to the persistent difficulties that I have had in accessing my online account since Thursday night last (and some people report problems back as far as the previous weekend), I decided to withdraw the rest of my money today and close my account. That didn't go smoothly either (having completed the withdrawal process I was thrown right back to step 1 again and had to repeat the withdrawal), but it now shows as a pending transaction.

Despite the views of some people here that this level of online service is only to be expected, I completely disagree. This situation would obviously push anyone's online service pretty hard, but NR's service has spent the last few days compeletely unusable, for the most part. I am repeating myself here, but this technology is not rocket science, it is well understood and capable of being used to provide a very good service for those companies for whom quality of service is important. Based on the shoddy behaviour of NR's authentication process, and the fact that the online service itself is simply unreachable most of the time for several days now, this particular service has shown itself to be sub-standard. 

I have drawn the comparison to Google's search service before and I will repeat that again too - if the Google service was unusable for an hour or two (let alone several days), people would be annoyed and would flock to an alternative searcing service, and would possibly complain to others later that Google's quality had gone downhill. And that is a free service to most people who use it. Why should we be so forgiving of the online service of a bank which we fund with our own cash and which has potentially serious implications for us when it is unavailable?

Just to be clear, I view the bank service and its online service as separate entities. My decision to close my account today is motivated more by my dissatisfaction with the online service than anything else, as the new assurances from the BoE seem to me to be as good as anyone could have expected. I just don't wish to continue to fund, or be a frustrated user of, an unsatisfactory online service when I know it can be done much better.


----------



## z109 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> Despite the views of some people here that this level of online service is only to be expected, I completely disagree. This situation would obviously push anyone's online service pretty hard, but NR's service has spent the last few days compeletely unusable, for the most part. I am repeating myself here, but this technology is not rocket science, it is well understood and capable of being used to provide a very good service for those companies for whom quality of service is important. Based on the shoddy behaviour of NR's authentication process, and the fact that the online service itself is simply unreachable most of the time for several days now, this particular service has shown itself to be sub-standard.
> 
> I have drawn the comparison to Google's search service before and I will repeat that again too - if the Google service was unusable for an hour or two (let alone several days), people would be annoyed and would flock to an alternative searcing service, and would possibly complain to others later that Google's quality had gone downhill. And that is a free service to most people who use it. Why should we be so forgiving of the online service of a bank which we fund with our own cash and which has potentially serious implications for us when it is unavailable?


Askew70 - having disagreed with you initially about the comparison with google/the difficulties of OLTP, I now find myself agreeing with you. This is day 5 and the service has not significantly improved. I would expect any reasonably IT department to have made improvements by now. As you say, the lack of responsiveness to the problems (or quite probably the badly constructed system that makes it difficult for them to respond) diminishes confidence in their ability to conduct day-to-day internet operations.

No doubt businesses all over the world are reviewing their models and peak capacity expectations, so maybe some good will come of it?


----------



## Afuera (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> I am repeating myself here, but this technology is not rocket science, it is well understood and capable of being used to provide a very good service for those companies for whom quality of service is important. Based on the shoddy behaviour of NR's authentication process, and the fact that the online service itself is simply unreachable most of the time for several days now, this particular service has shown itself to be sub-standard.


I don't think this is being fair on NR. No secure webservice could maintain its QoS given the flash crowd scenario that NR has experienced these last days. The compute time required to negotiate SSL handshakes for all of NR's customers runs into the order of 72 hours alone. Not to mention that the computation time needed for the DB back end would be several times that. The cost of over provisioning for that level of a spike makes no economic sense.


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Afuera said:


> I don't think this is being fair on NR. No secure webservice could maintain its QoS given the flash crowd scenario that NR has experienced these last days. The compute time required to negotiate SSL handshakes for all of NR's customers runs into the order of 72 hours alone. Not to mention that the computation time needed for the DB back end would be several times that. The cost of over provisioning for that level of a spike makes no economic sense.



The expense in establishing an SSL connection is on the initial connection (verification of security certificate, negotiation of encryption keys, etc.). Once you have an SSL connection in place, which is before you are prompted for your account/login ID, then subsequent connections are pretty "cheap" in terms of server resources. At that stage, database access becomes the next expensive operation, and given that database technology has existed for a long time that is capable of handling load very well, there is no reason for this to be an insurmountable hurdle by any means either.

The nature of the failure of the NR web service right now is probably compounding their problems. Once you get an SSL connection established, to proceed with the authentication stage, more often than not authentication fails. If you subsequently quit that window, you lose your SSL connection and have to establish a new one from scratch, leading to greater load on the server. The success of people logging in through refreshing the existing window, with SSL connection already in place, might be down to the fact that these re-fresh connections don't add to the "SSL load" on the server.

People have referred here to the likes of Amazon and Ticketmaster as examples of services that might well fare badly in this situation too. I have had poor experiences with the Ticketmaster site, and am no fan of it, but I haven't experienced it being unusable for days at a time. I have never experienced significant problems with any of the other heavily used online services that I make use us (Google, Yahoo, Amazon, eBay, etc.). The difference is, I believe, that those companies give adequate importance to the quality and reliability of their online services, whereas NR clearly has not done so. 

I can only hope that this issue will make other companies sit up and take notice that their online service is important. There are many, many, companies that have an online presence which they seem to take no interest in - symptoms of this range from lack of SSL connections, to sensitive information (such as credit card info) being transferred in the background via unprotected e-mail. In the worst cases, some of those services seem to me to verge on the criminally negligent, and will likely remain haphazardly implemented until there is pressure to improve. While I have concerns about the NR authentication process, it is the poor (lack of) availability of the service that is currently the biggest issue, and there too companies will feel no pressure to improve that aspect of their service if people deem current performance acceptable.


----------



## Afuera (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> At that stage, database access becomes the next expensive operation, and given that database technology has existed for a long time that is capable of handling load very well, there is no reason for this to be an insurmountable hurdle by any means either.


Nothing is insurmountable but the cost of provisioning for a system that could handle so many transactions at the same time just does not make sense economically. I'd rather see my bank put its funds elsewhere than have it wasted on resources for crazy "what-if" scenarios.



askew70 said:


> The nature of the failure of the NR web service right now is probably compounding their problems. Once you get an SSL connection established, to proceed with the authentication stage, more often than not authentication fails. If you subsequently quit that window, you lose your SSL connection and have to establish a new one from scratch, leading to greater load on the server. The success of people logging in through refreshing the existing window, with SSL connection already in place, might be down to the fact that these re-fresh connections don't add to the "SSL load" on the server.


Of course there is room for improvement on QoS and reusing SSL connections is a technique that can speed things up. This could compromise security somewhat which means it's use in banking web services is limited. Higher security obviously has to take preference over higher quality of service in the banking world.



askew70 said:


> People have referred here to the likes of Amazon and Ticketmaster as examples of services that might well fare badly in this situation too. I have had poor experiences with the Ticketmaster site, and am no fan of it, but I haven't experienced it being unusable for days at a time. I have never experienced significant problems with any of the other heavily used online services that I make use us (Google, Yahoo, Amazon, eBay, etc.). The difference is, I believe, that those companies give adequate importance to the quality and reliability of their online services, whereas NR clearly has not done so.


I don't think that the likes of ticketmaster or amazon have yet encountered a situation where up to 1.5 million customers were all trying to log in and perform transactional operations at the exact same time, and over such an extended period. I would imagine they would experience a lower QoS too given the circumstances.


----------



## MugsGame (18 Sep 2007)

The Google comparison is invalid. Google's income scales with the number of simultaneous users they can serve, so they have a financial incentive to ensure they can serve as many simultaneous users as possible. Banks do not have the same financial incentive, which makes it uneconomical to overprovision for the worst case scenario. Of course NR could provide an online service capable of handling this load, but very few customers would accept the costs of such a "gold-plated" service. If anything (as pointed out earlier in the main thread) there is a financial disincentive to build systems that facilitate a run on a bank.

Additionally, banking is transactional, and search is not. Google can copy their read-only search database to scale indefinitely (often with inconsistent distributed copies of the database). Banks need to store a consistent record of transactions on an account. Google's approach to scaling more transactional services (such as Google Mail) has been far more cautious, and less reliable.

I've never been particularly impressed by NR's online banking, but it does the job.  The main fault with NR is that the service didn't degrade gracefully, though they seem to be taking steps to address that. I think it comes back to cost, which is reflected in interest rates. Rabo's online service has far more bells and whistles, but a lower interest rate overall. Whereas First Active's new online savings account has a higher rate, but by all accounts worse service and minimal online functionality.


----------



## jrewing (18 Sep 2007)

In my experience as someone who uses 4 different bank websites, the _NR_ website has always been of poorer quality (slower, more prone to crashes/freezing) than other banks.


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Afuera said:


> Nothing is insurmountable but the cost of provisioning for a system that could handle so many transactions at the same time just does not make sense economically. I'd rather see my bank put its funds elsewhere than have it wasted on resources for crazy "what-if" scenarios.



I'd rather see my bank provide the service that they advertise. One of the main selling points of online banking is that you have access to your account 24 hours a day, 7 days a week. I have yet to see a proviso on that along the lines of "unless more than X number of you all want access to your accounts at the same time, in which case you may have to wait for a few days". 

The NR online service seems to be a service that has had very little in the way of quality control, and perhaps (but not necessarily) funding, thrown at it. The current "what-if" scenario, as you describe it, is a mad panic by worried customers. We have no way of knowing whether the service would suffer similar problems if, for example, 20 people were to access it at the same instant at any other time. You may choose to believe that the system is currently creaking and groaning because over a million people are accessing it simultaneously, but in fact the only people who really know what load the system is capable of handling are those that implemented the service (you'd hope) - I am not convinced that it is capable of handling even a small percentage of customers simultaneously.



Afuera said:


> Of course there is room for improvement on QoS and reusing SSL connections is a technique that can speed things up. This could compromise security somewhat which means it's use in banking web services is limited. Higher security obviously has to take preference over higher quality of service in the banking world.



I have written in previous posts in this thread of the inherent security weakness in how the authentication phase of NR's service is behaving. They have clearly, and demonstrably, not given preference to higher security over higher quality of service.



Afuera said:


> I don't think that the likes of ticketmaster or amazon have yet encountered a situation where up to 1.5 million customers were all trying to log in and perform transactional operations at the exact same time, and over such an extended period. I would imagine they would experience a lower QoS too given the circumstances.



I haven't checked for any statistics on the number of Amazon customers, and particularly the number of simultatneous customers, but I would not rule out the possibility that it does handle 1.5 million customers well. At the very least, I would be shocked if the system effectively fell apart in such circumstances, as NR's online service has. 

I have already stated that I am not a fan of Ticketmaster's site, but it does handle tens of thousands of credit card transactions, typically in a short few hours. Admittedly, it can be an painful experience, but it works because they are aware that if their online system was unusable for days at a time then they are likely to be out of business.

Whatever about the others, Google's service does handle a ridiculously high load, and if you ever use their service when was the last time that you ever experienced anything worse than a slight delay in search results being returned? I am risking turning this into an ad for Google though, which is not my intention, as I am sure that Yahoo's search engine, and those of others, are more than capable of handling far far greater load than that which has NR's service falling flat on its face. 

Again, the key to this difference in service quality is that some companies treat their online service as core to their business, and others don't. When a bank is selling an online account, you would expect them to treat their online service as core to that aspect of their business, but NR clearly did not.


----------



## Afuera (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> I am not convinced that it is capable of handling even a small percentage of customers simultaneously.


In normal circumstances only a small percentage of its customers will require to use its services simultaneously. Your demands are not realistic and I doubt you will be able to find any bank out there to guarantee the service you expect.



askew70 said:


> I have written in previous posts in this thread of the inherent security weakness in how the authentication phase of NR's service is behaving. They have clearly, and demonstrably, not given preference to higher security over higher quality of service.


Reusing an SSL connection for the same person does not constitute a weakness in security. Reusing it in other ways does.




askew70 said:


> I haven't checked for any statistics on the number of Amazon customers, and particularly the number of simultatneous customers, but I would not rule out the possibility that it does handle 1.5 million customers well. At the very least, I would be shocked if the system effectively fell apart in such circumstances, as NR's online service has.
> 
> I have already stated that I am not a fan of Ticketmaster's site, but it does handle tens of thousands of credit card transactions, typically in a short few hours. Admittedly, it can be an painful experience, but it works because they are aware that if their online system was unusable for days at a time then they are likely to be out of business.
> 
> ...


Apples and oranges.


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Afuera said:


> In normal circumstances only a small percentage of its customers will require to use its services simultaneously. Your demands are not realistic and I doubt you will be able to find any bank out there to guarantee the service you expect.



You choose to be happy with the level of service you are currently receiving, which implies that you expect no bank's online service to ever be able to cope with whatever simultaneous number of users are hitting NR's service for the last several days. And you are making this judgment without knowing what that number of users is, at which the service ceases to function at any kind of a reasonable level. You are defining an acceptable level of service based on little or no quantifiable data, but simply on the hope that NR have done everything reasonable within their power to provide a robust service. Your choice, but certainly not mine.



Afuera said:


> Reusing an SSL connection for the same person does not constitute a weakness in security. Reusing it in other ways does.



The security weakness that I referred to was not re-use of an SSL connection (maintaining an SSL connection is not "re-use" by the way, in the conventional sense of that word in an IT environment - the SSL connection remains "in use" until such time as it is explicitly torn down or times out. It is a subtle difference, but an important one if talking about security).  

The weakness I was referring to is the behaviour of the authentication step which prompts you to enter a "random" pair of characters from your password. If you fail to login because of a problem with the service (and perhaps under other circumstances too), the next time you attempt a login you are asked to supply the very same supposedly random characters. As the particular characters being asked for are the same (i.e. not random) each time (until you have successfully logged in), then you effectively have a 2-character password for as long as the service won't successfully log you in.  That pretty much makes a nonsense of the otherwise very sensible approach of having you provide some randomised portion of your password every time you attempt to login, leading to weaker security.



Afuera said:


> Apples and oranges.



Somehow I find that unconvincing as an argument. Maybe if you could elaborate you might convince me.


----------



## Afuera (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> You are defining an acceptable level of service based on little or no quantifiable data, but simply on the hope that NR have done everything reasonable within their power to provide a robust service. Your choice, but certainly not mine.


I never made any definition on what an acceptable SLA should be. Your expectation of an SLA that is able to satisfy all NR's customers simultaneously is absurd though.



askew70 said:


> The weakness I was referring to is the behaviour of the authentication step which prompts you to enter a "random" pair of characters from your password. If you fail to login because of a problem with the service (and perhaps under other circumstances too), the next time you attempt a login you are asked to supply the very same supposedly random characters. As the particular characters being asked for are the same (i.e. not random) each time (until you have successfully logged in), then you effectively have a 2-character password for as long as the service won't successfully log you in.  That pretty much makes a nonsense of the otherwise very sensible approach of having you provide some randomised portion of your password every time you attempt to login, leading to weaker security.


I don't know the ins and outs of the security process that NR have implemented but I'm presuming that after a certain number of attempts it blocks the account completely. If it does this then it would still be in agreement with current best practises in security. If not, then you have a point as accounts could be easily bruteforced.

The reason that this technique you refer to as "nonsense" is considered a best practise is that it's possible that a customer may have been observed entering their account on one occasion. An attacker could keep reloading the login page until the questions that they observed randomly appear. You might need to look into upskilling your security knowledge as it doesn't sound too hot right now.




askew70 said:


> Somehow I find that unconvincing as an argument. Maybe if you could elaborate you might convince me.


To compare the technical capacity of an entity that has a side business in selling compute power (i.e. Amazons Elastic Compute Cloud) with another entity that is only in the business of banking is neither here nor there. If you can find an example of a bank that is able to deal with all it's customers logging in to make transactions at the same time then you might be on to something.


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Afuera said:


> I never made any definition on what an acceptable SLA should be. Your expectation of an SLA that is able to satisfy all NR's customers simultaneously is absurd though.



My expectation is that the service either functions as advertised, or it degrades in a graceful way that is both meaningful and useful. Accessing the service and having to wait up to several minutes, watching a supposedly active login session, to find out whether or not your login succeeded is not "graceful", it's just a wing-and-a-prayer approach to providing a service. You seem happy with it though, so clearly there is a market for this approach.



Afuera said:


> I don't know the ins and outs of the security process that NR have implemented but I'm presuming that after a certain number of attempts it blocks the account completely. If it does this then it would still be in agreement with current best practises in security. If not, then you have a point as accounts could be easily bruteforced.
> 
> The reason that this technique you refer to as "nonsense" is considered a best practise is that it's possible that a customer may have been observed entering their account on one occasion. An attacker could keep reloading the login page until the questions that they observed randomly appear. You might need to look into upskilling your security knowledge as it doesn't sound too hot right now.



Hmm, you say that you don't know the ins and outs of the authentication process in use by NR (which is a well defined approach and has been in use for years, by the way), yet you describe it as "best practice" and suggest that I "upskill" my security knowledge as I seem not to understand it. You should probably have thought that through before you wrote it - telling someone that they are talking crap tends to lose it's effectiveness when you yourself admit that you don't know what you are talking about.

I have described this issue already, in some posts from the original thread here:




...but I'll try again if you feel up to the task of trying to understand what in reality is a very simple concept.

Right, basically, your password is only useful as long as some malicious person, lets say JoeBloggs, doesn't know it. If you are always prompted for the same password, then JoeBloggs just needs to know that one password to login as you. JoeBloggs might learn your password by looking over your shoulder as you type or, more prevalent these days, by managing to install some software on your machine to record what you type at the keyboard in response to a prompt on your screen. He might also try to brute force his way in by guessing every possible combination of characters - most systems, NR's included, provide some level of protection against this by putting a limit, usually 3, of the number of wrong passwords you can enter before you are locked out of your account.

It would be better if you were prompted for a different password every time, 'cos now JoeBloggs needs to know all of your passwords to ensure that he can get in. If you have 10 passwords, and are asked for any one of them randomly, then if JoeBloggs has only 4 of those passwords he may expect to successfully login as you only 40% of the time - that is still a lot of the time but it is better than the worst case scenario of 100% of the time. 

However, if you have 10 passwords, but the service only ever prompts you for passwords 5 and 7 each time you try to login, then if JoeBloggs has those two passwords he will now get in 100% of the time = a decrease in security.

With NR's service, you have a single password made up of multiple letters, which is just a variation on the above theme - basically, with the NR system your effective password is 2 characters long every time. That is a very short password and by its nature very weak, but the strength of its security lies in the fact that those two characters are pseudo-random (i.e. they are random from within the limited set of characters that make up your full actual password). If the system keeps asking you for the same two characters on successive occasions, then the random element of this security mechanism is lost = the same 2-letter password each time = a decrease in security.

Until something better is devised, about the best approach to authentication right now is for the user to have a hardware token that generates characters which are much closer to being truly random (they are not truly random as the server side must also be capable of generating/predicting the same "random" numbers). The display on the hardware token changes regularly (maybe every minute or so, or at the instigation of the user), and the user supplies what this hardware token displays plus some piece of information known only to the user (essentially a static password). In that scenario, what the token displays is not something that JoeBloggs can reliably predict = greater security. Rabodirect use this type of solution, for example. The solution that NR use is reasonable when you are trying to keep costs down, but it is usually a choice based purely on cost. I have no problem with the NR approach, but only when it is implemented properly, which isn't the case here.



Afuera said:


> To compare the technical capacity of an entity that has a side business in selling compute power (i.e. Amazons Elastic Compute Cloud) with another entity that is only in the business of banking is neither here nor there. If you can find an example of a bank that is able to deal with all it's customers logging in to make transactions at the same time then you might be on to something.



I don't have to find a perfect service to recognise one that provides a service that is very far from perfect. My own level of acceptable service lies quite a bit below perfect, because I know of at least some of the issues that make a perfect service difficult, if not impossible, to achieve. NR's online service falls very far short of even my relatively modest expectations.

Worse still, the poor performance of NR's online service has served to cause even more concern and panic amongst people unable to access their savings, leading to greater demands on the service itself as more people got caught up in the rush, and so it spiralled ever downwards. It is a very effective service is self-destruction is one of its goals.


----------



## Afuera (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



askew70 said:


> Hmm, you say that you don't know the ins and outs of the authentication process in use by NR (which is a well defined approach and has been in use for years, by the way), yet you describe it as "best practice" and suggest that I "upskill" my security knowledge as I seem not to understand it. You should probably have thought that through before you wrote it - telling someone that they are talking crap tends to lose it's effectiveness when you yourself admit that you don't know what you are talking about.


Maybe you should actually read what I wrote properly instead of mincing your words.

If you are constantly only getting asked for the same 2 passwords, even with a new clean session, and on different terminals, than that would be insecure. Frankly though, your writing is so bad I'm not actually sure if that describes the situation you were encountering or whether your browser is simply badly set up. Why don't you go and take it to Northern Rock as maybe they can understand your ramblings and conjecture better than I?


----------



## askew70 (18 Sep 2007)

*Re: Northern Rock bailed out by Bank of England*



Afuera said:


> Maybe you should actually read what I wrote properly instead of mincing your words.
> 
> If you are constantly only getting asked for the same 2 passwords, even with a new clean session, and on different terminals, than that would be insecure. Frankly though, your writing is so bad I'm not actually sure if that describes the situation you were encountering or whether your browser is simply badly set up. Why don't you go and take it to Northern Rock as maybe they can understand your ramblings and conjecture better than I?



And with that vitriolic response dies any hope of reasonable debate. Oh well.

I tried taking it to Northern Rock, as you suggest, but I couldn't get through. Maybe there is something up with their system...


----------



## jpd (18 Sep 2007)

why stop at banks ? Telephone service providers can't cope with a rush of customers eithder - try sending a text message from Croke Park when Dublin are playing!


----------



## MugsGame (18 Sep 2007)

On a day like last Friday, the scene in the IT department at Northern Rock would have been manic.


----------



## gearoid (18 Sep 2007)

Bandwidth is not the only issue.
I can now login but NR says I don't even have an account with them. I closed it on saturday but I should still be able to see the progress of the EFT. This is appalling. The integrity of their account enquiry processes is compromised. This is a core part of their system.


----------

