Posts by stevej.cbr

From hard drive to over-heard drive: Boffins convert spinning rust into eavesdropping mic

Re: Shouting at your discs

and reported by El Reg in 2009

https://www.theregister.co.uk/2009/01/05/shouty_sun_engineer/

HPE ignored SAN failure warnings at Australian Taxation Office, had no recovery plan

Something Not Quite Right with ATO report

EDS won the initial ATO outsourcing contract in 1996/7 and was bought by HP, a hardware vendor, in 2008 to become HPE and this years' merger with CSC created yet another entity, DXC.

Now we have a hardware vendor acting as an I.T. services provider, which creates confusion over their preferences for new hardware.

1. I've never seen one byte of data stored on a SAN, they are NETWORKS only, so why do the ATO / HPE claim their "SAN" stores data and has drives?

Either the HPE & ATO don't know the difference between a Network and Storage Array or Device (!), or they are deliberately mis-using terms for a reason. This is not a rookie mistake.

2. Sure, cables fail, but all your cables don't all suddenly need replacing after 6-12 months, nor do they suddenly become "stressed" if properly installed and maintained.

From the scale of the task, we can infer large numbers of cables, perhaps all or most, were replaced.

Why is the ATO (or HPE/DXC) insisting on replacing the 3PAR hardware and a full forensic tear-down and investigation, including of all the "fibre optic cables", if there has been no unusual event or physical damage?

That action wouldn't be justified on technical grounds alone, but would result from a serious legal dispute between client and vendor over liability. EDS was know for its

If the root cause suspected is physical damage, whoever ordered the action that caused the damage will be responsible.

Yet nothing explaining this unprecedented forensic examination is contained in the ATO report.

3. If the ATO is talking "Cloud", it's quite bizarre.

The 3PAR 20850 device named is a low-latency, high-performance All-Flash Storage Array that must be locally connected to hosts to be usable.

The ATO already extensively uses VMware to manage its workload and move executing instances between hosts, even Datacentres.

They already run a "Cloud", so is this code for outsourcing these operations to another supplier - which would mean breaking the existing contract with years to run.

I've not seen the EDS / HPE / DXC contracts, but I'd expect breaking them would cost the ATO dearly in time and money.

The ATO refers to the 3PAR Storage Array as a "20850 SAN" - if you look up the device, it's an All-Flash Array, not a Storage Area Network.

This mis-use of the term "SAN" is consistent - they report they replaced "an EMC SAN" with the 3PAR.

EMC have always just made Storage Arrays, never switches and Fibre Channel network gear.

Mentioned in passing, as a label in "Figure 1", is another device, "XP7". There is a HP storage array with that product designator that provides PB using HDD's, not Flash.

How does this relate to the system, given it gets mentioned just once? Why is the XP7 included at all in the diagram if not part of the functional system?

What devices & cables are the SAN (Network) and what are the Storage Devices?

Why do the ATO & HPE conflate these specific terms? "Ignorance" is as daming as "Obscuring".

What about HBA's, switches, routers & "directors" that are in the network? We hear nothing of them.

Are they using Fibre Channel (16Gbps?) or FCoE with 10Gbps ethernet interfaces (or faster)?

If it's FCoE, are they using HP or CISCO as their fabric? Or someone else entirely?

Where were the fibre problems reported by SNMP between?

Cables aren't active devices, they don't log errors themselves, only the devices that attach to them can detect & report errors.

We hear "cable", but is that a permanent cable between patch panels or patch lead(s)? (panel to panel, panel to Array, panel to Host)

So what devices were logging the SNMP errors over the 6 months prior to the first outage?

Where do they sit within the environment?

If we had 3PAR HBA's logging internal errors to its drives over local, non-SAN links, that's very different to a host connecting via the Network to the 3PAR controller.

The ATO reports takes special care to mention "data paths", disk drives and "SAS" (Serial Attached SCSI), but never cares to provide any sort of explanation of their importance/relevance or diagram of connections.

No, that's not restricted information in a simplified document. No reason to suppress that level of detail - they've disclosed other very specific technical details.

Over decades, I've never seen an optically connected drive, the HDD & SSD's I've seen have only ever had _copper_ connectors. SAS is electrically the same as SATA, but allows dual-ports and daisy-chains.

The "state of the art" is fibre connections between backplanes, into which the devices are plugged with copper connectors.

That'd make sense for either SSD or HDD drives in either 3PAR 20850 or XP7.

It's where you'd expect "data paths" to run: from Array Controllers to drives/shelves. That makes these "stressed" cables internal to the 3PAR Array, not part of the Network.

Hosts connect to the SAN via at least dual connections, while on the other side, the Array controller has multiple connections to the SAN router / director for performance & reliability.

Were the fibre cables that caused errors on "data paths" internal to the 3PAR Flash Storage Array or within the SAN (network)? HPE & the ATO keeps that obscured.

The whole point of the 3PAR, in fact any Array, is to hide the individual devices from the hosts and create virtual error-free devices of any size.

With Network attached Storage Arrays, there aren't any direct "data paths" from a host to a drive - where the errors described occurred.

We suck at backups. So let's not have a single point of failure any more

Nightmare Scenario happened in 2011 at Oz service provider / DNS registrar

Distribute.IT, June, 2011

Backups were on-line on RAID. Hacker with Admin privs (presumed to be insider, no logs left), turned off backups and waited for the cycle to delete all data. Then zero'd live disks. Nobody ever charged.

http://www.smh.com.au/technology/security/4800-aussie-sites-evaporate-after-hack-20110621-1gd1h.html

Sell the NBN not long after rollout ends says Infrastructure Australia

Re: Labor's NBN: headlines with zero substance

> Today: 1Gbps has been available wholesale since Dec 2014, but not one RSP will offer it as a retail plan. 79% on fibre connected at 25Mbp or slower and 16% connected at 100Mbps (down 3% in 12 months)

Retailers, not NBN Co, set pricing and select the default plan offered to customers.

That explains the initial high take-up of 100/40Mbps and the latest 3% contraction.

Thanks for making this point for me: its essential for understand the _Economics_ of Fibre vs Copper.

What's the difference in cost to NBN between supplying 12/1, 100/40 and 1000/400 services?

Almost NIL.

They may need to bring forward planned expenditure on the internal Transit network if demand for higher access rates runs ahead of their plans, but that's more than offset by the increase in profits being generated. [Upgrading links is changing out the GBIC for a faster one, or adding another WDM colour, possibly putting a spare fibre into service. Fast, simple and very low cost.]

Essentially, there's ZERO input cost difference for NBN for all services, any additional charges for differentiated services are pure PROFIT.

[I'll use 2013 pricing, I haven't bothered learned the new price sheet.]

NBN Co could've done what ISP's did with ADSL and offered everyone the maximum speed for the same price. It costs them NOTHING extra and might even get them a little more CVC traffic.

But there are two massive problems with that approach:

- as previous note, it places an intolerable backhaul cost on the RSP/ISP's, esp with 121 PoI's.

- it's Economically naive. The company can make _much_ higher profits with tiered pricing. [The 2012/3 Business Plan did make AVC pricing for 12/1 to 100/40 identical after 10 or so years.]

The concept is "Consumer Surplus", when the _same_ product (or nearly same) is sold by the Producer for multiple prices.

Customers with a willingness to pay more for the same service can select a higher price and get a 'premium' version of the product. Sellers make their money on the 'premium' sales.

The seller doesn't "leave money on the table".

[Google Fiber is only offering three services. 'Entry level', 1Gbps Internet and 1Gbps Net+Cable-TV. They're operating nearly on a cost-recovery basis, not trying to maximise profits or pay off expensive loans. If a low-cost player like this shows up in Australia, they will "disrupt the market" in a way that will surprise most.]

Do you recall the first ADSL-1 offerings from Telstra?

They did exactly this, 3-4 offerings up to 1.5Mbps.

Its Good Business, both offering steep discounts to those that prefer 'cheap' and extracting high profits from those that are willing to pay, for whom that 'top' service provides that much value.

Quigley et al priced (2013) 12/1Mbps at $23, 100/4-Mbps at $38 and 1000/400Mbps ~$150,

although they ALL cost NBN Co _exactly_ the same to deliver.

If they made $2 from the 12/1Mbps service, they were making $17 from each 100/40 Mbps service.

Profit wise, they increased Earnings very quickly (the thing that good businesses focus on), if more than 6% customers selected the 'top' plan. [again, NBN is the wholesaler, the retailers aren't forced to pass on savings to consumers.]

Something buried in the Ergas/Vertigan report is the Price Elasticity of Demand.

At 1Gbps, it "-60" (negative sixty). I can't recall Elasticity for 100/40, maybe -20.

The Elasticity is the changes in sales made or lost for every $1 _change_ in the price, either up or down. Normally sellers _raise_ prices, so highly elastic markets are very bad for them.

But high Elasticity also works to your advantage when you can, like in Tech & Telecoms, drop prices over time.

By dropping the price by $1, total Revenue is increased $60 (1000/400) or $20 (100/40).

That's a 59-times and 19-times increase in PROFIT.

When the input costs of the services are identical, this a way to dial-up profits on demand.

As in previous note, NBN Co is a wholesaler, not retailer.

While they can charge low prices to Retailers, they cannot affect what the Retailers charge.

If Telstra and Optus don't pass on the discounts, NBN Co won't realise the revenue.

This is another Economics effect that savvy Retailers, especially in the Tech Industry (think Apple) use to drive PROFITS into the stratosphere, and at the same time, be applauded and thanked by their customers. A win-win for everyone: cheaper prices _and_ higher profits for shareholders.

By passing on _part_ of the cost savings, from The (organisational) Learning Curve, Scale Efficiencies and Moore's Law, Tech companies can drop prices and stimulate demand over decades.

We've seen that for PC's and HDD's and international telephone calls.

Re: Labor's NBN: headlines with zero substance

Good point. Looks bad, eh?

You've confused a Retailer problem with an NBN problem.

As well, where's "multicast" being sold? It's apparently built within the NBN network, so why aren't Retailers pushing it?

NBN Co aren't the people who set the Retail pricing nor do they provide network services & interconnects to customers.

Why no 1Gbps?

That's the retailers either not offering it, or pricing it too high.

[NBN is a wholesaler without control over retailer price offering. Their profits "get clipped" by the Retailers as well. Who'll get rich off the NBN services? the retailers not the common wholesaler.]

NBN, IIRC, didn't offer a CVC holiday on 1Gbps either, nor would any backhaul owner.

RSP/ISP's have to run backhaul from 121 PoI's, each dimensioned to support 1Gbps. Expensive.

A shout-out to the ACCC for that bizarre decision. Never been able to understand how that improved either Competition or promoted Consumer Interests.

Internode (simon's blog) is on record in Tasmania as discovering mixed lo-&hi-speed connections require different dimensioning rules for backhaul (they had a few 100Mbps mixed with many 12/1 IIRC).

Internode had constant complaints from everyone (lo- & hi-speed) about hi latency, freezing and 'congestion', when their link utilisation figures didn't support that.

I'node then increased the backhaul to cope with _two_ simultaneous 100Mbps connections and normal service was restored.

For 1Gbps, this 'midget & giant' effect is going to be much worse.

To offer 1Gbps services, RSP/ISPs need to dimension PoI interconnect and backhaul for a minimum of 2Gbps, on all 121 PoI's.

I haven't checked prices recently, but I'd guess currently $300M-$400M/year for all 121 PoI's.

Telstra and possibly Optus are the only ones who have the infrastructure to offer that, and neither have shown any interest in pushing new fixed-line services, not unless they can extract a premium.

So, how's that a problem of the wholesaler?

I see a failure of the Private sector and a larger Policy failure from the Government.

Re: MTM: the Road to Nowhere. Waste on a truly grand scale.

Thanks. appreciate you saying that.

MTM: the Road to Nowhere. Waste on a truly grand scale.

The Infrastructure Plan and Turnbull's MTM both suffer from an appalling gap:

- How does the built-network ever get upgraded? It can only be overbuilt, not reused.

Both the FTTN & HFC components can only be upgraded by "Forklift Upgrade", or full replacement.

NOTHING of value can be extracted from the sunk investment. Nothing...

We're not just on a road to nowhere, we're deliberating creating an expensive removal & remediation problem plus won't get a cent of value from the current investment. It's not just worthless in 10 years, it will cost us to pull out.

That's because:

a) Fibre is installed in long (20km) runs, as two-ended loops (survives a single cut), and

b) for Passive Optical (GPON or NG-PON2) you need to install high-count fibre cable everywhere.

The FTTN & HFC NBN rollouts are star-networks (hub & spoke), not loop, and they've not installed (in the docs I've seen) any of the high-count cables necessary for a subsequent upgrade of the network to GPON/NG-PON2.

When we want to upgrade the Customer Access Network to "Full Fibre", nothing that's currently being deployed in the MTM-NBN can be reused. Another lie in Turnbull's 2013 "Broadband Plan" was "50% Capital Investment reuse". Only it's zero...

But its worse than that:

in the future, we the taxpayer, will have to pay to rip out these crappy Nodes and pull out that obnoxious coax. There's a (high) cost in removal and site-remediation of all this new MTM work.

Every Node has lots of new 200-pair cable being rolled out to connect it to the many DA's (Distribution Areas, served by a 'pillar') it connects. They're big, heavy and expensive.

But more importantly, they fill up 100mm conduits quite quickly. The Telstra standard is conduits are only to be filled to "70 %". [Does that count the empty space between cables, or the space above?]

The legacy of the FTTN network will be full conduits, forcing a PON Fibre rollout to either route around those full conduits or be forced into civil works (the really expensive part of the rollout) to parallel.

What happens to all those thousands of tonnes of old copper, hidden in the ground and overhead, after they've been bypassed?

Do we leave them there to rot for 100 years (literally) or do we try to remove them?

If the new Fibre shares the same pits, pipes and conduits, it must've been laid on top of the old copper (for those services to remain active in the 18-month cutover), so how do we remove tonnes and tonnes of old, decaying copper from _under_ new Fibre cables without damage or disrupting services?

If you thought the "Asbestos Pits" debacle was a big problem, wait until the 'nbn' Co and the political classes "discover" that a) Copper is toxic and will leach into the groundwater and b) to remove it all, means ripping up every single metre of buried conduit and replacing the existing fibre.

If you thought the big problem with nodes was lousy speeds and highly congested uplinks, "you ain't seen nothing yet". Not only will ALL this new equipment have to be decommissioned well before it's service life is finished, a waste of money in itself, every node and every metre of copper-pair and coax will have to be laboriously removed and disposed of. It costs us coming and going, to do a rubbish job.

The sting in the tail of the Infrastructure Australia report is pretty significant as well:

- who gets to own and operate (maintain, upgrade, replace) the common parts of the NBN: the transit network, Points of Interconnect, Software (Billing, Monitoring & Provisioning) and standards, testing, integration and co-ordination / planning activities?

- how do those necessary internal services get paid for by the 'fractional operating companies'?

- How will individual operators fund the replacement of their whole operating asset? That's economic suicide, no sane business would consider it, no sane Bank would lend them $$$ to do it.

The plan that Turnbull, Vertigan and now Infrastructure Australia is promoting is not just terrible, its catastrophic.

Once balkanised and split into 100 different, small and aggressively competing 'operating companies', there can NEVER be, ever, a single common-access network in Australia.

Selling off the NBN assets will result in a complete and permanent road-block to Communications Infrastructure development in Australia.

Proof: look at all the incompatible State Railway gauges operating in Australia after nearly 150 years.

There is precious little 'standard gauge' (common access) running in the other states, making trans-shipping impossibly expensive. Rail should trounce road transport because of its dedicated right-of-way and incredible physical & economic efficiencies. Instead, it a terrible service, uncompetitive and always making a loss or close to it.

That's the future of Australian Communications when these plans to carve up NBN assets are implemented. It's almost the worst possible outcome that could be planned.