Colonial pipeline hack

[Ward Smith + 6,615]

or was it? Does anyone believe a multi billion dollar company doesn't have anyone who knows how to operate valves? Inquiring minds want to know. 

[Ward Smith + 6,615]

Amazingly, the FBLie got the ransom from the ransom ware. 

Should have been impossible. Or, we assume that supet, brilliant, secret hackers can get right thru firewall security layers, but are such internet noobs they couldn't secure the crypto keys? All righty then 

[Ward Smith + 6,615]

Wouldn't be anything like attacks on Oil trains

Interestingly, one of those perps worked for the DNC 

Quote

According to the Justice Department, the device is comprised of magnets and wires that are stretched between the tracks. On a now-deleted LinkedIn page, Reiche was previously employed as a Deputy Field Organizer for Washington State Democrats.

 

[Jay McKinsey + 1,490]

1 hour ago, Ward Smith said:

Amazingly, the FBLie got the ransom from the ransom ware. 

Should have been impossible. Or, we assume that supet, brilliant, secret hackers can get right thru firewall security layers, but are such internet noobs they couldn't secure the crypto keys? All righty then 

The hackers cracked one password and got into the billing system which they shutdown. Colonial then shutdown the pipeline because they didn't know who to bill. There is no evidence that they were brilliant hackers. 

[Ward Smith + 6,615]

56 minutes ago, Jay McKinsey said:

The hackers cracked one password and got into the billing system which they shutdown. Colonial then shutdown the pipeline because they didn't know who to bill. There is no evidence that they were brilliant hackers. 

That's "batsht crazy". Here's what Reddit thinks about this. Dr Evil is making a house call after his stint in Austin Powers

 

[turbguy + 1,540]

Honestly, I don't believe anything digital is totally secure. 

Very difficult, yes.   Totally secure? No.

Worthwhile to spend time and effort on?  Rarely.

Perhaps there's even some quantum computing involved here?

There has not been a lock made by man that could not be unlocked by man.

Edited June 9, 2021 by turbguy

[turbguy + 1,540]

I have also heard (but cannot confirm) that it was back-office systems that were attacked, not "plant operating" systems.

[Eyes Wide Open + 3,554]

20 minutes ago, turbguy said:

Honestly, I don't believe anything digital is totally secure. 

Very difficult, yes.   Totally secure? No.

Worthwhile to spend time and effort on?  Rarely.

Perhaps there's even some quantum computing involved here?

There has not been a lock made by man that could not be unlocked by man.

What lies in AZ. 

[Ward Smith + 6,615]

Worst. Hackers. Ever. 

12 year olds? 

This would be akin to kidnapping someone and then having the family put the ransom in a bank's safety deposit box. Not even stupid criminals are this stupid

[turbguy + 1,540]

9 hours ago, Eyes Wide Open said:

What lies in AZ. 

Digital only?  NO!

Digital, backed up by real paper.  

 

[Eric Gagen + 713]

General consensus seems to be that this 'hack' was very amateur - they used a spearfishing attack on some colonial executives and one of them had a password which was like 'password' or '1234' or something simple like that, and they got into the billing section of the system using that information beacuse the billing section of their system had basically no security on it.   By analogy the hacker is s a bit like a guy who went out fishing one day with his pole and some shrimp for bait and hooked a whale - he never prepared for that level of success, and didn't have a plan for what to do next.  

[RichieRich216 + 454]

These people must have no concept of what a password should be, ONE WORD!

 

[Ward Smith + 6,615]

1 hour ago, Eric Gagen said:

General consensus seems to be that this 'hack' was very amateur - they used a spearfishing attack on some colonial executives and one of them had a password which was like 'password' or '1234' or something simple like that, and they got into the billing section of the system using that information beacuse the billing section of their system had basically no security on it.   By analogy the hacker is s a bit like a guy who went out fishing one day with his pole and some shrimp for bait and hooked a whale - he never prepared for that level of success, and didn't have a plan for what to do next.  

There's a paucity of meaningful information about this "hack". People who watch too many movies believe nefarious actors have gotten control of the valves and switches directly, threatening to destroy the system if their demands aren't met. Yours seems more plausible, they merely went in and corrupted the accounting system, probably encrypted it into gibberish, then demanded ransom to release the encryption key. Pedestrian stuff, not Stuxnet level by any means. But speaking of Stuxnet, who sponsored that again? Why do so many people think this was a false flag operation? Enquiring minds want to know. 

[Jay McKinsey + 1,490]

1 hour ago, Ward Smith said:

There's a paucity of meaningful information about this "hack". People who watch too many movies believe nefarious actors have gotten control of the valves and switches directly, threatening to destroy the system if their demands aren't met. Yours seems more plausible, they merely went in and corrupted the accounting system, probably encrypted it into gibberish, then demanded ransom to release the encryption key. Pedestrian stuff, not Stuxnet level by any means. But speaking of Stuxnet, who sponsored that again? Why do so many people think this was a false flag operation? Enquiring minds want to know. 

So yesterday I provided the same basic explanation of the hack and you called it "batshit crazy" but today you think it seems "more plausible"? Interesting. What changed your mind?

Edited June 9, 2021 by Jay McKinsey

[Ward Smith + 6,615]

15 minutes ago, Jay McKinsey said:

So yesterday I provided the same basic explanation of the hack and you called it "batshit crazy" but today you think it seems "more plausible"? Interesting. What changed your mind?

You misunderstood. My point was that it is batsht crazy that one password in a billing system would shut down an entire pipeline serving millions of people? Which part of that seems like ordinary operating business procedures to you? 

[Jay McKinsey + 1,490]

1 hour ago, Ward Smith said:

You misunderstood. My point was that it is batsht crazy that one password in a billing system would shut down an entire pipeline serving millions of people? Which part of that seems like ordinary operating business procedures to you? 

One password did not shut down the pipeline. One password allowed the billing system to be shutdown. The company then used a different system and a different password to intentionally shutdown the pipeline because they were more worried about billing people than the mayhem caused by the shutdown. 

It isn't good business practice but it definitely sounds ordinary. Businesses are notorious for lax security protocols. 

• In a poll of 500 business owners, 81% said their employees understood the importance of data security.
• The survey also reported that 60% train their employees on data security practices. 40% go untrained!
• Still, 27% of business owners said they weren't leveraging any type of data security software to protect their devices.
• https://www.businessnewsdaily.com/15316-lax-cybersecurity-for-businesses.html

Then you have cases like SolarWinds who are an actual security company selling a security product and they don't notice that a product password was solarwinds123. 

All it takes is to get into a permissioned user's account and if it is setup with the proper tokens and keys to the billing system or that user has made a password file with all the passwords they use listed then you are in. Or maybe the billing system isn't set to even require login from behind the firewall.

The biggest mistake Colonial made was not to have two factor authentication required for user logins. 

 

Edited June 9, 2021 by Jay McKinsey

[Eric Gagen + 713]

3 hours ago, Ward Smith said:

You misunderstood. My point was that it is batsht crazy that one password in a billing system would shut down an entire pipeline serving millions of people? Which part of that seems like ordinary operating business procedures to you? 

Unfortunately all of it.  I gather you don't work with or for any major company.  The level of cyber (and often physical) security at a lot of places is surprisingly lax.  If you 'know where to look' it's rather simple to get into a lot of supposedly secure places, both online, and in physical plant.  Most 'barriers' to entry are intended to discourage amateurs, and delay petty theft until some other action can be taken - not defeat intentional intruders with an objective.    

And again to reiterate what @Jay McKinsey said they didn't get control of, or the ability to shut down the pipeline.  They got control of the billing system.  The pipeline was shut down because

1. The pipeline company didn't feel like giving away product for free, and 
2. At least initially it was unclear if seizing control of the billing system was a warning of a larger second attack (which might have already infiltrated the pipeline control system which WAS adequately safeguarded, and not hacked

If for some reason the whole mess would have gotten extended, (for example if their billing sotware and records were deleted) Colonial would have bashed together some sort of kludgy billing system and re-started the pipeline at some point.  However the whole incident only went on for a few days before they paid the ransom and returned to business as usual (hopefully with better security) 

 

Edit:  Real 'hacking' actually works like one of these 2 ways in 99.9% of cases by the way:

Edited June 9, 2021 by Eric Gagen

[Ward Smith + 6,615]

2 hours ago, Jay McKinsey said:

One password did not shut down the pipeline. One password allowed the billing system to be shutdown. The company then used a different system and a different password to intentionally shutdown the pipeline because they were more worried about billing people than the mayhem caused by the shutdown. 

It isn't good business practice but it definitely sounds ordinary. Businesses are notorious for lax security protocols. 

• In a poll of 500 business owners, 81% said their employees understood the importance of data security.
• The survey also reported that 60% train their employees on data security practices. 40% go untrained!
• Still, 27% of business owners said they weren't leveraging any type of data security software to protect their devices.
• https://www.businessnewsdaily.com/15316-lax-cybersecurity-for-businesses.html

Then you have cases like SolarWinds who are an actual security company selling a security product and they don't notice that a product password was solarwinds123. 

All it takes is to get into a permissioned user's account and if it is setup with the proper tokens and keys to the billing system or that user has made a password file with all the passwords they use listed then you are in. Or maybe the billing system isn't set to even require login from behind the firewall.

The biggest mistake Colonial made was not to have two factor authentication required for user logins. 

 

I understand all that. I am the only person I know of who has put a real hacker in prison, and gathered 6 months of keystroke evidence on him to boot. I used to sell managed security services. This isn't rocket surgery. Interesting that you mention Solarwinds. Of every known customer they have, only one claimed they were never hacked. That customer? The software provider to the 4 largest electronic voting systems. Totally plausible, right? 

As for "giving their product away free". That's preposterous! It isn't their product. They sell transit, they don't buy the refined product and hope the price doesn't change in the 4 days it takes to make delivery at 3 miles per hour, which is how fast the oil flows in pipelines on average. The refiners have their own records of how much they put into the pipeline. These are trivial problems. Deconstructing the data is nothing. 

[Jay McKinsey + 1,490]

2 hours ago, Ward Smith said:

I understand all that. I am the only person I know of who has put a real hacker in prison, and gathered 6 months of keystroke evidence on him to boot. I used to sell managed security services. This isn't rocket surgery. Interesting that you mention Solarwinds. Of every known customer they have, only one claimed they were never hacked. That customer? The software provider to the 4 largest electronic voting systems. Totally plausible, right? 

As for "giving their product away free". That's preposterous! It isn't their product. They sell transit, they don't buy the refined product and hope the price doesn't change in the 4 days it takes to make delivery at 3 miles per hour, which is how fast the oil flows in pipelines on average. The refiners have their own records of how much they put into the pipeline. These are trivial problems. Deconstructing the data is nothing. 

I didn't say "giving their product away for free". I said they were "worried about billing people", which they were. 

If you understood all that then why did you think that one password to the billing system shutdown the pipeline? 

Billing isn't about the refiners putting product into the system it is about customers taking it out, duh, and it is anything but trivial to collect and deconstruct that data. Again with the failing to impress the forum with your genius, have you figured out airplanes sitting on the ground at airports yet?

Edited June 10, 2021 by Jay McKinsey

[Ward Smith + 6,615]

4 hours ago, Jay McKinsey said:

I didn't say "giving their product away for free". I said they were "worried about billing people", which they were. 

If you understood all that then why did you think that one password to the billing system shutdown the pipeline? 

Billing isn't about the refiners putting product into the system it is about customers taking it out, duh, and it is anything but trivial to collect and deconstruct that data. Again with the failing to impress the forum with your genius, have you figured out airplanes sitting on the ground at airports yet?

Quit embarrassing yourself. You're not even remotely correct about airplanes taxiing on the tarmac. I know this is beyond you (everything is) but  imagine this scenario. A plane full of bad guys is on the ground and just moseyed over to within a hundred yards of Air Force One. Everyone jumps out, shoots the handful of ornamental guards and destroys the plane. Still think you can taxi in a restricted area? 

Trivial to figure out who took the gasoline too. I know people in midstream, they're all laughing at this. Improbable doesn't begin to describe it.  

[Eric Gagen + 713]

13 hours ago, Ward Smith said:

I understand all that. I am the only person I know of who has put a real hacker in prison, and gathered 6 months of keystroke evidence on him to boot. I used to sell managed security services. This isn't rocket surgery. Interesting that you mention Solarwinds. Of every known customer they have, only one claimed they were never hacked. That customer? The software provider to the 4 largest electronic voting systems. Totally plausible, right? 

As for "giving their product away free". That's preposterous! It isn't their product. They sell transit, they don't buy the refined product and hope the price doesn't change in the 4 days it takes to make delivery at 3 miles per hour, which is how fast the oil flows in pipelines on average. The refiners have their own records of how much they put into the pipeline. These are trivial problems. Deconstructing the data is nothing. 

The product that colonial sells is a tarrif on the volume of material transported based on how far it went, from who and to who, and what the product is. That's at least 5 data points you have to keep account of for each barrel of volume, and excludes things like product quality, penalties or bonuses for timely delivery, hedging, weather delays, etc.  That's what they bill for.  It may be trivial to take a static forensic snapshot of all the actions and activities of the pipeline over a certain past period of time and then reconstruct which barrels of which product went where and for how much, but it takes a lot of time to figure this by hand, especially if nobody has ever done it by hand before, and you have to create a way to do it with. After all this is an industry where things got widely computerized in the 1970's, and this particular pipeline has had computer controlled systems in place since it was constructed in the 1960's.  Creating an alternate/parallel billing infrastructure from scratch takes TIME.  It's not a matter of falling back on the old system, or estimating or something like that, because there is no old manual system to fall back on.  The manual systems as a backup iif they ever existed were probably removed in the 80's.  It will take more time to resolve all of this than it takes for the product to actually travel around the system, by which time you are now  even further'behind' on figuring out the destinations and prices of the products which are now in the system.  This requires that you  do it in real time while the whole system is up and running, and this is NOT a trivial exercise.  

Edited June 10, 2021 by Eric Gagen

[Ward Smith + 6,615]

12 hours ago, Eric Gagen said:

The product that colonial sells is a tarrif on the volume of material transported based on how far it went, from who and to who, and what the product is. That's at least 5 data points you have to keep account of for each barrel of volume, and excludes things like product quality, penalties or bonuses for timely delivery, hedging, weather delays, etc.  That's what they bill for.  It may be trivial to take a static forensic snapshot of all the actions and activities of the pipeline over a certain past period of time and then reconstruct which barrels of which product went where and for how much, but it takes a lot of time to figure this by hand, especially if nobody has ever done it by hand before, and you have to create a way to do it with. After all this is an industry where things got widely computerized in the 1970's, and this particular pipeline has had computer controlled systems in place since it was constructed in the 1960's.  Creating an alternate/parallel billing infrastructure from scratch takes TIME.  It's not a matter of falling back on the old system, or estimating or something like that, because there is no old manual system to fall back on.  The manual systems as a backup iif they ever existed were probably removed in the 80's.  It will take more time to resolve all of this than it takes for the product to actually travel around the system, by which time you are now  even further'behind' on figuring out the destinations and prices of the products which are now in the system.  This requires that you  do it in real time while the whole system is up and running, and this is NOT a trivial exercise.  

This isn't a quickie mart. These are large volumes going to large customers. No one would risk trying to cheat them by hiding gasoline. The shutting down of the pipeline was an overreaction. It's not about money it's about fear they'd gotten much further than they did. This is all speculation, including that it was a billing hack. Given how lax accounting software security is, this kind of attack will occur more often if they don't figure it out.

[Eric Gagen + 713]

10 hours ago, Ward Smith said:

This isn't a quickie mart. These are large volumes going to large customers. No one would risk trying to cheat them by hiding gasoline. The shutting down of the pipeline was an overreaction. It's not about money it's about fear they'd gotten much further than they did. This is all speculation, including that it was a billing hack. Given how lax accounting software security is, this kind of attack will occur more often if they don't figure it out.

Sure - that was my very first answer - that they shut it down as a precautionary measure, but you kept asking why they did it, so I explained some more.  More on the billing side however, it's not that the colonial customers are trying to 'cheat' or 'hide' product - in many cases without accurate data from the colonial billing system they don't know what they are receiving in sufficient detail to be useful. 

For an example, I actually worked at a place where we had that problem.  We were receiving regular deliveries of liquid nitrogen to our on-site storage tanks from one supplier on a consistent basis.  However we kept having to place a small number of extra orders above and beyond what we thought we were using up - about a 10% excess.  It turned out that one of the tanker delivery drivers was not accurately metering the volume of product he was delivering - instead he wrote up his delivery tickets based on the volume which he received from the shipping plant.  Why does this matter?  Because there is always some amount at the 'tank bottom' that isn't efficiently offloaded, as well as some loss in transit.  The equipment we had at our facility were not accurate enough to measure exactly how much product we had before and after delivery - errors could be as large as 5% of the total volume (much more than just 5% of the volume delivered) meaning that in most cases the best we could do was verify if we had received a delivery or not, but without knowing the size of the delivery.  An additional complicating factor was that we were offloading our large storage tanks to our own smaller delivery and distribution tanks, sometimes simultaneously while receiving deliveries to top up the storage system, adding another variable.  We suffered our own measurement losses in the process (partially filled tank bottoms, being the biggest factor) but were willing to live with them, since the cost of fixing them was pretty large.  We HAD to get accurate data from our supplier/distributer because we simply could not back calculate it accurately for ourselves. 

It took us about 3 months of investigation to figure out what was going wrong, with everything from a large scale search for hidden leaks, a review of our accounting and record keeping for sales, meter calibrations, changes in our process for filling and accounting of tankage for sale, checking our service supervisors to see if they were improperly accounting for or measuring  the materials they sold, etc.  We went so far as to get estimates on replacement costs for our storage supply system (millions of dollars) to stop the 'leakage'.  The way we figured it out was by comparing the discharge receipts from the supply plant with the intake receipts to our facility.  When we discovered that some of them were always identical, we were able to trace it to a specific person who was not properly following the chain of custody.  If our supplier did not have accurate data for all of this stuff, we would probably have wound up replacing the storage portions of our facility at great cost, and not even solved the problem.  

[-trance + 114]

6 hours ago, Eric Gagen said:

 

For an example, I actually worked at a place where we had that problem.  We were receiving regular deliveries of liquid nitrogen to our on-site storage tanks from one supplier on a consistent basis.  However we kept having to place a small number of extra orders above and beyond what we thought we were using up - about a 10% excess.  It turned out that one of the tanker delivery drivers was not accurately metering the volume of product he was delivering - instead he wrote up his delivery tickets based on the volume which he received from the shipping plant.  Why does this matter?  Because there is always some amount at the 'tank bottom' that isn't efficiently offloaded, as well as some loss in transit. 

Gas waste is a huge problem.

The lab always had to have liquid argon in stock, but as you know the dewars have pressure release vents, so some is always lost in transit and storage. 

Occasionally they would ship a container that had lost almost all in transit or had a faulty vent valve and would leak almost constantly. They were really good about refunding your losses - if you actually told them about it.

 

 

 

 

 

 

[Eric Gagen + 713]

58 minutes ago, -trance said:

Gas waste is a huge problem.

The lab always had to have liquid argon in stock, but as you know the dewars have pressure release vents, so some is always lost in transit and storage. 

Occasionally they would ship a container that had lost almost all in transit or had a faulty vent valve and would leak almost constantly. They were really good about refunding your losses - if you actually told them about it.

 

 

 

 

 

 

Yep - similar problem, but it was made worse by improper accounting of what was being sold in part of the cycle.  Hence everyone trying to figure out where the extra boil off was at.  This was with liquid nitrogen in volumes of 2-10 tankerloads a week - the cost per unit volume isn't very high, but the lost/unaccounted for costs started to add up fast.  We would loose a lot on transporting liquid N2 in 2,000 gallon (7,800 liter) tanks at sea - all that sloshing around in the waves led to really high boil off.  Some of our customers wouldn't believe how much until they sent representatives to gauge the before and after of the trip themselves.  Under bad weather conditions it would be possible to loose 1/2 of a load in under a week.  

 

My main point for this discussion is that if you don't have accurate data on what's coming in (say because your pipeline company can't tell you exactly what it sold you and when in a timely manner) it's surprisingly easy to loose track of what you actually have, or what you have actually purchased or sold.  Seems impossibly difficult on the outside, but when you are in the middle of a mess like that, you have no idea how to get back to a known condition.  

Edited June 11, 2021 by Eric Gagen