Samsung reportedly leaked its own secrets through ChatGPT Less than three weeks after Samsung lifted a ban on employees using ChatGPT, the chaebol has reportedly leaked its own secrets into the AI service at least three times – including sensitive in-development semiconductor information. The ban was intended to protect company data, though was lifted on March 11 to enhance …
There's been plenty of talk about how ChatGPT is going to replace skilled programmers, but if employees at companies the size of Samsung are already doing things like this, it makes me wonder whether they have any in the first place.
And if everything that gets asked of it also gets re-ingested back into the model, then anything commercially sensitive will automatically become available to everyone else who uses it, making it a total no-go area for anything to do with your business if you have any sense.
Why am I picturing a clippy-like answer from CGPT?
"
Hello, it seems you're trying to fire employees. Would you like help with:
* safely starting a controlled employee fire;
* how to avoid fires starting from a fireplace;
* ways to ensure they burn to the bone;
"
"Meanwhile HR is inputting the entire personnel file into ChatGPT and asking it which employees should be fired ..."
That then becomes released to the wider world and the company winds up facing an class action lawsuit from over 50% of it's employees that's open and shut. The case then goes on to be tried under privacy laws and yet another company's assets are up for auction.
You missed the "iron humour". Obviously, use of this shit for HR (or anything important at all, for that matter) should never be allowed. But it will, and probably already is.
As long as they don't admit to the tribunal that they used it, then they will get away with it.
Just like the "wally" programmers, who (probably) won't be fired unless their boss realises that they are using a bot to produce the drivel that they commit.
There are gazillions of skilled programmers out there, but only a small percentage of them actually have full track record behind them. A massive chunk of what is out there is re-trained professionals that moved into programming from another profession. It's the same in cybersecurity.
I've been in tech wearing various hats for almost 25 years at this point, I've always been in tech and the gulf in skill and ability between experienced people in the industry and those that have retrained is absolutely massive, yet employers value them the same and measure them on metrics that make no sense. This leads to a general "brain drain" effect...I personally wouldn't want to be outnumbered by zombies, I certainly wouldn't want to be responsible for them either. The net result is you end up with shit PMs and absolutely sprawling teams of mediocre talent...you only have to look at the layoffs at the FAANG firms to see this. Whilst some of the people laid off are probably solid at what they do, the majority are not...I was speaking to a company recently trying to hire new devs and because of the layoffs it has become even harder than it usually is to find actual talent...the absolute shower of shit they've had come through the doors to interview is apparently staggering. They have actively started filtering CVs that include Twitter, Facebook etc etc.
The common aspect that they note is that they are way too dependent on other people to get stuff done. How do I do this? How do I do that? etc etc as well as constantly trying to get themselves off whatever they've been put on and trying to worm their way into other projects that are above their skill level because they're "less boring" and have more people working on them, thus it's easier to melt into the background...I've never worked at a FAANG company, but based on what I know from what I've heard, it sounds like hell on earth.
This is a firm that does serious engineering as well, they build testing gear for labs and automate a lot of hardcore testing stuff, not a fly by night data aggregator building some sort of news feed gig economy social dating second hand tat flinging pizza delivery taxi app with video shorts.
If you have FAANG on your CV and you're out of work and you don't seem to get any replies from job applications...now you know why.
Circling back to Samsung somewhat...I have had experience pissing off Samsung developers before, same client...quite a few years ago now they had a team of devs that build a virtualised test harness out in Korea and it was licensed by a client of mine but the license activation routine in the VM was terrible (i.e. it didn't really work) and the VM would sporadically not work (because the Apache service in the virtual machine would randomly crap out or just not start at all). When I finally got my hands on it and I was able to...*ahem*...take a look under the bonnet...I scraped out the shitty activation routine and had a dig into the Apache config. It turned out that the VM was an absolute shit show of a mess. There were multiple HTTP servers installed that all tried to start at start up, Apache would usually win...but occasionally NGINX would jump in and very rarely LightHTTPD all battling for port 80...there were files with .bak extensions everywhere and the code was laden with massive chunks of commented code...it was like a teenagers bedroom project. Absolutely terrible.
Anyway, I fixed all the issues (which go way beyond what I mentioned above, I won't go into further detail as it might give away which test harness it was) and I sent an email back to Samsung with a link to the "fixed" VM...absolutely nothing happened for ages. But one day out of the blue, I got an email forwarded to me from someone at the client that had been sent by Samsung and written entirely Korean and it was basically a tirade calling me all sort of things. :)
I was summoned into the CEOs office at the client and for about 10 minutes, I was convinced I was about to lose my contract and get the bollocking of my life...but mercifully the CEO saw the funny side and he broke down in tears of laughter...it was amazing to him that via me they'd managed to piss off Samsung. He was laughing his tits off! Relief swept over me like a cool breeze, it was awesome. From that point on, they've hired me over and over again to dismantle any third party test harnesses they get from electronics manufacturers ever since! Been doing it for 12 years now.
"There are gazillions of skilled programmers out there"
and that's what HR departments are for, to filter them out so they aren't considered. In the old days when a department head or supervisor would review applications, there was a better chance that qualified people could be found.
That's the problem right there, HR. They filter by paper qualifications first. The qualification that used to be used in this way was experience. Got years of experience? On the pile! These days, its vague qualifications that don't mean much. My nephew is currently doing Comp Sci at Manchester Uni. The syllabus reads like the Network+ combined with Security+ and the current versions of 70-210, 70-215 and 70-216 and that's it. 4 years to get the equivalent of less than an MCSE...which I did in a month. Followed by my CCNA, VCP and various other qualifications...which at the time were meaningless because it was experience that opened the door...I've gone the load road to experience and now it's the other way around...
We've had a whole generation of engineers get fucked...that's why there is a shortage. Too little change, too slowly.
"And if everything that gets asked of it also gets re-ingested back into the model"
Surely that's also away to poison it. Just keep asking it questions that make it suck up the question as an answer. Somehow.
"And if everything that gets asked of it also gets re-ingested back into the model, then anything commercially sensitive will automatically become available to everyone else who uses it, making it a total no-go area for anything to do with your business if you have any sense."
Everything said during a session is not ingested into this model, because the model can't do that. It is trained once and if you want to put some more data in, you have to start training it again from scratch, so that never happens. That's why GPT tends to be at least a few months out of date and can't explain anything that involves something happening recently. This means that if you put in some fact it didn't already know, it's not going to show up in anyone else's conversations, nor even yours if you have a later one. That is not a risk.
What still is a risk is that you've still sent a bunch of data to a different company without analyzing what they say they're going to do with it and what they actually do. Nothing prevents them from using that to train the next version of the chatbot, being hacked, or selling your conversation history, so unless you have a legal contract, that could still be a risk.
"Everything said during a session is not ingested into this model, because the model can't do that. It is trained once and if you want to put some more data in, you have to start training it again from scratch, so that never happens."
Really? That's a pretty shit model. Surely the whole point of an "AI" is that it can continue learning. That would be like a human leaving school and then never, ever learning anything new ever again. Obviously what it is allowed to learn would need to be curated since it's probably not a good idea for it take every session as gospel when $random users are interacting with it. I can see how easy it would be to "poison" an "AI" with say, a group of Redditors or 4Channers asking questions in conversation and report correct results as bad and vice versa, but basically marking a cut-off point in the "AI's" learning feels very very wrong.
That might be better, but it's incompatible with the methods they're doing and prohibitively expensive to do more than once a year or so based on the way they've been making new ones. Part of the reason for this is that the models don't keep every byte of their text training data, although they keep a lot of it, and they thus can't tell how to weight some new text because they don't know how large their sample sizes for various calculations are. Maybe AI researchers will come up with new types of models that can do this better, and if they do I won't be surprised to see those methods tested and employed, but for now the model types they're using do all the training from scratch.
OK we all know there's skilled staff shortage everywhere.
Engineers being so stupid to publish proprietary information to a no at all intelligent chat-thingy hoping to get a solution for their issue?
Skilled staff shortage just turned serious, Samsung engineers are daft or their 70 hour workweek just backfired. What's next?
"Imagine how much you can mess around with confidential information when you got 70 hours a week to look extremely busy at work."
Working those hours means that half the time you are at work, there isn't any management around. They've all buggered off to play golf, attend dinner parties or, gasp, spend time with their families so they continue to have a family.
It's always been serious, but unfortunately the hirers don't take it seriously.
I have 25+ years of experience at quite a few very high profile places, I have a proven track record and I can get you glowing references for days, yet my CV would never make it through the filters at an agency because I don't have a degree, and I don't hold many recent industry certifications, I used to hold loads but when I finally learned that they hold no actual value, I stopped taking them. This is why I work freelance. It's seriously way easier to get high paying work when you can pick up the phone and speak to a business directly.
The best (and probably only) way to get a high paying job in a large business these days is to build something, blown their minds with it, and have it bought off you with a position at the business attached. This is probably why the tech industry has become so focused on acquisitions rather than hiring and building in house. All the risk is on the nerds building things out of their own pocket and they can pick and choose the successful bits with very low risk and just a cost attached.
The recruitment scene though is just fucked and nobody in it has any idea what they are doing. Last time I went through an agency for anything (we're going back at least 15 years here, I still had a lot of experience (espcially for my age at the time) and yet, they put me through these bullshit tests like measuring my typing speed etc...which was a colossal cock up in itself because their testing system couldn't handle anyone that could type faster than 90wpm...I got the result and went to sit down with an "agent" and first thing I did was apologise for how low my score was, I used to be able to reach 140wpm+ in my younger days, but the agent looked at me confused and said "Low? 90wpm is insane! It's where the system caps out"...I then had to sit a really basic technical test which I was given 2 hours for, I was done in 20 minutes...but they automatically assumed I'd just skipped a load of questions..."Oh dear, well sometimes people do skip a lot of questions, you can still pass though"...then the score sheet came off the printer...99%. No idea where I lost that final 1%...but it has bugged me for years...anyway, even after absolutely destroying the tests they gave me, they still only offered me first line support roles because "I didn't have a degree"...this is where I believe "the shortage" comes from...there are tons of engineers out there like me with bucket loads of experience that can just crack on with the job, but we fall through the cracks because we don't have the paperwork. The people doing the hiring don't understand us, what the world was like when we were entering the market during the dot com boom / crash etc etc.
I'd imagine a lot of people my age are in a similar boat (35-45), because around the time I was considering University, the Comp Sci courses were crap, this was the late 90s and they were still teaching COBOL, FORTRAN and Punch cards, no web technologies, no advanced networking basically nothing of any practical use, certainly nothing I'd been teaching myself for years, I built my first website when I was about 9 years old in 1996...it was crap, sure...but I did it. It was even worse if you wanted a Masters...because by the time you got out the other end and started job hunting, you'd have been more than a decade behind someone that didn't go to University...so the common, and possibly sensible, choice was to just not go and just get straight into the workforce at a junior level after getting a couple of Microsoft Certs. I was around 19 when I started job hunting. Went freelance in my mid 20's after I'd gone 6 years without a payrise...despite holding a relatively senior position (unofficially) where I worked. People started to be hired over my head that were less capable than me and I realised that there was no chance I'd ever progress...so I fucked off and started hoovering up the customers as their contracts with my former employer ended. They ultimately went bust because I wasn't the only talent being shat on, they ended up with a team of University fuckwits that structured, theorised and over-planned their way into oblivion. Project completion times sky rocketed, outstanding tickets went through the roof and because all of them held degrees, none of them would work the helpdesk as they saw it as beneath them...so they ended up with no help desk. To be fair though, they were pioneers in the field of having complicated tiered payment systems and no contact details on the website way before it was "cool".
"build something, blown their minds with it, and have it bought off you with a position at the business attached."
That doesn't always work as Elon showed. He sacked somebody whose company Twitter bought and brought the former owner on board as part of the deal. It really didn't help Elon when the person turned out to be somebody with a medical condition that wanted to continue to work as long as they could and generally from home as they needed more accommodation in addition to having a contract that spelled that out.
The people at Twitter that did that deal either did no due diligence or were incredibly woke.
That situation is an edge case for sure. Either Twitter didn't want to be seen to crap on disabled people or they didn't know he was disabled. I think probably the latter.
Also, in this instance the project wasnt core to the business, it was more like a funded side project...probably written off as some form of R&D.
"Either Twitter didn't want to be seen to crap on disabled people or they didn't know he was disabled. I think probably the latter."
So the contract specifically saying it, agreeing on accommodations, signed by the company didn't tip you off that they might have known a thing about it? Musk probably didn't know, but the company did. This also suggests they did, in fact, do some investigation into what they were buying and they thought it wasn't going to cause problems they couldn't handle. And, since he was in a job that involved a lot of managing between teams, his mobility restrictions probably weren't that big a problem until Musk was looking for an excuse and still chose wrong.
I'm also not sure why you think that Twitter agreeing to let someone work from home when they were using a policy where everyone could work from home and for an employee in a different country who presumably didn't want to move across eight time zones counts as "woke". However, since you used the word, I can safely assume that your logic would be incomprehensible or stupid and thus I don't have to look into it too deeply.
Well the guy still works there apparently.
Unfortunately, we'll never really know if he was kept on because Twitter was starting to get a lot of discrimination suit threats and they wanted an example to prove that not all disabled people were fired, or whether the tit for tat was so public they needed a tidy bow in it to quieten it down, or whether he was kept on because he genuinely is a unique talent.
This post has been deleted by its author
I can sympathise, having a shed load of experience coming from 8 bit home micros, modems & electronics & over 20 years of desk side support 1st\2nd & moving into tier 3 at times.
I got a contract position, the recurryment agency that wanted my Ltd company details & registration details on receiving & accepting the offer urgent urgent urgent we want you NOW.....
Then they decided they wanted full background checks email conversations went like this:
Where did you do your degree - I haven't got a degree, I never stated I had a degree, I have a HNC in Electronics engineering its "near equivalent vocational" grade to a degree, only more practically biased.
How long were you on it - It was a 1 year full time course.
So you never actually completed the course - Did you even read my last email?
Can you give us your course tutors name & contact details - I don't recall them & even if I did he's probably dead by now.
After weeks of now no employment, no start date etc.....I got a interview out of the blue, for the following day & was offered the (permy) job verbally on the spot.
Three days prior to the start date, the recurryment agency\end client finally decided, contacting me by phone to inform me I could start the following Monday "generously waiving" the things that they couldn't verify beyond the last 8 years.
I'm sorry but I have tired of waiting for a start date that was supposed to be within a week & you\client have dragged out to nearly 6 weeks of pointless email exchanges going back over 30 years of ancient employment & academic history, leaving me in a state & expense of setting up a ltd company & giving my notice, thus without wages.
What? How can you take another employment? - Very easily as you have left me high & dry while asking questions that I have no reasonable expectations of getting personal references from beyond the grave & it has been over two weeks since I last heard from you in any follow up capacity. Goodbye! /end_call
Email arrives the next day..........
It occurs to us that your new employers will also be following the same background checks to ours, this will take many many weeks & you will have to wait for your start date. we recognize the inconvenience you have suffered as a result of the delay, so we can offer a rate increase of $3ph to compensate if you will start Monday - They did their checks in less than a week, took my word at face value & have gained my loyalty & respect, while you have evaporated yours into the wind. I gave them my word I would start with them, you gave me your word for a start date that rapidly disappeared into the ether with unreasonable expectations, I leave it to you to explain to your client why I will not be starting on Monday & please refrain from contacting me in future about any employment opportunities..
Mate, I hear your frustration, but let's not forget there's more to the story. Not all companies and agencies are hung up on formal education; some actually care about skills and experience. You just have to find the ones that share your values.
And about those Comp Sci courses from the 90s – things have changed, big time. Universities are now teaching cutting-edge stuff, with internships and co-op programs thrown in for good measure.
As for certifications, yeah, they're not a magic ticket, but they can show you're keen on learning, especially if you don't have a degree. And here's a tip – don't ignore the power of networking and personal branding! Get yourself out there, make connections, and flaunt your skills on LinkedIn. You never know what doors might open.
The tech world's a big place, with loads of options – freelancing, startups, big corps, you name it. You've just gotta find a gig where they appreciate your skills and help you grow.
Long story short, the recruitment game might be a bit naff, but the tech industry's still got loads of opportunities. Keep learning, networking, and showing off your skills, and you'll find a way to beat the system.
"if everything that gets asked of it also gets re-ingested back into the model, then anything commercially sensitive will automatically become available to everyone else who uses it"
That is how I understand these things work. The other option would be separate training and use phases, which would make the AI less useful (it would never learn after training).
It's the other way around. It doesn't keep the data and reprocess it because it isn't designed to be easily retrained on new data. In order for this type of model to get new data permanently added, it has to be added to the training set and the training routine run over again. That full retraining takes a lot of time and money, so they try to do it as little as possible. Technical limitations prevent them from automatic reingestion.
Updating the model on new inputs would also play merry hell with the post-training phases, such as fine-tuning and RLHF. It's simply Not How Any Of This Works.
Of course, if (when) the next generation is trained in a similarly indiscriminate manner to this one, hoovering up all available corpora, then it'll pick up a lot of LLM-generated content. Microsoft/OpenAI and Google and the other LLM trainers aren't going to be any better at filtering that stuff out than anyone else is.
When AWS first started getting bigger and S3 buckets were left public lots of info got leaked. Here we are again with a new piece of tech kit and it's happening again.
Pound-to-a-penny the second you let a public service inside a corp without proper oversight, some numpties will not even think once about what the external org with do with the info they get given for free.
I went on a security course about 5 years ago and the guy running it said the best place on the entire internet to get free accounts into systems is Github repositories. Devs are under such pressure and security is always the last thing considered in most projects that they rush code out, commit in and don't rememebr to annoymize, you can pretty much get free account logins to anywhere if you look for just an hour or two.
Not to mention harvesting keys and credentials from GitHub repositories. Or people throwing coredumps up on Pastebin for someone to look at. Or a hundred other things.
Your point is well taken: if an information service is writable, on the open Internet, and doesn't have tight information controls enforced from the beginning, people are going to leak information into it.
This is just ridiculous. Who in their right mind would input anything confidential, whether personal or professional, into any kind of online service??
Back when t' internet was a new thing for Joe public, in the mid 90s, I used to run seminars for local small business owners - a talk about the net, what it is, how it works, what it can do for them, followed by a brew then hands on surfing for an hour or so. I drummed into these newbies over and over again, never say anything in email that you wouldn't be happy to see on the front page.
The same is true today, but also includes web forms of all descriptions. It just defies common sense that people would feed anything confidential into a system they do not control, regardless of whether that system promises not to use that information or not. Policies change, mistakes are made, information leaks.
It may be that folks are so used to posting their every movement onto social media that they just do it automatically. Or it may be they're bloody stupid. I'm thinking it's the latter.
"Who in their right mind would input anything confidential, whether personal or professional, into any kind of online service??"
You should insert the /sarc tag immediately after the sarcasm. People do that sort of thing every 5ms somewhere in the world if it sounds like it will halve the time it takes to pay their utility bills or they will be able to see inside their fridge from anywhere in the world. They won't stop to consider that paying the electric company after spending moments to make sure the bill is correct takes no more time than a commercial break on the telly. Over the course of a year they could save an hour if they sign up for auto-pay. Never mind that it might take several years to sort out an incorrect billing and all during that time the money isn't in their account.
I don't see any evidence that people are taught about privacy and security at any level. Coupled with no education in personal finance and it's a perfect storm to wind up having to live in one's car. The benefit doesn't have to be Earth shattering, just that is sounds good enough to get people to sign up. Social media sites make the bulk of their money selling user data to their customers, which are the advertisers. The advertisers then spend money on ads they can craft to target to those social media users they have all of the insight on. The SM site is bait that's only mildly poisonous or similar to MSG that keeps people coming back for more and divulging more when they do.
If a company can't manage their own data themselves, there is something very wrong with the business plan. Information is often the prime asset of a company and that asset that is easiest to secure. If your company builds a product, anybody can get a copy of that product and see what you've done. If you hand them the design documentation and component vendors, you've poo'd the scrooch by handing out the part that you could keep secret.
"I drummed into these newbies over and over again, never say anything in email that you wouldn't be happy to see on the front page.
The same is true today, but also includes web forms of all descriptions. It just defies common sense that people would feed anything confidential into a system they do not control"
Maybe they did it because your advice was absolute to the point of uselessness. Before those businesspeople used email, they would say the information into the phone, connected to lines they also didn't control, or sent through the post, which they likewise didn't control. Neither was encrypted, both could be intercepted as easily as email could, and there was no available alternative that had more security short of in person only meetings. Some might have interpreted your warnings to mean that email was especially bad at security, which compared to existing mass-market technologies it wasn't. Those who understood that they were both insecure probably thought about your advice, realized they hadn't been told about a better option, so went with it anyway.
People have a poor understanding of security, most often due to not caring to learn about the details. It doesn't really help to give them unrealistic standards that they won't be able to meet, because they'll come to the incorrect conclusion that computers are never to be trusted and they have to decide between living off the grid or having no security. If you present someone with that false dichotomy, they're likely to be among those who decide that, if everyone is spying on them and there's nothing they can do about it, why not hand over all their data to Facebook, Google, and any page that suggests you log in, because at least that's convenient.
User education is hard. I get the idea that it might be better to overstate the risks in order to err on the side of security. The problem is that simplistic answers may lead the users into errors they wouldn't make if the more difficult work is done or if the IT security team makes the decision for the company and uses network blocks and warnings to enforce it.
And continual vigilance is simply impossible for humans. It consumes too much cognitive resources. We're not physically capable of considering the security implications of every decision, or even of considering every decision, period.
And we know from the past several decades of IT debacles that blaming the users just doesn't help. It achieves nothing. Some level of training is worthwhile, but you hit the point of diminishing returns pretty quickly, and punishing people only makes them hide things, which delays incident response.
You just have to have good information controls at multiple levels – including restricting access to low-value services like LLMs. (I am utterly unpersuaded by the arguments I've seen so far about "developer productivity". If an LLM improves your productivity significantly, you're not a good developer.)
"We're not physically capable of considering the security implications of every decision, or even of considering every decision, period."
I can agree with that which is why it's good to have a default position of "no" on providing certain bits of information. Spend some time thinking about the sorts of things that you shouldn't disclose so you aren't having to go through a considered reason tree each time. I go to the DIY store and they want me to go to the URL on the receipt to fill out a survey. The first thing they want on the survey is your name, email and physical address and a phone number (modern unique identifier). I expect if you use the QR code, they'll be able to reference the purchases you made to your survey answers. The local dollar shop wants me to do the same thing along with the post office and plenty of other places. My default answer is no. I don't care that I might wind $5,000 in a giveaway since I won't win. The best I'll do is get a small percentage discount if I make a very large purchase in the next 30 days. If you automatic answer is no answer or no, you will avoid a ton of grief. From time to time you can spend some brain power to analyze whether it might make sense to say yes. Never have a fear of missing out. The one time something might work to your benefit won't outweigh the 99 where you get taken.
"Samsung reportedly applied "emergency measures" that include limiting upload capacity to 1024 bytes per question."
Samsung's IT security operations are one ongoing "emergency measure" though?
Their VPN capability is a joke, the blocking of copy and paste, the blocking of any upload to Google docs, etc. It's one hard-coded (yet so easily-circumvented) rule after another.
You specialists on here (I'm just a humble tech marketer) would be in stitches if you saw it in inaction (sic) first-hand.
Anyone knows how these guys even managed to upload a PDF to ChatGPT? When I asked ChatGPT this question, this is the answer I got:
"I'm sorry, but as an AI language model, I don't have the capability to receive or store files, including PDFs, from users." ~ https://sharegpt.com/c/7ttzFqL
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
