Lol, I could kind of understand (not really support) his use of AI. While I doubt it’s something you can do for long without starting to make compromises, people who were a really good programmers before will probably use it responsibly for a while, until their skills inevitably erode and (more importantly) their willpower slowly runs out.
But this argument made me loose all my respect for him. It’s so utterly stupid. I hate it when i have to check my assembly code line by line, because it’s not guaranteed that the compiler will compile the same code I’ve written deterministically, and basically just randomizes what it outputs. Or when my editor does not save what I typed, and instead decides based on a random language statistics what to actually save, so I have to double-check my text word-by-word every time.
Oh, wait. No, because the tools are deterministic, and that doesn’t happen. But of course, adding randomness into your workflow, where you have to focus and review everything line-by-line, is the same as using deterministic tools, lol.
His point is that an LLM is just another tool for increasing productivity, the same as a compiler or IDE. The quality of the code is the only metric that should be judged.
Except they’re not? Very few tools are deterministic. Only Mathematics is, and that’s only a small subset of tools.
You’re free to argue that non-deterministic tools lack value, but that’s clearly false: OCR only exists because of lossy training machine learning algorithms. Or, going even more basic, Monte Carlo simulations are by far the most efficient method of testing entire categories of problems that are intractable deterministically.
LLMs can be shitty, as a business practice, due to their unpredictability, but (hopefully obviously) that depends on how it’s used.
I’d argue that Monte Carlo is a bad example here, and is an example of a deterministic tool.
You use a tool to create an outcome, you’re looking for an answer or a solution that you can trust.
If you want to examine behavior of a system that has input variables across some known range, you could brute force it, try every single possible combination and wait a millennia. Or you use a monte carlo strategy, let randomness enter the simulation to build out a basic characterization. You want more detail? More confidence? Increase the number of runs. I can trust that if I increase the number of runs I gain higher fidelity, if the tool/strategy was non deterministic I couldn’t do that
Not parent commenter, but no, not to the same degree.
The level of trust I have in a source greatly changes how I review it. If I know that I’m reviewing code from a competent developer, I’m going to see a for loop and look for why is this here, how does it play into the larger objective and if I’m happy with it, I move on. But if I don’t trust the developer as much, I’m looking at far more like what data type is being used in the indexing, what’s the break condition, can go off the rails into random memory.
Competent developers make mistakes too. I think it’s worth at least skimming every line in a PR. Things like indexing out of bounds should probably be handled by testing or using a memory safe language.
I don’t think code review is the place where you should be trying to catch bugs in the minutia of the code. You should be assessing readability/maintainability, architectural decisions, risk, and testing strategy. That does involve reading all of the code, even if only by skimming.
I guess I don’t really count skimming as reading haha
When looking for readability, maintainability, etc what I mainly look for are design patterns. If you’re using the same boilerplate and layout I’m going to start skipping over sections
I also mainly work on high performance embedded systems that are difficult/nearly impossible to unit test and I dont always have line coverage metrics. A lot more verification by inspection in my corner of the world
There’s a couple big reasons, sometimes you only have to deal with one or two, other times you’re hit with all of them.
The data source for the algorithms often comes from arbitrary memory addresses that live outside the context of the C, C++, or Rust runtime. If I want to read CPU core temp I send magic numbers to one address to configure ADC channel conversion, another magic number somewhere else to do the conversion, wait a few dozen microseconds, then read a third new address that I squint one eye, spin around 10 times, and do a handstand and I’ll get a number that looks like temperature.
If I’m running on the microcontroller, I’m essentially in production environment and I can’t control what the data is. If I recompile for a different system (say x86 Linux) then I’m almost certainly going to segfault. I could abstract out the reading of the data, but then on my tiny 8MHz microcontroller that has to run it’s main loop at 1000Hz, I only have a few thousand clock cycles to get everything done, calling functions, pushing/popping the stack takes a lot of time.
Adding to the pile-on most microcontrollers have <1MB of RAM, many in the tens of kilobytes range so I don’t have much of a stack to work with and I need to keep even the number of instructions down.
Even if you do all the abstractions, you then have extra logic you have to write to simulate what is happening and subtle behaviors that are timing dependant. It frequently is extremely difficult to do that as the documentation rarely fully describes everything, so you can lull yourself into a false sense of security that you’ve tested the code and it’s correct, but then you get stuck on a register value that never changes and a watchdog timer comes along and resets the hardware on you.
And that’s not the getting into the runtime or build environments, flashing procedures, limited flash cycles, and a million other business constraints that technically you can do it, but at that point your dev team has all taken the month off to go play Half Life 3
That definitely increases the challenge of writing tests or trusting that they are faithfully representing your production environment.
I could abstract out the reading of the data
I was going to suggest this, and it sounds like that ends up being expensive. It seems like you need some kind of test harness that’s able to emulate your specific hardware while still allowing for testing hooks that don’t impede the tight clock scheduling.
If unit tests are being done, some kind of emulator is usually what gets used and the scope of testing is significantly limited.
The more common approach is to abandon unit tests all together and go for board functional tests where as many of the external interfaces are controlled and put through representative loads. The complexity of the software is rarely that high so we can be pretty confident that we’re covering everything we need to care about anyways.
You do that inherently when you are writing the code. Every line you write has already had that focus, because you had to come up with it.
That you should also not trust every line of the compiled assembly after you wrote the code, and check it lime by line, is an extremely stupid argument.
Of course, you should get someone else, with a fresh/unbiased opinion to review your code too, but just the act of figuring out every line you write is in some sense a rewiev, that there is at least a plan or reason why it was written.
AI just randomly generates code without thought or reason, and you have to check everything. Unlike assembly generated from code.
That you should also not trust every line of the compiled assembly after you wrote the code, and check it lime by line, is an extremely stupid argument.
I guess it’s good that no one is making that argument then.
that is the argument linus is making… he is comparing a deterministic tool (a compiler that outputs the same assembly every time) to a non-deterministic tool and saying that they are the same type of tool. that require the same level of review, when they do not.
instead of writing what you want directly you are writing tests for it.
LLMs write pretty good requirements, specifications, code, and automated tests from chat sessions. You do need to keep an eye on what it’s writing to make sure it’s not hallucinating, but hallucinations have been reducing in practice rather dramatically across the last 12-18 months of model development. They still happen, you will always need to guard against them, but usually a quick read of what is written followed by a quick test of the product will find the worst offenders. In the end, you need to check the final product to be sure it does EXACTLY what you want, just like you always did. And, it’s still a good idea (and as possible as ever) to keep the product modularized and also check every module to ensure it works as you want. How small you make those modules has always been a matter of preference / dictated by levels of concern, etc.
If you are generating tests with an LLM, it’s not deterministic.
Also, have you actually read anything an LLM wrote that’s longer than a hundered words? It’s usually immediately obvious. In the open source world things I keep seeing are exchanges like this one. And Niri is a fairly good case with it’s maintainer actually reading the PRs and catching all the bullshit that an LLM spits out. Very often nowadays people will also do reviews with an LLM and this sort of stuff will just accumulate in the codebase with nobody to reject it.
But even if it was as useful as big tech paints it to be, I’ll always be against using it based purely on morals. There is an enormous number of issues in that regard, most importatnly LLMs literally kill people both directly and indirectly. Condoning the adoption of this technology is either being ignorant or misanthropic.
If you are generating tests with an LLM, it’s not deterministic.
Once the tests are generated, they have become deterministic - unit test code is checked in alongside the repository, CI (continuous integration) systems can prevent the code from merging to main branch until all CI tests pass. No LLM is involved in the test execution process, and if all LLMs stop functioning tomorrow, the tests still work just like they did before.
read anything an LLM wrote that’s longer than a hundered words? It’s usually immediately obvious
About 250,000 lines of code so far, and no… I don’t read it all - my personal music player isn’t that important, I just care that it works. Other more important projects I have read every single line. Even in the music player I have read a few hundred lines and skimmed a couple of thousand. Yes, the style is obvious. It’s what you might call “too complete” attempting to cover lots of edge cases that a human would usually neglect, though as human code is maintained over the long haul, it tends to mature in that direction. And just because it’s very complete doesn’t mean it’s perfectly correct, and that’s a major point of conflict with me and LLM code, I’m constantly asking it to simplify because the complexity is easy to hide mistakes in. You know what happens when you ask it to simplify? It simplifies.
people will also do reviews with an LLM and this sort of stuff will just accumulate in the codebase with nobody to reject it.
That’s a management / maturity problem. As new as all this stuff is, it’s virtually inevitable there will be an adjustment process. Our company runs on processes and procedures, they average 50 pages each and there are thousands of them. I am given almost 100 of them to “train on” on a near annual basis - whenever they get revised. No way in hell can I do an accurate review of 5000 pages of intertwined procedures, not code, natural language work instructions. LLMs can. LLMs are. Simple questions to an LLM like: “review this procedure, is it internally self consistent” will find issues, real issues - usually minor - but undeniable when you read the review, the problem it describes is absolutely present in the document, page 14 says “X” and page 23 says “not X” and the context is basically identical - so which are we really supposed to do: “X” or “not X”? Well, that depends. Depends on what? Additional context and detail that the already too long 50 page procedure doesn’t address. We just did an internal audit which turned up thousands of these mostly little issues. Addressing them is going to be… interesting. We’ve been in business over 70 years, we know we’re not going to tackle it all in a single year, there will probably be a plan to work down the worst of the issues, then a second round for additional polish, of course corrective actions to ensure that we at least don’t make things worse along these metrics with future changes… I bet by 2035 we’ll have 90%+ of today’s real observations fully addressed.
LLMs literally kill people both directly and indirectly
As does nearly every activity of life, both in human society and the natural world.
Why the are you getting more text than you can physically review at your job? That sounds like a recipe for poor reviews. How are you sure LLMs are not missing anything if nobody actually has the time to read through all the text? Why does the text that gets to a developer contain logical mistakes in the first place? Back at my last job we had people assigned to collecting feature requests, talk them through with whoever wanted the feature implemented and produce proper specifications that developers would implement without wasting time on specifications containing garbage.
As does nearly every activity of life, both in human society and the natural world.
Yea, all people who drank water died. Not a very good argument. Most things don’t have a mental disorder named after them. If I leave a webpage with a calculator running unattended it won’t magically persuade a random visitor into commiting suicide. Webpages with LLM-powered chatbots are doing just that. You may choose to neglect it, but to me it sounds like adding lead to your gasoline. Yea, your car runs better. Are you sure it’s worth it, though?
Why the are you getting more text than you can physically review at your job?
Regulatory required documented training, old conservative company.
How are you sure LLMs are not missing anything if nobody actually has the time to read through all the text?
You’re not. Neither do the humans tasked with the job do a 100% thorough review. Side by side, the LLM catches more flaws than the humans do.
produce proper specifications that developers would implement without wasting time on specifications containing garbage.
Sounds nice. We’ve got ~500 employees at our “site” (which is now distributed across 3+ states with remote workers), of those, about 8 do software development, so developers end up doing virtually all of the translation of customer “needs” to specifications / requirements. We also have a marketing department that changes their mind about twice as fast as the product development / delivery cycle.
If I leave a webpage with a calculator running unattended it won’t magically persuade a random visitor into commiting suicide.
That’s a touchy subject. A friend of ours recently had her son die of drowing in his bathtub, he was epileptic, had a seizure and fell in the tub. There was no suicide note, it was ruled an accident. He used ChatGPT to write his own obituary, as well as a book explaining his life from his perspective - each seizure was “taking away a little of who he was,” and the seizures were getting more frequent, none of the treatments or medications slowed them. It’s pretty clear, to me, that he was going to commit suicide and/or die soon with or without ChatGPT. After he passed, his mother had a ton of grief / guilt about not being closer with him in his life, but then… he actively pushed her away, even violently. It’s doubtful he would have written or expressed what he did as completely without ChatGPT assistance - how much was him, how much was LLM hallucination? Does it really matter? He wrote the book, and he put it where she would find it.
The next time you go driving, count the oncoming cars. When you get to 100, realize this: statistically, one - maybe two - of the drivers of those oncoming 100 cars will, eventually, end their life by suicide. And those are just the deaths officially classified as suicide. Engagement with ChatGPT undoubtedly works both ways, some cases accelerating the process or even causing them to take their lives when they wouldn’t have, and other cases slowing or stopping the pursuit of suicide altogether. It’s going to be hard to say if Chatbots or other world events drive the statistical changes starting a couple of years ago. The data isn’t clear yet, but the concerns have certainly been raised, and I would assume the supportiveness of Chatbots should improve as a result in the future.
I’m sorry you are working in such conditions, I hope you get better management that actually understands how much data one person can process.
I’m not sure how your story is at all relevant to what I said.
I don’t have a driver’s license, I don’t live in the US, I don’t need one. The lead in gasoline example was just the first thing that came to mind that fit a “technology that is profitable and useful, but kills people” description. Please pay attention to the road while driving instead of perfoming mental math exercises.
I would also like to ask you to think about what exactly you are arguing for right now, as I’m starting to think that our understanding of the value of human life differs quite a bit.
people who were a really good programmers before will probably use it responsibly for a while, until their skills inevitably erode
Damn, dude, I had mad 6502 assembly skillz in 1985, I could even write it out by hand - pencil on paper, and those slowly eroded through use of compilers for higher level languages until my last assembly experience in 1996, today… I’d be learning everyhing specific to assembly code from scratch. I guess I’m just useless now.
the people I respect and call intelligent are not the ones that shy away from using new and interesting tools.
Gotta agree there. We had two clear camps of programmers before LLM agents came along, and the camp I least enjoyed reviewing the code of before (and now) of is the camp that resisted adoption of new interesting tools the most loudly.
I think there is no equivalency between compilers and other tools, and LLMs. As the comment states, they are just the statistical prediction of what would come next. Compilers may sometimes give unexpected outcomes, but they are working to a defined set of rules and there is potential to see how they actually came to that outcome and address it.
Added to this is the prevalence of people thinking that LLMs can allow them to do anything, whether or not they understand it, it’s even sold like that. In my own organisation I have an operations manager thinking he can make pages for our Intranet - where does it store the data? How is access controlled? So many basic questions that an actual programmer would know to think of. Sure you can just keep asking people if they can fly a plane and maybe one will do well enough that they get you all the way there, till it comes to landing and it’s too late - maybe better to just ask a pilot to do it.
Our software engineers use it really well, they can read what it’s outputting to understand if it’s wrong, they know how to prompt it to avoid issues, they use it for snippets and fixes and that it won’t just write a whole program. However, it’s benefit to us (and the world) is nowhere near worth the monetary and environmental cost. The investors pouring in money are hoping they can be the company that people become reliant on, so they can survive the bubble and their ultimate goal of replacing real people.
“Deterministic” compilers and reproducible builds are actually fairly difficult problems. (Yes I know technically the compiler is deterministic, but AI is also technically deterministic since you can set the seed).
It’s not just about being deterministic. A compiler crucially follows clearly defined formal rules on what is a correct output given some input, which the programmer can fully rely on.
Lol, I could kind of understand (not really support) his use of AI. While I doubt it’s something you can do for long without starting to make compromises, people who were a really good programmers before will probably use it responsibly for a while, until their skills inevitably erode and (more importantly) their willpower slowly runs out.
But this argument made me loose all my respect for him. It’s so utterly stupid. I hate it when i have to check my assembly code line by line, because it’s not guaranteed that the compiler will compile the same code I’ve written deterministically, and basically just randomizes what it outputs. Or when my editor does not save what I typed, and instead decides based on a random language statistics what to actually save, so I have to double-check my text word-by-word every time.
Oh, wait. No, because the tools are deterministic, and that doesn’t happen. But of course, adding randomness into your workflow, where you have to focus and review everything line-by-line, is the same as using deterministic tools, lol.
His point is that an LLM is just another tool for increasing productivity, the same as a compiler or IDE. The quality of the code is the only metric that should be judged.
And the person you replied to is saying it is not just another tool because tools are deterministic.
Except they’re not? Very few tools are deterministic. Only Mathematics is, and that’s only a small subset of tools.
You’re free to argue that non-deterministic tools lack value, but that’s clearly false: OCR only exists because of lossy training machine learning algorithms. Or, going even more basic, Monte Carlo simulations are by far the most efficient method of testing entire categories of problems that are intractable deterministically.
LLMs can be shitty, as a business practice, due to their unpredictability, but (hopefully obviously) that depends on how it’s used.
I’d argue that Monte Carlo is a bad example here, and is an example of a deterministic tool.
You use a tool to create an outcome, you’re looking for an answer or a solution that you can trust.
If you want to examine behavior of a system that has input variables across some known range, you could brute force it, try every single possible combination and wait a millennia. Or you use a monte carlo strategy, let randomness enter the simulation to build out a basic characterization. You want more detail? More confidence? Increase the number of runs. I can trust that if I increase the number of runs I gain higher fidelity, if the tool/strategy was non deterministic I couldn’t do that
Reviewing AI code is harder than reviewing human code. I’ll judge humans on what they do, and LLMs on what they are.
Do you not think all code deserves this level of focus?
Not parent commenter, but no, not to the same degree.
The level of trust I have in a source greatly changes how I review it. If I know that I’m reviewing code from a competent developer, I’m going to see a for loop and look for why is this here, how does it play into the larger objective and if I’m happy with it, I move on. But if I don’t trust the developer as much, I’m looking at far more like what data type is being used in the indexing, what’s the break condition, can go off the rails into random memory.
Competent developers make mistakes too. I think it’s worth at least skimming every line in a PR. Things like indexing out of bounds should probably be handled by testing or using a memory safe language.
I don’t think code review is the place where you should be trying to catch bugs in the minutia of the code. You should be assessing readability/maintainability, architectural decisions, risk, and testing strategy. That does involve reading all of the code, even if only by skimming.
I guess I don’t really count skimming as reading haha
When looking for readability, maintainability, etc what I mainly look for are design patterns. If you’re using the same boilerplate and layout I’m going to start skipping over sections
I also mainly work on high performance embedded systems that are difficult/nearly impossible to unit test and I dont always have line coverage metrics. A lot more verification by inspection in my corner of the world
Sort of a change in topic, but I’m curious why that would be so difficult to test.
There’s a couple big reasons, sometimes you only have to deal with one or two, other times you’re hit with all of them.
The data source for the algorithms often comes from arbitrary memory addresses that live outside the context of the C, C++, or Rust runtime. If I want to read CPU core temp I send magic numbers to one address to configure ADC channel conversion, another magic number somewhere else to do the conversion, wait a few dozen microseconds, then read a third new address that I squint one eye, spin around 10 times, and do a handstand and I’ll get a number that looks like temperature.
If I’m running on the microcontroller, I’m essentially in production environment and I can’t control what the data is. If I recompile for a different system (say x86 Linux) then I’m almost certainly going to segfault. I could abstract out the reading of the data, but then on my tiny 8MHz microcontroller that has to run it’s main loop at 1000Hz, I only have a few thousand clock cycles to get everything done, calling functions, pushing/popping the stack takes a lot of time.
Adding to the pile-on most microcontrollers have <1MB of RAM, many in the tens of kilobytes range so I don’t have much of a stack to work with and I need to keep even the number of instructions down.
Even if you do all the abstractions, you then have extra logic you have to write to simulate what is happening and subtle behaviors that are timing dependant. It frequently is extremely difficult to do that as the documentation rarely fully describes everything, so you can lull yourself into a false sense of security that you’ve tested the code and it’s correct, but then you get stuck on a register value that never changes and a watchdog timer comes along and resets the hardware on you.
And that’s not the getting into the runtime or build environments, flashing procedures, limited flash cycles, and a million other business constraints that technically you can do it, but at that point your dev team has all taken the month off to go play Half Life 3
That definitely increases the challenge of writing tests or trusting that they are faithfully representing your production environment.
I was going to suggest this, and it sounds like that ends up being expensive. It seems like you need some kind of test harness that’s able to emulate your specific hardware while still allowing for testing hooks that don’t impede the tight clock scheduling.
If unit tests are being done, some kind of emulator is usually what gets used and the scope of testing is significantly limited.
The more common approach is to abandon unit tests all together and go for board functional tests where as many of the external interfaces are controlled and put through representative loads. The complexity of the software is rarely that high so we can be pretty confident that we’re covering everything we need to care about anyways.
You do that inherently when you are writing the code. Every line you write has already had that focus, because you had to come up with it.
That you should also not trust every line of the compiled assembly after you wrote the code, and check it lime by line, is an extremely stupid argument.
Of course, you should get someone else, with a fresh/unbiased opinion to review your code too, but just the act of figuring out every line you write is in some sense a rewiev, that there is at least a plan or reason why it was written.
AI just randomly generates code without thought or reason, and you have to check everything. Unlike assembly generated from code.
I guess it’s good that no one is making that argument then.
that is the argument linus is making… he is comparing a deterministic tool (a compiler that outputs the same assembly every time) to a non-deterministic tool and saying that they are the same type of tool. that require the same level of review, when they do not.
You can add deterministic testing to AI output though?
Wouldn’t it be just… writing code yourself? It’s just instead of writing what you want directly you are writing tests for it.
LLMs write pretty good requirements, specifications, code, and automated tests from chat sessions. You do need to keep an eye on what it’s writing to make sure it’s not hallucinating, but hallucinations have been reducing in practice rather dramatically across the last 12-18 months of model development. They still happen, you will always need to guard against them, but usually a quick read of what is written followed by a quick test of the product will find the worst offenders. In the end, you need to check the final product to be sure it does EXACTLY what you want, just like you always did. And, it’s still a good idea (and as possible as ever) to keep the product modularized and also check every module to ensure it works as you want. How small you make those modules has always been a matter of preference / dictated by levels of concern, etc.
If you are generating tests with an LLM, it’s not deterministic.
Also, have you actually read anything an LLM wrote that’s longer than a hundered words? It’s usually immediately obvious. In the open source world things I keep seeing are exchanges like this one. And Niri is a fairly good case with it’s maintainer actually reading the PRs and catching all the bullshit that an LLM spits out. Very often nowadays people will also do reviews with an LLM and this sort of stuff will just accumulate in the codebase with nobody to reject it.
But even if it was as useful as big tech paints it to be, I’ll always be against using it based purely on morals. There is an enormous number of issues in that regard, most importatnly LLMs literally kill people both directly and indirectly. Condoning the adoption of this technology is either being ignorant or misanthropic.
Once the tests are generated, they have become deterministic - unit test code is checked in alongside the repository, CI (continuous integration) systems can prevent the code from merging to main branch until all CI tests pass. No LLM is involved in the test execution process, and if all LLMs stop functioning tomorrow, the tests still work just like they did before.
About 250,000 lines of code so far, and no… I don’t read it all - my personal music player isn’t that important, I just care that it works. Other more important projects I have read every single line. Even in the music player I have read a few hundred lines and skimmed a couple of thousand. Yes, the style is obvious. It’s what you might call “too complete” attempting to cover lots of edge cases that a human would usually neglect, though as human code is maintained over the long haul, it tends to mature in that direction. And just because it’s very complete doesn’t mean it’s perfectly correct, and that’s a major point of conflict with me and LLM code, I’m constantly asking it to simplify because the complexity is easy to hide mistakes in. You know what happens when you ask it to simplify? It simplifies.
That’s a management / maturity problem. As new as all this stuff is, it’s virtually inevitable there will be an adjustment process. Our company runs on processes and procedures, they average 50 pages each and there are thousands of them. I am given almost 100 of them to “train on” on a near annual basis - whenever they get revised. No way in hell can I do an accurate review of 5000 pages of intertwined procedures, not code, natural language work instructions. LLMs can. LLMs are. Simple questions to an LLM like: “review this procedure, is it internally self consistent” will find issues, real issues - usually minor - but undeniable when you read the review, the problem it describes is absolutely present in the document, page 14 says “X” and page 23 says “not X” and the context is basically identical - so which are we really supposed to do: “X” or “not X”? Well, that depends. Depends on what? Additional context and detail that the already too long 50 page procedure doesn’t address. We just did an internal audit which turned up thousands of these mostly little issues. Addressing them is going to be… interesting. We’ve been in business over 70 years, we know we’re not going to tackle it all in a single year, there will probably be a plan to work down the worst of the issues, then a second round for additional polish, of course corrective actions to ensure that we at least don’t make things worse along these metrics with future changes… I bet by 2035 we’ll have 90%+ of today’s real observations fully addressed.
As does nearly every activity of life, both in human society and the natural world.
Why the are you getting more text than you can physically review at your job? That sounds like a recipe for poor reviews. How are you sure LLMs are not missing anything if nobody actually has the time to read through all the text? Why does the text that gets to a developer contain logical mistakes in the first place? Back at my last job we had people assigned to collecting feature requests, talk them through with whoever wanted the feature implemented and produce proper specifications that developers would implement without wasting time on specifications containing garbage.
Yea, all people who drank water died. Not a very good argument. Most things don’t have a mental disorder named after them. If I leave a webpage with a calculator running unattended it won’t magically persuade a random visitor into commiting suicide. Webpages with LLM-powered chatbots are doing just that. You may choose to neglect it, but to me it sounds like adding lead to your gasoline. Yea, your car runs better. Are you sure it’s worth it, though?
Regulatory required documented training, old conservative company.
You’re not. Neither do the humans tasked with the job do a 100% thorough review. Side by side, the LLM catches more flaws than the humans do.
Sounds nice. We’ve got ~500 employees at our “site” (which is now distributed across 3+ states with remote workers), of those, about 8 do software development, so developers end up doing virtually all of the translation of customer “needs” to specifications / requirements. We also have a marketing department that changes their mind about twice as fast as the product development / delivery cycle.
That’s a touchy subject. A friend of ours recently had her son die of drowing in his bathtub, he was epileptic, had a seizure and fell in the tub. There was no suicide note, it was ruled an accident. He used ChatGPT to write his own obituary, as well as a book explaining his life from his perspective - each seizure was “taking away a little of who he was,” and the seizures were getting more frequent, none of the treatments or medications slowed them. It’s pretty clear, to me, that he was going to commit suicide and/or die soon with or without ChatGPT. After he passed, his mother had a ton of grief / guilt about not being closer with him in his life, but then… he actively pushed her away, even violently. It’s doubtful he would have written or expressed what he did as completely without ChatGPT assistance - how much was him, how much was LLM hallucination? Does it really matter? He wrote the book, and he put it where she would find it.
The next time you go driving, count the oncoming cars. When you get to 100, realize this: statistically, one - maybe two - of the drivers of those oncoming 100 cars will, eventually, end their life by suicide. And those are just the deaths officially classified as suicide. Engagement with ChatGPT undoubtedly works both ways, some cases accelerating the process or even causing them to take their lives when they wouldn’t have, and other cases slowing or stopping the pursuit of suicide altogether. It’s going to be hard to say if Chatbots or other world events drive the statistical changes starting a couple of years ago. The data isn’t clear yet, but the concerns have certainly been raised, and I would assume the supportiveness of Chatbots should improve as a result in the future.
I’m sorry you are working in such conditions, I hope you get better management that actually understands how much data one person can process.
I’m not sure how your story is at all relevant to what I said.
I don’t have a driver’s license, I don’t live in the US, I don’t need one. The lead in gasoline example was just the first thing that came to mind that fit a “technology that is profitable and useful, but kills people” description. Please pay attention to the road while driving instead of perfoming mental math exercises.
I would also like to ask you to think about what exactly you are arguing for right now, as I’m starting to think that our understanding of the value of human life differs quite a bit.
Damn, dude, I had mad 6502 assembly skillz in 1985, I could even write it out by hand - pencil on paper, and those slowly eroded through use of compilers for higher level languages until my last assembly experience in 1996, today… I’d be learning everyhing specific to assembly code from scratch. I guess I’m just useless now.
Gotta agree there. We had two clear camps of programmers before LLM agents came along, and the camp I least enjoyed reviewing the code of before (and now) of is the camp that resisted adoption of new interesting tools the most loudly.
I think there is no equivalency between compilers and other tools, and LLMs. As the comment states, they are just the statistical prediction of what would come next. Compilers may sometimes give unexpected outcomes, but they are working to a defined set of rules and there is potential to see how they actually came to that outcome and address it.
Added to this is the prevalence of people thinking that LLMs can allow them to do anything, whether or not they understand it, it’s even sold like that. In my own organisation I have an operations manager thinking he can make pages for our Intranet - where does it store the data? How is access controlled? So many basic questions that an actual programmer would know to think of. Sure you can just keep asking people if they can fly a plane and maybe one will do well enough that they get you all the way there, till it comes to landing and it’s too late - maybe better to just ask a pilot to do it.
Our software engineers use it really well, they can read what it’s outputting to understand if it’s wrong, they know how to prompt it to avoid issues, they use it for snippets and fixes and that it won’t just write a whole program. However, it’s benefit to us (and the world) is nowhere near worth the monetary and environmental cost. The investors pouring in money are hoping they can be the company that people become reliant on, so they can survive the bubble and their ultimate goal of replacing real people.
“Deterministic” compilers and reproducible builds are actually fairly difficult problems. (Yes I know technically the compiler is deterministic, but AI is also technically deterministic since you can set the seed).
It’s not just about being deterministic. A compiler crucially follows clearly defined formal rules on what is a correct output given some input, which the programmer can fully rely on.