This is an automated archive.
The original was posted on /r/singularity by /u/Xtianus21 on 2024-01-14 04:07:38+00:00.
Original Title: AI Researchers Write A Paper Ultimately Proving They Have No Clue About How Modern Day AI Works - GPT 4.5/5 Will Be The End Of Data Science As We Know It - AI Winter Revisited - The AI Revolution Will Be Televised
I don't say this lightly but there is a battle/war going on in the enterprise right now. Moreover, there are wild statements by an EA advocate suggesting we should all Get Ready for the Great AI Disappointment. Why is EA so hell-bent on curbing AI's advancement? But I digress.
On one side, you have large-scale IT teams made up of Engineers, PO's, PM's, Designers, Stakeholders, BA's, Managers, etc.
On the other side, you have data scientists and AI/ML DS teams. As of 12 months ago, let's call it BG3 (Before GPT3), there were very limited influences by AI led organizational segments in enterprise. This means, to put it bluntly, those segments weren't really doing much beyond their daily activities of report building, data analysis, and prediction algorithms for certain business segments. Yes, there was some slight AI/ML model building, but I would argue it was ineffective and limited in scale at best.
There were SOME teams that would do other things that would take advantage of the transformer technologies such as NLP and data extraction type projects. At the time, those projects had the most real-world impact beyond data reporting and analytics. Keep in mind, tools like GPT and LLMs today are very much low-code/no-code. Simply put, you don't have to be an AI researcher guru to use them, even though, they WOULD LOVE to make you think that you do.
I know this situation speaks to probably 90+% of most IT departments out there. If I'm wrong please let me know in the comments but I don't think I am.
Before I get into the absurdity of the paper in question results, I want to relive what it was like just days prior to GPT 3 being released. TLDR: not a lot of everyday people gave great care about AI or noticed that it was just in much of our everyday lives - think, social media.
I'm not saying the profession isn't a great profession but the tangible value was limited at the time. It was challenging to produce something of value. These are just facts.
The level of compute and expertise that goes into creating a quality and market valuable model is very difficult to pull off. Mostly because you need data but also you need the expertise of the AI/ML engineer to be way above average. And we know this not to be the case in most enterprises all over the planet. Specifically, you can get really mediocre models which lead to frustrations by CTO's and stakeholders.
Unbelievably, when asked which jobs they think will be the first to be impacted by LLMs like GPT, AI researchers overwhelmingly believe that their jobs are the safest. They're not and I'd argue they will be some of the first to go.
Lol you can't make this stuff up. Also, truck driver? LOL, where the hell are truck drivers going? Even in a self-driving truck I still want a damned truck driver in there. Maybe, they do other things than just drive a truck; but hey, why would you know that. Install wiring in a house? Huh? Has there been an advancement in home wiring that I don't know about?
This is the fundamental concern with AI Researchers being christened & "crowned" to lead this type of technology at all. These groups aren't like Oppenheimer or Steve Jobs; they're researchers and data scientists. They're jobs aren't to innovate on a task or a product. That is firmly in the hands of people who are of the process and know about the process/problem. This is why innovation many times over is done by people who are in need of a solution to a problem. They go to the researcher to see if the byproduct can be used for their specific problem.
Very simply stated, 99.9999999999% of AI Researchers/Data Scientists have nothing to do with what the tacit level creation that OAI, Anthropic, Meta, Microsoft, Mistral and Apple (maybe) have created.
This is what makes using foundational models from OpenAI so freaking attractive. I implicitly understand the world's best AI/ML researchers creating a product that I should just use. In opposition, I don't want to be fine tuning on a limited and or shit data set that may or may not even be there is no longer an attractive prospect.
The writing is on the wall here; fine-tuning will soon become obsolete. What does your AI/ML Researcher DS person do if they don't do that? It's called General AI for a reason and it's like they don't get it or just are refusing to get it perhaps.
Just think about 15 years ago compared to today. In tech terms you might as well of been talking about 100 years ago. The decline of system administrators is well documented to the point where people can make a career of cleaning up the last remnants of on-prem server / data centers and moving that infrastructure to the "cloud".
How are we thinking that data scientists and AI shops in-house won't go the same way of the on-prem sysadmin?
In speaking with many DS people they have all explicitly said - there are no more models to create and that is clearly where all of this is going. However, in certain enterprise circles (AI leadership) they are closing ranks and don't want to hear it. GPT-4.5/5 will make them hear it you can guarantee that.
You're getting the GPT-4 isn't as accurate, it hallucinates, my F1 score is better with my training. There's no way to prove that out. You can't question or bring light to the methodology they've employed on their work. Imagine sitting through a presentation where someone compares GPT-4 versus a fine-tuned model in pure statistical outcomes.
Then, when you bring up RAG and how actually planning out your pipeline and data leads to very low and or non-existent hallucinations they don't understand that a proper RAG data design can lead to better results than just shoving in prompts and hoping for the best.
It's like someone saying that on-prem data centers are more efficient than cloud and the way I will prove it to you is by showing you SLA uptime charts versus AWS/AZURE uptime SLA agreements. Or better yet, I will show you throughput from an on-prem server versus throughput via the cloud. Remember, in this example it's the same engineer who is at risk of becoming irrelevant showing you this chart and purposely not using the litany of features that come with horizontal or vertical scaling in cloud architecture.
For me, watching this presentation I would want to have a lot detail about what your infrastructure is onprem versus what you employed via the cloud. Did you use a basic free instance to show your throughput analysis? I have no clue what are apples and apples in this situation. Most likely, it is apples and oranges.
The exact same pain but worse (IMHO) is playing out today with AI/ML. You have a group of people who are adjacent to said technology and think they are complete control of said technology while at the same time disparaging it when it doesn't fit their narrative. "this model isn't as good as mine look" - When it is completely not the case and who the hell can even know because you won't allow people to really know. To this date of I have taken down about 5 models because they just weren't good in their results.
I guarantee this exact nonsense is playing out in many IT organizations across America because self-preservation is a helluva drug.
AI Winter: https://medium.com/the-modern-scientist/the-ai-winters-17c7e7d21729
https://en.wikipedia.org/wiki/AI_winter
The Calm Before The Storm
The AI Winter i'd argue wasn't just 2 times in the 60's - 70's and 80's - 90's.
I'd take it a step further. The AI Winter was much, much longer. I'd argue it was from the 2010s to 2017 and then again from 2017 to 2022. The money may have been pouring in but the results were medicore and very specific to say the least. Yes, there was VR and self-driving cars but really outside of social-media algorithms what the hell was really AI?
But why would I said that we were in an AI winter in 2017 - 2022? Even though, the glorious paper 'Attention is All You Need' was released there still wasn't anything that was "groundbreaking" until well you know what happened.
Even as of 2021 there where people like Michael I. Jordan that were rallying around the new AI idea of not calling everything AI, as per his paper titled 'STOP CALLING EVERYTHING AI'.
Look at these hype cycle graphs to just show where we were prior to 2022/2023.
This peak hype-cycle was from VR and self-driving cars... Where are those today?
What the hell is Smart Dust?
