What It Do?
The great and eccentric systems theorist Stafford Beer, one of the father’s of modern management consulting had a heuristic maxim:
“The purpose of a system is what it does. There is after, all no point in claiming that the purpose of a system is to do what it constantly fails to do.”
Let’s say you develop a product called the FeetsyFeeler 3000™ which you market and sell as a 'foot massager.' However, instead of massaging feet, the FF3K (patent pending) breaks 90% of a user’s toes (in 80% of all use cases) as opposed to massaging their feet. By Beer’s heuristic maxim, it is more accurate to describe the FF3K as a ‘toe breaker,’ and not a ‘foot massager.'
The U.S. ‘healthcare system’ is another example to which we can apply this heuristic. It’s a system which purports to heal people, but its most consistent output is instead generating bankruptcy-inducing medical bills. It is therefore not correct to call it a ‘healthcare system,’ when it’s actually a ‘wealth transfer system with occasional nominal medical benefits if you get lucky and aren't particularly sick.'
Consider a streaming music service that claims to ‘support artists’ but that only pays them $0.0000003 per stream of their recordings. By Beer’s maxim, it is not actually a ‘music streaming service,’ it's a ‘highly efficient mechanism for converting music fandom into shareholder value.’
Stafford Beer’s maxim is a way to say don’t bullshit and don’t doublespeak: Either take the time to build something that does what you say it does, or don’t bother to build anything at all.
As a general technology nerd, I love trying new products and I’ve previously written about the bizarre philosophies within the political economy of modern product development. My proximity to the technology industry writ large over 12+ years in my professional career has often allowed me the privilege of beta access to new products prior their launch, either as a consultant, an advisor, or as a known quantity of some (marginal) taste and (semi) decent ideas within start-up circles.
Most of the time, the people and companies asking me to test their products sincerely want to know if their product works in the way the describe it. They are honest and want to adhere to Beer’s maxim.
Then there are the others.
I recently gained access to a small, sought after beta program for a new SaaS product that will hereafter be referred to as ”The System.” The System claims in its description to go far beyond the means of a traditional AI large language model and reach a deep level of personalization that allows the user to access, in real-time, forgotten pieces of their documents, messages, memories, emails, schedules, and anything else connected to The System’s API. The ultimate knowledge recall tool with perfect memory and predictive capabilities.
Products such as Perplexity or the web-search functions within ChatGPT and Claude are also RAG-based system architectures.
After meeting some members of the team last year and becoming interested in their work, I received an invitation to a 1-on-1 onboarding of their beta test earlier this year.
The signs, omens, and portents for The System’s failure as a product to actually do what it was described as doing should have been clear from the jump - principally their highly stylized and very expensive-looking website with sophisticated motion graphics and a trendy asymmetrical design which made promises of functions that seemed both highly impressive and highly implausible for a company of it’s size.
You can call that 'Jaffe’s Maxim.’
Before joining the onboarding call, I received a Google Form asking me what services I would like to connect to The System’s API in order to best suit it for my personal use. This was the next red flag and it was amongst the biggest screaming fire-engine red flags that I have ever seen. The optional servicecs for connection to the System’s API included the following: Google Drive (!), Google Calendar, Gmail (!), Notion, Slack (!), iMessage (!), WhatsApp (!), Apple Notes, Spotify, and any Chrome-based web browser via an extension.
Now, I don’t know about you, but I treasure what few meager shreds of digital privacy I still retain under modern surveillance capitalism and I opted to select only Notion, where I organize and draft essays, and the browser extension which I could toggle on and off as I pleased.
BECAUSE WHO IN THEIR RIGHT MIND WOULD GIVE READ AND WRITE ACCESS TO ALL OF THEIR MOST INTIMATE PERSONAL COMMUNICATIONS INCLUDING EMAIL, TEXTS, AND WORK DMS TO A RANDOM START-UP?
Sigh.
Despite the glaring and obvious privacy violations baked into The System before I had even tried it, I figured at the least I could test The System as a research tool for my writing without any real harm being done. No one could do anything with my Notion data beyond surmise what an insane person I must be and how difficult it must be to live inside of my mind.
During my personal video onboarding with The System employee, it became immediately clear that not only did The System not function as described - it didn't even function at the level expected of a minimum viable product beta test.
Having shepherded numerous products to market myself, I understand how a beta test is meant to work, with all of the attendant imperfections and bugs that a beta test implies. A product in beta is by definition, unfinished, and in need of user testing. The System, however, was barely in an alpha state for internal testing, let alone a beta state for external testing.
After exchanging pleasantries, I asked several basic questions: When did they start working on the product? Who was their target customer? What was the size of their team? Had they raised any funding or were they purely bootstrapping?
And most importantly, since The System was obviously a form of RAG: What large language model was being used for their foundational model, since a company of their stage and size could not possibly have trained their own?
For the last question, I assumed that The System was using an open-source model like Meta’s Llama, and I asked them this directly. In response I was told that the underlying model could not be legally disclosed because their "investors want to keep it a trade secret” and that The System was using “unique and proprietary technology.”
This told me two things were true immediately:
- The System didn’t actually have much, if any, actual proprietary technology of their own; and
- The System was likely just a “wrapper” for a vastly more established enterprise AI product.
In practice, what this means is that if you use The System’s API connection, The System in turn sends your information on to another system's endpoints using that other system’s API. This is not inherently a bad thing. There are lots of reputable products and services that operate in this way and at scale. But because The System is asking to handle reams of both confidential and personally identifiable information, this is is not something they should be hiding.
This type of architecture would allow for a “fourth party” to enter the equation of my most intimate personal data without my knowledge or consent. The totality of the parties in this case being 1) me, 2) the product/services I personally use and connected to The System, 3) The System itself, and 4) the other entity who’s technology The System was built upon, but who they would not disclose.
Despite my questions remaining mostly unanswered, it was now time for me to connect to The System.
To begin, The System employee began by completely disregarding my pre-meeting Google Form responses and directing me to connect my iMessage and WhatsApp accounts so that I could “easily search my text message history for key insights into my conversations and group chats.”
When I replied that I was absolutely not doing that, the employee seemed both surprised and confused. They asked me, "Why not?"
After gently explaining that I did not feel comfortable sharing my personal message data for what I feel are rather obvious broadside-of-a-barn reasons, the employee proceeded to tell me that other beta testers had been uploading things like their medical bills, bank statements, and tax documentation, again for “key insights.” I assume at this point that my eyes had bugged out of my head like shrunken head guy from Beetlejuice, but thankfully they did not press me on the point any further and asked me what I wanted to connect The System to instead of my iMessage and WhatsApp.
Perhaps I would like to connect my Gmail? Instead of again further explaining why I was still not going to do that, I disregarded the suggestive question entirely and asked to start by connecting a Notion Page and downloading the browser extension that would allow me to forward webpages to The System’s API for future reference and query within The System.
The technical struggles began immediately when neither the Notion connection nor the browser plugin proved to be functional. As the employee nervously engaged in some live messaging with their backend engineering team - who were ostensibly monitoring the onboarding process for technical support - I was assured that I had indeed connected to The System, even though the UI showed no connection to either Notion or my web browser. I was asked to sign out and sign back in to resolve the issue, but once I did, the connections were still not visible.
At this point, The System's unfinished and non-functional demo state visibly unnerved the employee. Their nervousness became so intense so quickly that, in a startling burst of anxious verbal diarrhea, they began inadvertently revealing the confidential "trade secrets" about The System, which they had just claimed minutes earlier could not legally be disclosed.
They apologized profusely and said that their backend had been running more slowly for the past week because OpenAI had been throttling third-party API requests to ChatGPT.
The System was indeed nothing more that a web-based front-end wrapper for OpenAI’s ChatGPT API. Any information that I, or anyone else shared with The System was not only going to The System, but also to OpenAI.
I assume at this point I was barely concealing my bemusement, but I held it together enough to try and finish the demo, because I had still not actually seen what The System could (allegedly) do. I guess you could say I was morbidly curious.
First, since it was apparently connected despite not being visibly so, I tried to query my Notion with the prompt, “What was my most recently drafted essay?” a question that The System should surely be able to answer given what it claimed to do on a very basic level. (Not to mention that all file management systems have done this with one click sorting by most recent date or edit for the past 30-ish years - it's literally in the metadata). Notion’s database would have a timestamp of both my most recent creation as well as my most recent edits, so I assume The System would return either one or both of these options in reply.
It returned neither option.
Instead it reported that I had no recent activity in Notion, even though I had literally been updating a Notion page not five minutes prior to the call.
Next, I asked if I could upload some PDFs directly to The System and store them for query, which I was assured I could.
I then attempted to upload a PDF of Yancy Stricker’s excellent essay The Post Individual and watched as The System failed three consecutive times to accept the PDF, as the employee, presumably now with very sweaty palms and in desperate need of deodorant, asked me to try uploading one final time.
On the fourth attempt, the PDF uploaded successfully and was visible in crude list of uploaded files hidden in one of the menus. But when I went to prompt The System again, asking it to “Please summarize the PDF I uploaded today,” The System told me that I had not uploaded any PDFs today. The employee asked me to double check the list of files just in case. When I did, there were now four copies of The Post Individual in my library.
By now, the employee had clearly had enough of The System’s obvious failures to do quite literally anything it was described as being able to do, and abruptly told me that the demo was now over. Before signing off, they thanked me for my time emphasized that I should “keep trying” and email them any feedback.
Over the next couple of weeks, I occasionally checked back in on The System, which eventually did take PDFs on the first try, but also continued to give me non-responses related to prompt queries about my documents, and was seemingly never able to parse my Notion or any of the handful of website which I sent via their browser plug-in.
This is not just a story about a single bad product. It’s a parable for the complex psychology and political economy that molds how new technology is developed, marketed, and monetized.
The universally derided “Move Fast and Break Things” ethos of early Facebook and techbro wannabe LinkedIn thought leadership work-fluencers everywhere, has only ever resulted in broken things. Shocking, I know.
The employee’s nervousness during my demo of The System wasn’t just run-of-the-mill demo anxiety, it was an expression of the underlying systemic precarity upon which a product must be built and released, forcing people like the employee to pretend that the emperor is still wearing clothes so that they do not lose their jobs.
This is a precarity built primarily on the bad incentive structures of venture capital investment in technology which demand growth on timelines which are often physically and temporally incompatible with the actual means of production.
It's less about creating good products than it is about creating the impression of good products.
The race of bad products to market isn’t a bug of the technology funding ecosystem, it is is primary and actual function of that ecosystem: To bring products to market no matter what, as quickly as possible, even if they do not work as described.
This machinery of of this epic dysfunction operates with the ruthless efficiency of capital - venture funds need returns within specific windows (usually 3-5 years for early investment rounds), founding teams need to hit arbitrary growth metrics by arbitrary dates unmoored from actual progress, and employees need to participate in collective Kool-Aid drinking delusions that their half-baked products are not only ready to use, but are also revolutionary and disruptive.
It’s no longer the age old debate of “Form Over Function.”
It’s become “FOMO Over Function” as investors fear missing the next big thing in the hype cycle, founders fear losing their funding, employees fear missing their next paycheck if they don't keep the puppet show going, and the only real product ends up being a sexy pitch deck and slick website.
The purpose of modern tech funding is not to fund the building of novel and useful products. It's purpose is to create exit opportunities and liquidity events, with actual products as a sort of happy accident and occasional byproduct.
The System itself is less a product than a Russian nesting doll of misaligned incentives in an era of mostly derivative innovation. We live in a time where nothing is what it seems and where most every digital product is merely a different method to mine your data in order to monetize you far beyond whatever fees you're charged for using said product.
Why create when you can aggregate?
As long as you can be a funnel for the upward suction of data, it doesn’t really matter where in the data stream you are, because even at the bottom, there will always be more funnels being built below you, placing you higher up on the value chain.
Most tellingly, the pieces of data which would be aggregated into The System if you were to actually connect your personal emails, text messages, and search history - effectively your entire digital life - are the pieces of data that are still, depending on your paranoia level, not within the training data of large language models.
Which makes it the equivalent of digital gold.
Data too intimate to share is the last undiscovered resource buried on the billions of asteroids floating in cyberspace waiting for extraction. The suggestion that people have been uploading their taxes and medical bills into The System wasn’t a use-case recommendation, it was a prospector eyeing mineral rights.
So what IS the purpose of The System? What does it DO?
The System is an elaborate Rube Goldberg consent-manufacturing mechanism designed to normalize the scraping of your most intimate communications in exchange for conveniences that never materialize.
This is a cautionary tale, and a call to embrace Beer's heuristic maxim as both an ethical compass and a likely competitive advantage in the marketplace. Build things that actually do what you say they do, and describe your products as what they are, not as you imagine them to be.
The products that survive aren’t be the ones with the coolest asymmetrical websites or the most impressive-sounding capabilities. They are the ones that actually work. They are the ones that respect customers enough to be honest about what they are. They are the ones that align what they claim with what they consistently do.
Otherwise, we will just keep getting this. Or this. And whatever this is.
As for "The System"? The smart money would bet on a quiet sunset along with the plethora of other GPT wrapped products that have launched in the past two years. I’d say that I will check back in a few months to see where they are at, but to be honest I’ll probably just forget they exist like every other half-baked and overhyped product I have ever tried in both my personal life and professional capacities.
In a world increasingly defined by systems that do something other than what they claim to do, maybe it is a revolutionary act to simply call things what they are.
A toe-breaker, not a foot massager.
A wealth transfer system, not healthcare.
An accretive shareholder-value machine, not a music streaming service.
A poorly repackaged piece of ChatGPT surveillance middleware that promises to remember everything while consistently remembering nothing.
Sometimes the most radical truth is the most obvious one:
The purpose of a system is what it does.
⛳️ Hit The Links
In other news, rich people have discovered buttons and switches!
Alexandria Abramian
America is truly just being able to spend $150 to take a family of four to the movies without being interrupted by TikTok trends.
Zack Sharf
🎧 Free Your Earballs
Röyksopp have made what is probably the best live electronic music album since Daft Punk's Alive 2007 or Justice's A Cross The Universe. A true banger in every sense, spanning genres and styles with each track creating new pathways to make you scream for more like a 14 year old girl at Shea Stadium awaiting The Beatles.
If this does indeed get your goat, I would also suggest checking out their 2022 triptych of albums, Profound Mysteries I, II, and II which went largely missed by the culture at large.
