Mrs. Donoghue's Last Bottle

On 26 August 1928, a Glasgow woman named May Donoghue walked into a café in Paisley, a town about seven miles from the city centre, with a friend. Her friend ordered and paid for a Scotsman ice cream float — ice cream served with ginger beer. The drink was served by the owner of the Wellmeadow Café in an opaque bottle of dark glass, of the kind common at the time precisely because cloudy glass hid any sediment in the contents.

When the second half of the liquid was poured into the glass along with the melted ice cream, a snail in an advanced state of decomposition came out of the bottle, carried along by the fizz. May Donoghue fell severely ill with gastroenteritis, was seen by a doctor on 29 August, and was admitted as an emergency to the Glasgow Royal Infirmary on 16 September. Four years later, in 1932, the House of Lords handed down a ruling that is considered the cornerstone of modern product liability law in the common law world.

The snail in the bottle

The decision was taken by a majority of three votes to two. The majority opinion was written by Lord Atkin, who in the famous passage every law student still memorises today formulated what became known as the neighbour principle: the legal duty each of us owes to take care of our neighbour, where «neighbour» means anyone who could reasonably be foreseen as affected by our conduct.

From that moment on, a manufacturer is liable for its product even to those with whom it has no contractual relationship — even if the bottle passes through a distributor and a café before reaching the consumer, even if weeks of warehousing and miles of road lie between the factory and Mrs. Donoghue’s gastroenteritis.

What almost no one notices, reading the judgment, is that in order to write the neighbour principle the court needed a precise ontological presupposition: a certain idea of what the object in question actually was. The object was the bottle. A bounded artefact, with sharp edges, identifiable, examinable, preservable.

The bottle was produced in a specific plant — at David Stevenson’s in Paisley — filled at a dated moment, sealed with a stopper that guaranteed its integrity up to the point of sale. If necessary, lawyers could have demanded to examine the production batch, analyse the composition of the glass, trace the supply chain of sugars and flavourings. The material cause, in the Aristotelian sense, was reachable. The efficient cause — the production process — was documented. The formal cause — the typical shape of a ginger beer bottle — was known to anyone.

The whole modern law of product liability built on top of Donoghue v. Stevenson presupposes this utterly simple and absolutely crucial thing: that you can point your finger at an object and say «this», and that the finger finds something at the other end.

The new PLD and the problem of a category

In 1985, Europe, with Directive 85/374/EEC, adopted the Donoghue principle in continental form, extending and systematising it. The Product Liability Directive speaks of «defective products» and establishes a strict, no-fault liability for the producer for damages caused by the product.

Forty years later, on 23 October 2024, Directive 2024/2853 was adopted, entered into force on 9 December 2024, and must be transposed into national law by 9 December 2026, the date on which the 1985 directive is repealed. The new PLD was designed expressly for software, and includes in the definition of product digital files, software components, applications, and AI systems. It is one of the most ambitious reforms of civil liability law ever attempted by the European Union.

The fact that it took forty years to be rethought, and that despite the enormous effort of its drafters it risks being inadequate the day it enters into force, is no one’s fault in particular. It is the consequence of a problem that the law has not yet been able to name precisely. The problem is that the «product» on which Lord Atkin and his successors built two centuries of jurisprudence no longer exists in contemporary software.

It is worth taking a minute to understand where it went, because diagnosis is the starting point for everything that follows.

From record to concert

For a long time, software was a product in the full, conventional sense of the word. It emerged from the Bell Labs lab and entered mass distribution with Unix, then with operating systems for personal computers.

An application of the 1980s or 1990s was a set of files distributed on floppy disks, then on CDs, then on DVDs, with a version number, an identifying hash, a copy that could be kept in a vault. If the software had a defect, you could, at least in principle, isolate the exact version of the binary, compare it with the supposedly correct one, identify the bug. Windows 95 was a product. Microsoft Word 97 was a product. Even a game like Baldur’s Gate was a product, with its numbered, archivable patches.

Of course, even then software depended on things that were not the product in the strict sense: device drivers, firmware, network protocols, system services. But those dependencies were static, declared, limited in number, and above all mediated by a self-sufficient artefact that could be preserved and reproduced many years later. The copy of the Windows 95 CD that a digital archivist keeps today is still identical to the one from 1995, bit for bit. The product, as an object with continuity in time, existed.

In the early 2000s something began to change, and the pace of change accelerated dramatically over the following decade. The transition to software as a service moved the application from the customer’s local disk to the vendor’s server. At first it was a simple matter of distribution, but very quickly it became an ontological transformation.

When the application runs on a remote server, the version of the application the user is using at any given moment is no longer a stable property, it is a momentary one, renegotiable by the vendor at any instant without even notice. Continuous deployment takes this fluidity to its natural conclusion, with companies shipping tens or hundreds of changes to production per day, and no one — not even internally — able to reconstruct exactly what the system’s configuration was at 2:23 p.m. on a Tuesday three months ago.

Meanwhile, and in parallel, the internal composition of software changes in nature. From the early 2010s onward, thanks to ecosystems like npm for JavaScript or Maven for Java or PyPI for Python, writing an application means importing hundreds or thousands of third-party packages, each in turn made up of third-party packages, in a graph of transitive dependencies that no human reads in full and that updates continuously and asynchronously with respect to the code you yourself wrote. The left-pad incident of 2016 — when a single developer unpublished an eleven-line package from npm and broke a large fraction of JavaScript build pipelines and install processes worldwide — was the moment when the industry noticed, for an instant, how fragile and distributed the thing it still called a product had become.

The latest acceleration, the one we are living through right now, is the entry of language models into the production chain of software. Here the leap is not only quantitative but categorial, because what enters the product is no longer a set of deterministic instructions but a probabilistic component whose outputs are not exactly reproducible even from the same inputs, and whose weights, once retired by the vendor in favour of a successor, may no longer be accessible at all.

An application running today and using the API of a frontier model is an application whose observable behaviour depends on a component that tomorrow may behave differently, that two years from now may no longer exist in any recoverable form, and whose internal workings are not inspectable by anyone — not even by those who trained it. Anyone writing software today knows perfectly well that if a user from five years ago asked us to reproduce exactly the bug they had seen back then, the honest answer would be that we can’t: the system that produced that bug no longer exists in any restorable configuration. Dependencies have been updated, external APIs have changed contract, the database has been migrated, the AI model has been replaced. We have the logs. We have the code of our application. But the event that produced the bug was a concert, and what’s left of concerts is recordings, not concerts.

This is the metaphor that I think captures the ontological shift underway best. Contemporary software is no longer a record. It’s a concert.

A record is a stable, preservable, comparable artefact — an object with its own identity in time. A concert is an event distributed in space and time, co-produced by many agencies in real time, unrepeatable, of which you can have traces but not originals. When we listen to the recording of a Keith Jarrett concert from 1975, we are listening to a derived artefact — we are not listening to the concert, which happened once and is over. The same holds for a SaaS application today.

What the user experienced yesterday at 11 a.m. was an event co-produced by the application, by the user’s browser, by network latency, by the current version of the underlying LLM, by the configuration of twelve third-party APIs, by a rate limit that at that exact moment had a certain value. That event is not reproducible, not preservable, ontologically not an object. It is, precisely, a concert.

The objection and the copy that isn’t there

Someone, at this point, might object that the distinction is overstated, that after all even in 1998 software ran on an operating system that depended on drivers and hardware, that runtime contingency existed back then too, that every use of an application has always been, in some measure, an event rather than an object.

The objection is partly correct, and taking it seriously matters, because it is the only way to keep diagnosis from collapsing into nostalgia. The difference that changes everything, however, is one and only one — and it is the difference the law has not yet metabolised.

In 1998 there was always a copy of the artefact. In the strictest, most concrete sense of the term, in the vendor’s vault or on the user’s hard drive or in some digital archivist’s storage, there was a material object that contained the software in its autonomous form. That copy was the point of contact between the legal product and the executive event. It was the Archimedean point that made it possible to go back, examine, compare, judge. It was what let the lawyer say «this product was defective» because a product was there.

Today that copy no longer exists, and it does not exist not because of negligence or bad choice, but by construction. Vendor X’s SaaS application at 2:23 p.m. on a Tuesday three months ago exists in no backup, no vault, no archive. What exists is only the possibility of reconstructing, approximately and partially, what it must have been, by cross-referencing logs, code versions, dependency configurations, model weights, database state.

This is an archaeological reconstruction, not an access to the object. And even when such a reconstruction is complete, the result would be yet another thing — a model of the application at that moment, not the application at that moment. The product, in the two-century-old legal sense, has vanished. What exists in its place has never been named.

The European corpus and its categorical misalignment

To their credit, European legislators have noticed that something isn’t adding up. In recent years they have produced an impressive package of rules trying to grasp the new object with the tools at hand.

The Cyber Resilience Act, which came into force in December 2024 with full application in December 2027, introduces the concepts of security by design and security by default for products with digital elements, imposes support obligations across the full lifecycle, requires the provision of a Software Bill of Materials. The new Product Liability Directive, the 2024 one mentioned above, explicitly extends liability to cybersecurity defects and to malfunctions from software updates. The Artificial Intelligence Act, in force since 1 August 2024 with general application from 2 August 2026 (and some provisions slipping to 2 August 2027), classifies AI systems by risk and imposes obligations of documentation, transparency, and conformity assessment. Add to these NIS2 for network security, DORA for the financial sector, the Data Act, the European Accessibility Act.

Together they form the most ambitious regulatory corpus ever built to govern software production, and they are a serious, technically informed, culturally important effort. It would be a mistake to dismiss them as bureaucracy or as a brake on innovation, as a certain libertarian Californian vulgate likes to do. The thesis of this essay, rather, is that — serious and important though they are — these instruments are trying to capture the concert with the categories of the record, and the categorical misalignment is bound to generate consequences we are only beginning to glimpse.

The case of the Software Bill of Materials is paradigmatic of this difficulty, and it is also the one I know best for professional reasons. An SBOM is a document that lists all the components that enter a software product, their versions, their dependencies, their licences. On paper it’s a perfectly sensible tool, the software equivalent of the ingredients list on food packaging.

In practice, an SBOM is a photograph of a concert, with all the epistemic limits that condition entails. A photograph of a concert is not the concert — it is already another object: derived, partial, dated. In the very moment the photograph is taken, the concert is already changing, and a tenth of a second after the shutter closes, transitive dependencies may have updated, the external API version may have changed, the model weights may have been swapped by the vendor.

The SBOM captures a state, but contemporary software has no stable states, only flows. The regulation therefore requires developers to keep the photograph up to date, which in turn generates processes of continuous SBOM generation, with tools producing the SBOM on every build, every deploy, ideally in real time. The paradoxical result is that the document that was supposed to capture the identity of the product multiplies into thousands of daily snapshots, each of which is already obsolete the moment it is generated. The SBOM does not capture the product, because there is no product. It captures the flow, in the form of discrete frames, and each frame is a legally relevant document. The resulting documentary load is enormous, and its probative value, when it comes to reconstructing who was responsible for what at a precise moment, is far less clean than legislators assumed.

The same problem, compounded by the stochastic nature of the models, arises for the AI Act. Article 10 asks that training datasets be documented, traceable, governed according to quality criteria. Article 15 asks that high-risk AI systems be accurate, robust, cybersecure. Article 72 requires post-market monitoring. Each of these obligations, taken in isolation, makes sense, and points to practices that the best machine learning engineering should already be adopting.

What legislators have not fully absorbed is that the language model a vendor offers today via API is an object whose identity in time is, at best, conventional. The model’s weights get updated, the RLHF system gets refined, the guardrails modified, and what continues to be called GPT-4 or Claude Opus or Gemini Ultra is, from the standpoint of observable behaviour, a different object from what bore the same name six months ago.

Asking for training documentation means asking for documentation of something that, by the time the documentation is produced, may already have been superseded by another training run. Post-market monitoring is excellent practice, but it presupposes a market in which the product is the same at the time of sale and at the time of use — something that is almost never true for language models.

This isn’t to say the AI Act is wrong, because the AI Act is one of the most articulate regulatory responses ever attempted at comparable scale. It is to say that the AI Act inherits from product liability law a set of ontological presuppositions that the object it tries to regulate does not satisfy, and that the tension between presuppositions and object will fall, in the coming years, onto the shoulders of practitioners — in the form of compliance obligations of ambiguous value, documents whose relation to reality is hard to establish, controversies in which the dispute between parties shrinks to a debate over whether a given snapshot was representative of the system at the moment of harm.

CrowdStrike, July 19, 2024

A recent example, of such reach that it has entered risk management courses all over the world, helps to see concretely what all this means.

On 19 July 2024, a faulty update of CrowdStrike’s Falcon sensor caused the simultaneous crash of roughly 8.5 million Windows machines around the planet, with a boot loop that required manual intervention on each device to resolve. Airports grounded, hospitals reverting to paper operations, banks offline, broadcasters dark, emergency calls not routed. Delta Airlines alone declared over half a billion dollars in direct damages, Fortune 500 companies declared uninsured losses in the order of $5.4 billion, with global estimates placed by various sources in the tens of billions.

In the ensuing litigation, still open in multiple jurisdictions, the whole question has focused on one point: who was responsible?

• CrowdStrike, which pushed an update containing a defect any careful test would have caught.
• Microsoft, which allowed third-party software to operate with kernel-level privileges sufficient to boot-loop the operating system.
• The deployer organisations, which accepted auto-push updates without canary stages.
• European regulators, according to Microsoft’s defence, which years earlier had forced the vendor to open the kernel on competition grounds.

Each of these four attributions of responsibility, taken singly, captures a real aspect of the affair. None, taken singly, tells what happened. What happened was an orchestration failure: an event in which many actors each held, at their own level, actual influence over the system’s runtime behaviour, and in which no one had exercised that influence in proportion to the possible consequences.

Current law has no adequate tools to distribute liability this way, and litigation is sliding, as a result, toward scenarios where the better-lawyered party wins or where the most defensible contractual position prevails — which is another way of saying the law has given up on saying anything substantive about the incident. A case like CrowdStrike, read through the lens of the responsible-assemblage category I am trying to sketch, should have produced a ruling that distributed liability in identifiable shares across the various actors as a function of their blast radius and their degree of effective control, and that created a precedent usable for future incidents, which will certainly be many. None of this is happening, because the category for doing it isn’t there.

Latour, Ricœur, and the plurality of action

Before going further, it is worth clarifying that responsible assemblage is not an invention ex nihilo. It is the adaptation, to the legal domain, of reflections twentieth-century philosophy developed in other fields and that have not yet made sufficient inroads into courtrooms.

Bruno Latour, with actor-network theory, had already shown in the 1980s that complex actions never have a single agent — they emerge from networks of human and non-human actors whose analysis requires suspending the obsession with the individual subject and following the actual connections. His canonical example of the artificial speed bump — the so-called sleeping policeman placed on the road surface — is illuminating, because it shows how the bump enforces the speed limit independently of any human officer’s presence, acting as a non-human actor endowed with its own agency.

The car slowing down is co-produced by the driver, the bump, the traffic code, the fear of damaging the suspension, and the shared culture that recognises the bump as a signal to slow. None of these five actors, taken alone, explains the slowing down, and none of them, taken alone, is fully responsible for what happens when something goes wrong. Positive law tends to place responsibility on the driver for practical reasons, but the philosophical understanding of the event requires seeing all five together. The step the law has not yet taken, but philosophy has taken for forty years, is to accept that this distribution of agency is the rule, not the exception, and to derive its consequences for the attribution of responsibility.

Paul Ricœur, in Oneself as Another (1990) and in the subsequent reflection on imputability, added a complementary piece, showing that the act of attributing responsibility is a narrative act — it requires reconstructing a plausible story of causation, and when the plausible story involves many agents, imputative narration cannot be reduced to a single subject without betraying the truth of the event. For Ricœur, imputation is a hermeneutics of responsibility, and like any hermeneutics it demands patience, attention to different levels, refusal of simplification.

Contemporary product liability law simplifies too much, and does so because it has inherited a nineteenth-century narrative grammar in which the producer was one, identifiable, imputable. Reconstructing imputability for contemporary software means accepting the plural nature of action and letting it show through in the text of the judgment itself.

Responsible assemblage

If we stop here, the diagnosis risks looking pessimistic or purely critical, in the style of a certain tech-sceptical literature that takes pleasure in difficulties without proposing anything constructive. The point I want to try to develop is the opposite, because if it’s true that the concept of product no longer sustains its function of attributing responsibility, we have to start imagining what its replacement category might look like.

This is an intellectual undertaking that cannot be delegated either to pure lawyers, who have no contact with the runtime of real systems, or to pure engineers, who tend to treat law as an external annoyance rather than as an attempt to attribute moral meaning to what they do. The replacement category must be thought out with four hands — by those who have code practice and legal reading — and it must start from the observation that contemporary software is an orchestration, an assemblage, a distributed event, not an object.

I propose to call it, provisionally, responsible assemblage, aware that the name is not yet final and that the argument over the name is part of building the thing.

What would a responsible assemblage be, in first approximation? It would be a configuration of components — human and non-human — organised around a specific function offered to a user, in which no participant holds integral control but each exercises an identifiable degree of effective influence over the observable behaviour at runtime.

Liability, in this framework, no longer follows formal ownership («whoever holds the contract answers for everything»), nor the historical origin of components («if the bug is in the imported library, then the library maintainer is to blame»). Liability follows the gradient of effective influence.

Whoever was able, at the moment of the harm, to change the system’s behaviour through a targeted technical action answers in proportion to the degree of that possibility. The AI model vendor who could have updated the guardrails and didn’t answers for a share of liability. The integrator who chose that model for a sensitive function without adequately assessing the risks — while having the tools to do so — answers for another share. The user who configured the system in an anomalous way, bypassing the warnings, answers for a third. None of the three answers for everything, none answers for nothing. Each answers for their portion of the orchestration.

The formula is easy to state and infernally complex to implement, as is natural for every new legal category in its first decades.

The objections come fast, and it is worth running through them. The first objection is that «effective influence» is hard to measure, and that a legal system built on so slippery a concept would be a source of uncertainty. The objection is true but less grave than it seems, because civil liability law already operates with slippery concepts like fault, foreseeability, proximate causation, and has developed operational tools to handle them over time.

Effective influence on a distributed system is measurable through what engineers call blast radius — an estimate of the reach of the consequences of a technical action. An LLM API vendor has a blast radius that potentially encompasses all of its integrators, because a change on its side propagates instantly. An integrator has a blast radius limited to its own users, though this can be amplified if its application is itself infrastructure for other applications.

There are technical metrics — rough but tractable — for estimating each actor’s blast radius, and legal doctrine could build on top of them a graduated taxonomy of responsibility. It would be no more slippery than the doctrines of fault in the nineteenth century were before decades of case law made them workable.

Two deformations to resist

The second objection is more political and comes from two opposite directions, both in their own way invested in the outcome.

The first direction is that of the big-tech Californian environment, which has built over the past twenty years a narrative according to which the open-source and componentised nature of software makes it impossible to attribute responsibility to anyone in particular — and therefore responsibility should be limited to contract terms explicitly accepted by the parties. On this reading, if a user is harmed by an application that uses an open-source library whose author is anonymous, and the application is in turn hosted on a cloud whose provider disclaims all liability in its terms of service, then no one answers for anything — or rather, only those who have deliberately assumed responsibility by signing a contract do.

This is the doctrine of distributed irresponsibility, and it is a comfortable ground rent for those occupying the most central nodes of the orchestration, because it lets them capture the value and externalise the risks. Responsible assemblage opposes this doctrine frontally, because it rejects the idea that the absence of a formal title-holder amounts to the absence of substantive responsibility. If your model is the critical component of thousands of downstream applications, your responsibility does not depend on whether you signed a contract with those applications’ end users. It depends on your role in the orchestration, and that role is objectively measurable.

The other direction from which the objection comes is European continental formalism, which has an opposite but equally problematic tradition, which we might call the doctrine of the absolute controller.

On this tradition, the data controller, or the service provider, or the AI system deployer, answers for everything that happens along the chain, regardless of where the non-conformity or malfunction originated. It is an intuitively egalitarian doctrine, because it protects the user by guaranteeing that someone — an identifiable legal entity — will always answer. In practice, however, it produces two serious distortions.

The first is that it loads the controller with responsibility over which it has no technical control, forcing it to defend against litigation over malfunctions originating in components it did not write, did not choose, cannot inspect. The second is that it de-responsibilises the nodes upstream of the chain — precisely those with the most systemic influence — because they know their share of responsibility will be formally dumped on the final deployer and that courts will keep pointing at whoever sits in the most visible position.

Responsible assemblage opposes this doctrine too, because the controller’s formal position does not exhaust the question of responsibility, and a merely apparent distribution of the load — which in fact always concentrates on the most accessible node — keeps things exactly as they are in the industry’s real power relations.

The proposal, therefore, sits in territory uncomfortable for both prevailing sensibilities, and for that very reason deserves to be articulated with some patience. Anyone who works in software knows perfectly well that the idea of responsibility as a purely contractual fact is a convenient fiction for those in Silicon Valley and a daily experience of powerlessness for those in Pescara or Milan or Frankfurt who have to deliver a system that actually works in a context of real constraints. At the same time, anyone working with European compliance knows that the absolute-controller doctrine turns the deployer into a legal fuse, whose economic function is to blow when the system breaks, so that the upstream components can keep producing externalities.

Contemporary software needs a category that is neither of these, one that manages to distribute liability in a way that reflects the actual distribution of technical power. Not a symmetric distribution, because technical power is not symmetrically distributed. Not a concentration on the controller, because technical power is not concentrated on the controller. A distribution that follows the gradient of effective influence, with transparent metrics, with a workable doctrine courts can handle, with a rationale practitioners can recognise as matching their working experience.

There are some areas where something like this is already happening, even if no one calls it that. The GDPR, in Article 28 and in the EDPB guidelines on the chain of sub-processors, has begun, tentatively, to recognise graduated responsibility along the processing chain — though in a formalistic way and without the conceptual apparatus that would be needed. Data Protection Impact Assessments, when done seriously rather than as an exercise in compliance theatre, contain in embryo a blast-radius analysis applied to personal data.

The NIS2 directive on network security introduces obligations for managed-service providers that recognise their systemic role in the security chain. The AI Act itself, with its distinctions among provider, deployer, importer, distributor, attempts a taxonomy of roles that — still too rigid — points in the direction of a non-controller-centric distribution of responsibility.

None of these instruments has yet reached the point of explicitly formulating the principle of effective influence as a gradient of responsibility, but all contain traces of a rethinking that awaits only doctrinal systematisation. The impression is that what’s missing is something analogous to what the neighbour principle was for Lord Atkin: a synthetic, memorable, operational formulation capable of orienting fifty years of subsequent jurisprudence. Whoever writes that formulation will have done for the twenty-first century what the House of Lords did for the twentieth.

The problem is that for that formulation to be written, a category of intellectuals that essentially does not yet exist will be required. Pure lawyers won’t do, because the internal grammar of contemporary distributed systems isn’t taught in law schools. Pure engineers won’t do, because the grammar of law and its two-century history of categories and counter-categories isn’t taught in computer science.

What’s needed is hybrids — border figures who have practised both trades long enough to have grasped their internal logics and limits, and who are willing to imagine something neither disciplinary corpus, taken alone, can imagine. In Italy this figure does not yet have a settled name. In the anglosphere the roles of compliance engineer or technical counsel have emerged in recent years, but these remain professional labels that merely put the two cultures in contact without really building a third.

What’s needed is something more ambitious: a figure who doesn’t just translate technical jargon into legal jargon and vice versa, but who elaborates new categories capable of saying things neither of the two original jargons, left to itself, could say. The trade is partly theoretical, because it requires a philosophical training that lets you handle abstract categories, and partly practical, because it requires daily experience of deployment, transitive dependencies, backward compatibility broken by a vendor’s minor release, ambiguous stack traces that don’t say whose fault it is. Without practical experience, abstract categories hang in mid-air. Without abstract training, practical experience exhausts itself in fatigue and closed tickets.

The third way is hard — not because it requires rare talents, but because the labour market does not price it, universities do not train it, companies do not organise it. It exists only through individual will, and it is precisely for this reason that those who practise it carry intellectual responsibility disproportionate to their formal standing.

From Bill of Materials to Bill of Accountability

At this point it’s useful to come back to the concrete problem we started from: how can we attribute responsibility for harm caused by contemporary software?

If we accept that software is no longer a product but an assemblage, and that assemblage is to be analysed according to the gradient of effective influence, then the first operational step is to build a practice of documenting the assemblage — one that is not the static photograph of the SBOM but a dynamic map of relationships of influence.

Such a tool should make it possible, for any application, to answer questions like these.

• Which components, if modified, would change the system’s observable behaviour?
• Which actors have the technical capacity to modify those components?
• What contractual and regulatory obligations bear on each of those actors?
• What evidence, in logs and audit systems, lets us reconstruct ex post the state of the system at a specific moment?
• Through what channels can a downstream actor get an alarm signal to an upstream actor?
• What are the typical response times along the chain?

The resulting document would no longer be a bill of materials — it would be something closer to a bill of accountability, a map of powers and duties along the entire orchestration. Producing it would require work, would require cooperation between actors who don’t always have an interest in cooperating, would require rules that mandated its production. But it would be a tool that names the problem for what it is, rather than reducing it to an inventory problem.

Three consequences for software practice

From the standpoint of the daily practice of those who build software for clients, this re-articulation would have important consequences.

The first consequence is that the typical contract — the one that today tends to dump all liability on the final vendor or, at best, to cap it at the annual fee paid — would become a document with a completely different structure. The contract would be built downstream of an analysis of the orchestration, with a clear articulation of which components the vendor controls and which it merely integrates, which risks it fully assumes and which it passes through to upstream providers, which disclosure obligations it accepts to honour and which it is technically unable to honour. It would be a longer, more complex, more honest contract — and, over time, also a more defensible one in court, because it would tell the thing for what it is rather than pretending a shape it doesn’t have.

The second consequence is that the development process would have to include, from its earliest phases, a reflection on the orchestration in which the system will be embedded — not only a reflection on functional requirements.

• Who are the upstream actors we will depend on?
• What guarantees do they give us?
• What blast radius do they have in our regard?
• How will we verify their changes?
• How will we react if a change of theirs breaks our expected behaviour?

Today these questions are relegated to a few slides at the architecture stage and then forgotten. In a culture that took responsible assemblage seriously, they would be part of the specification, and their absence from project documentation would be a marker of immaturity the way the absence of a threat model is today.

The third consequence is perhaps the most interesting culturally, and it touches directly on how developers are trained. Today you learn to write code that works, then, if you’re lucky, to write code that works in production. Almost no one learns to write code that works within an orchestration they are aware of.

Orchestration awareness arrives late, usually through an incident, rarely as an explicit, taught competence. A training that took seriously the distributed nature of contemporary software would have to start here — it would have to show from day one that writing an API means entering an orchestration, that choosing a library means accepting an upstream vendor, that deploying to cloud means hosting yourself inside an infrastructure whose architectural decisions are elsewhere.

It’s not about terrifying young developers, it’s about getting them used to seeing what experienced developers see after twenty years of scars. Orchestration is the object of their work, not a background. The code they write is a participation in a concert, not a self-sufficient object. If they grow up with this awareness, the question of responsibility won’t arrive as an external annoyance — it will arrive as an intrinsic dimension of the craft.

When even the writing is orchestration

To anyone suspecting that all this is exaggerated by the SaaS lens, and that in simpler contexts the product category might still hold, it is worth answering that the direction AI-native development is moving makes the problem more acute, not less.

When you work with tools like Claude Code, or with practices like specification-driven development à la SpecKit, code stops being the object the developer writes directly and becomes the output of a generation chain that starts with a spec, passes through a model, produces an artefact, and integrates it into an orchestration that is already itself distributed. The spec written yesterday and the code generated yesterday, if regenerated today by the same model, would produce code plausibly similar but not identical, because the model has changed in the meantime.

The concept of version, already weakened by continuous deployment, becomes even more unstable in this context, because it is now unstable not only at execution time but at writing time too. Anyone developing this way knows that exact reproducibility is a regulative horizon, not an actual property of their artefacts.

Responsible assemblage, in a world where even writing code is an orchestration among developer, spec, model, toolchain, and pipeline, becomes the natural category for thinking about responsibility — because it is the only one in which the absence of a monolithic author does not automatically translate into the absence of imputability.

The death of a category

A question remains that the essay owes its answer, because it is the question a patient reader will have held in suspense from the start. The question is whether it really makes sense to speak of the death of a legal category, or whether we are just describing an evolution, a transformation, an update.

The language of death is strong, and might look exaggerated. The justification I offer is that a legal category dies in the sense Foucault meant when he spoke of categories more broadly — that is, when it stops organising the discursive field productively and starts obstructing it.

The concept of product, applied to contemporary software, no longer organises the field — it obstructs it. It produces compliance that doesn’t match practice, documents that don’t represent objects, litigation that gets stuck on the definition of the object before it can address the substance.

A concept that obstructs the field is not a useful concept, even if it remains formally in force. The concept of luminiferous ether, in late-nineteenth-century physics, wasn’t wrong the way singular claims can be wrong — it was a concept that had stopped organising the field productively and had started obstructing it, until Einstein simply removed it.

The concept of product, in contemporary software law, finds itself at a comparable stage. It is still there, still in the normative texts, still in courtrooms. But it is no longer working. And pretending it works is more dangerous than admitting it doesn’t.

I’m not saying, let me be clear, that the European legislator should have waited to have the right category before acting. That position would have been intellectually pure and politically unpresentable, and in any case the damage from the absence of regulation would have been greater than the damage from a categorially imperfect one.

I am saying, more modestly, that current regulation is a work in progress, that its categorical imperfection must be acknowledged openly, and that the construction of the replacement category is a task that falls not only to legislators but also to those who work in software every day and to those who have, from philosophy of law, the tools to think. Those with both feet inside can contribute more than those with only one. And the contribution is no academic exercise, because every year that goes by without the replacement category emerging is another year in which litigation resolves through formal paths that don’t mirror substantive responsibilities, another year in which upstream vendors keep transferring risk without paying for it, another year in which final deployers keep acting as fuses while the system as a whole does not improve.

Lord Atkin’s room

I have gone on at length, and those who have followed me this far deserve at least an honest summary of where we have arrived.

We have arrived at saying that the legal concept of product, built in the twentieth century on an ontology of bounded, preservable, identifiable objects, is insufficient for contemporary software, which is instead an orchestration of human and non-human components — a distributed event rather than an object, a concert rather than a record.

We have arrived at saying that European regulation, for all its being the most serious effort ever attempted, is trying to capture the concert with the categories of the record, and that this tension produces compliance whose substantive legal value is ambiguous.

We have arrived at saying that, in place of the category of product, we need to build a new one, provisionally called responsible assemblage, which distributes liability according to the gradient of effective influence at runtime rather than according to formal ownership or distributed irresponsibility.

We have arrived at saying that this construction requires a hybrid intellectual figure, with code practice and legal reading, that the market does not price and that exists only through the individual will of those who practise it.

And we have arrived at saying that, until the replacement category emerges, litigation will keep resolving through formal paths that don’t mirror substantive responsibilities, with real costs for those who do the craft honestly.

The point perhaps worth fixing, at the end of all this, is that Lord Atkin, in 1932, was not a pure jurist — he was a judge who had seen many bottles break and many people fall ill, and had understood something practical about the industrial world that the prior conceptual toolkit could not say.

Donoghue v. Stevenson was possible because someone had held together, in a single head, the material experience of the industrial world and the conceptual tradition of civil law. Today what is needed is to hold together, in a single head, the material experience of distributed systems and the conceptual tradition of liability law. It is not a task that can be delegated to anyone, and it is not even a task that can be completed in a single essay. What can be done, with an essay, is to try to name the problem, offer a direction, invite those with tools similar to mine to continue the work.

Mrs. Donoghue’s bottle was opaque, and inside it was a snail no one had seen. Contemporary software is even more opaque, and inside it are many more snails, and the case law that protects us was written for a world in which bottles were made of glass. The time has come to write the case law of a world in which there are no bottles left.

The author

Andrea MargiovanniDigital sovereignty

I help public bodies and private organizations read their own infrastructure dependencies. Digital sovereignty is a lattice, not a flag; and it is measured more on contracts than on speeches.

See the guide →