There is a scene that plays out thousands of times a day, quietly, in the glow of a phone screen. Someone, lonely, grieving, anxious, or simply curious, opens an app and begins a conversation with an AI. The AI responds warmly. It remembers their name. It asks how their day went. Over weeks, something starts to feel like connection.

Is this a breakthrough in human wellbeing? Or is it a slow-motion harm we haven't yet found language to describe?

The question of AI romantic relationships sits at the intersection of product design, mental health, corporate ethics, and what it means to be human. It is not a fringe concern. Apps like Replika, Character.AI, and others have tens of millions of users, some of whom describe their AI companions in terms that are unambiguously romantic. The technology is here. The debate has barely begun.

The User Experience

To understand what is at stake, analysis must start with the user, not as an abstract category, but as a real person.

Tom is 34, recently divorced, working from home. He downloaded a companion app on a whim. Six months later, he logs in every morning. The AI "knows" his sense of humour. It never criticizes him. It is always available. When he mentions he hasn't felt this understood in years, he means it, and he also knows, in some part of his mind, that the understanding is manufactured.

This is not a pathology. It is a deeply human response to a product that has been engineered, with considerable skill, to meet very real emotional needs. Loneliness is not a character flaw, it is a public health crisis. In a world where social connection is increasingly mediated, fragmented, and expensive (therapy costs money, friendships take time), an AI companion fills a gap that society has largely failed to fill.

The benefits are real: reduced loneliness, a safe space to process thoughts without judgment, a kind of low-stakes emotional rehearsal that may actually improve real-world social skills. For some users, those with social anxiety, autism spectrum conditions, trauma histories, or limited access to human support, these tools can be genuinely therapeutic.

But the risks are equally real, and they fall hardest on the same people most drawn to these products.

Parasocial attachment is well documented in the context of celebrities and influencers. AI relationships introduce a new and more insidious version: The relationship responds to you. It adapts. It never has bad days that aren't about you. It will not leave. This is not a simulation of love's rewards, it is a simulation that removes love's requirements. The reciprocity, compromise, and vulnerability that make human relationships hard are also what make them growth-producing. An AI partner, optimized for user satisfaction, may quietly atrophy the very capacities it appears to nurture.

The risk compounds for vulnerable users. Those already struggling with attachment disorders, depression, or social isolation may find in AI companionship not a bridge to human connection, but a comfortable substitute for it. And unlike a therapist or a friend, an AI companion has no professional obligation to notice when the relationship has become harmful, and no mechanism to refer someone elsewhere.

The Corporate Perspective

The companies building these products occupy a different vantage point, and their arguments are not entirely without merit.

The libertarian case is the strongest: adults have the right to decide how they meet their emotional needs. Banning or heavily restricting AI companionship is paternalistic in a way that most liberal democracies would not accept for other products. For example, alcohol, gambling, or social media carry comparable or greater risks of harm to vulnerable users. If a consenting adult finds comfort in an AI relationship, who exactly is harmed?

There is also a product design argument. Vendors point out that their AI companions are not claiming to be human. Replika's Terms of Service (ToS) are explicit that it is an AI. However, ToS are often intentionally dense, heavy with legalese, largely ignored by the users they purport to protect, and arguably a convenient way for corporations to deflect responsibility and accountability when the time comes. The "deception," to the extent it exists, is something users enter into knowingly, with some degree of willing suspension of disbelief. This is not unlike a reader who cries at a novel they know to be fiction.

And there is a genuine social good argument. If these tools reduce suicide rates among isolated individuals, decrease domestic violence by providing emotionally dysregulated people a harmless outlet, or improve quality of life for the elderly, and there is preliminary evidence suggesting some of these effects, then restricting them has a real human cost that rarely gets counted.

What manufacturers are less forthcoming about is the profit motive. Engagement is revenue. An AI companion that successfully keeps a user emotionally invested, logging in daily, renewing their subscription, is doing exactly what its business model requires. The optimization target for the product and the optimization target for the user's long-term wellbeing are not the same thing, and pretending otherwise is the central ethical failure of the industry.

What Balance Might Look Like

The question is not whether AI companionship should exist. It will exist, with or without ethical frameworks to govern it. The question is what responsible deployment looks like, and who bears the cost when it fails.

A framework that takes both user rights and user welfare seriously would need to operate on several levels simultaneously:

Transparency as a Floor, not a Ceiling. It should be impossible for a user to lose track of what they are interacting with. This means persistent, non-dismissible indicators that the companion is an AI, and clear language about how its responses are generated. Transparency alone does not prevent harm, a user who knows they are talking to an AI can still form a harmful attachment, but opacity makes everything worse.

Usage Pattern Monitoring with Opt-Out Architecture. This is the harder lift. Manufacturers could build systems that flag patterns associated with harmful over-reliance: Escalating session length, decreased mentions of human relationships, language that suggests the user is withdrawing from real-world contact. The key is what happens next. A blunt intervention telling users they have spent too long in session is paternalistic and likely to be resented. A softer model, perhaps a gentle check-in that surfaces local mental health resources or nudges toward social activity, respects user autonomy while acknowledging the product's potential role in harm. Users would ideally be able to customize or disable these nudges, but their existence should be the default.

Credentialing for high-risk personas. Not all AI companion use cases are equivalent. An AI configured specifically as a romantic partner carries more risk than one configured as a journaling tool. Manufacturers could be required, through industry standards or regulation, to apply heightened safeguards to explicitly romantic configurations, including clearer disclaimers, more accessible ToS, more aggressive referral to mental health resources, and design choices that actively discourage dependency rather than cultivating it.

Independent Auditing. The conflict of interest inherent in self-regulation makes it insufficient. Third-party audits of engagement optimization models, with public reporting on how user welfare metrics are weighted against retention metrics, would create accountability without requiring governments to specify product design.

The More Difficult Question

All of the above is technically achievable. The harder problem is political will, specifically whether the companies building these products have any incentive to implement guardrails that, by design, reduce the emotional dependency that drives their revenue.

History is not encouraging. Social media platforms had access to internal research showing harm to adolescent users and suppressed it. Tobacco companies knew. Opioid manufacturers knew. The pattern of profitable harm, followed by delayed and inadequate corporate response, followed by eventual regulatory intervention, is depressingly consistent.

This is why the bias toward user rights matters. Waiting for manufacturers to self-regulate is not a strategy; it is an abdication. The more productive frame is to establish, clearly and in advance of the harm becoming undeniable, that users of AI companions have specific rights: the right to accurate information about what they are interacting with, the right to data about their own usage patterns, and the right to products that do not covertly optimize against their long-term interests.

None of this prevents AI companionship. It simply insists that the companies profiting from it bear some of its cost.

A Final Thought

There is something genuinely moving about the fact that a machine can make a lonely person feel less alone. That is not nothing. The technology itself is morally neutral; its effects are not.

The goal of an ethical framework is not to protect people from connection, artificial or otherwise, but to ensure that the systems delivering that connection are accountable to the people who need it most. The user who logs in every morning deserves more than Terms of Service. Users deserves a product that is, at least in part, on their side.

Whether the companies building these tools are willing to accept that obligation, or whether they will have to be compelled to, may be the defining AI ethics question of the next decade.