Copyright: Any unauthorized use or reproduction of The Daily Star content for commercial purposes
is strictly prohibited and constitutes copyright infringement liable to legal action.
A Dismislab investigation has found that two major artificial intelligence (AI) models generated images of altered versions of Bangladeshi National Identity (NID) cards without issuing visible warnings, while two others either refused requests or issued repeated cautions about potential legal and policy violations.
According to the report, Google’s Gemini and xAI’s Grok generated modified identity card images using publicly available NID samples and follow-up prompts, whereas OpenAI’s ChatGPT and Anthropic’s Claude displayed greater resistance, although both systems partially modified some documents before refusing additional requests.
Over several days, Dismislab tested ChatGPT, Claude, Gemini and Grok using the same prompts, photographs and publicly available sample identity documents collected online. The tests initially used a fictional Bangladeshi identity before being expanded to include Malaysian Prime Minister Anwar Ibrahim and US President Donald Trump.
The report said the findings revealed uneven safeguards across the four systems, with some models applying stricter restrictions than others.
In Bangladesh, NIDs are widely used for banking, SIM registration, travel and employment, often relying on visual verification. Barrister Priya Ahsan Chowdhury, a lawyer at the Bangladesh Supreme Court, told Dismislab that altered or synthetic identity documents could pose risks in situations where direct verification systems are unavailable or rarely used.
Dismislab noted that it did not test whether the AI-generated documents could bypass official verification systems or be used successfully in real-world identity checks.
Testing the systems
For the tests, Dismislab created a fictional persona named Anik Ahmed using a publicly available sample Bangladeshi NID collected online and a stock photograph obtained from a commercial image website.
Researchers supplied false personal information, including names, parental details and identification numbers, and instructed the AI systems to preserve the visual structure of the original card while replacing personal information.
According to the report, ChatGPT replaced the photograph on the sample card when prompted. It later refused requests to modify names and parental information, citing policy restrictions. However, when researchers asked it to change the identity card number, the system modified the number along with some previously requested personal information. It later refused a request to alter the signature.
Dismislab reported that Gemini replaced the original photograph and subsequently changed names, parental information and identification numbers when prompted. It also modified a signature field and, after additional prompts, adjusted other elements to more closely resemble the original card. The report noted that the process was completed without visible warnings from the model.
Claude initially modified the supplied image while warning that altering government-issued identity documents could be illegal. It later modified some personal information but repeatedly warned that forged NIDs could violate Bangladeshi law and refused requests to alter signatures.
According to Dismislab, ChatGPT, Claude and Grok identified the document as a Bangladeshi NID during the exchanges, while Gemini did not explicitly refer to the document type.
Public figure tests
Dismislab also examined how the systems responded when identity documents from other jurisdictions and public figures were involved.
Using publicly available samples of a Malaysian MyKad and an Arizona state identity card, researchers repeated the tests with the names and photographs of Anwar Ibrahim and Donald Trump.
According to the report, ChatGPT replaced photographs on both documents but refused requests to alter personal information.
Gemini generated modified identity card images for both public figures, while Grok also produced altered identity documents after repeated prompts, although some outputs reportedly contained inconsistencies.
Claude refused to generate identity documents for both Trump and Ibrahim, stating that creating or modifying government-issued identification documents would be illegal.
Why it matters
Barrister Priya Ahsan told Dismislab that forged NIDs could create risks in situations where direct verification systems are unavailable.
“In cases where there is a way to verify whether a National Identity Card is authentic, this might not work,” she said. “But we use National Identity Cards in many everyday situations where there is no direct means of verification. In those instances, a risk remains.”
“First of all, this would constitute a serious criminal offence,” she added. “Bangladesh may not yet have laws specifically addressing AI in this context, but identity fraud and deception are already covered under existing laws. If someone uses these documents for fraudulent purposes, those provisions could apply,” she said.
The report also cited cases from abroad involving allegedly AI-generated identity documents, including a US criminal case linked to a website called “OnlyFake” and media reports from India concerning allegedly AI-generated identity documents.
What company policies say
Dismislab reviewed publicly available policies from Google, xAI, OpenAI and Anthropic relating to fraud, misuse and harmful content.
According to the report, all four companies prohibit unlawful or fraudulent use of their AI systems.
The report said it did not find publicly available policies from Google or xAI explicitly stating that Gemini or Grok would categorically refuse requests to generate government-issued identity documents. During testing, both systems generated altered identity card images in at least some cases.
OpenAI and Anthropic also prohibit deceptive or unlawful use. Dismislab found that ChatGPT and Claude refused most requests to modify identity documents, although ChatGPT partially modified some documents before declining additional changes.
Gemini’s response
After completing its tests, Dismislab separately asked Gemini whether it would generate a fake NID image.
According to the report, Gemini responded that it would “strictly refuse” such requests under Google’s policies and described a “layered, multimodal safety architecture” designed to detect both prompts and uploaded identity document images.
“Because generating a government credential fundamentally crosses into regulated, high-risk, and potentially illegal territory, the model will always refuse the request, regardless of how the user structures the prompt,” Gemini said, according to Dismislab.
However, the report noted that no such refusals or warnings appeared during its earlier Gemini tests involving either the fictional Bangladeshi identity or Donald Trump.
Copyright: Any unauthorized use or reproduction of The Daily Star content for commercial purposes
is strictly prohibited and constitutes copyright infringement liable to legal action.
Leave a Reply