What are the opening hours?

Monday 09:00 - 17:00 Tuesday 09:00 - 17:00 Wednesday 09:00 - 17:00 Thursday 09:00 - 17:00 Friday 09:00 - 17:00

http://www.ctnlpr.com/, https://github.com/ctnlpr26, https://huggingface.co/ctnlpr

Center for Tamil Natural Language Processing Research, 63, Sir Pon, Thirunelvelly, Ramanathan Road, Kallady, Jaffna (2026)

10/07/2026

🚀 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗙𝗶𝗻𝗲-𝗚𝗿𝗮𝗶𝗻𝗲𝗱 𝗧𝗮𝗺𝗶𝗹 𝗡𝗮𝗺𝗲𝗱 𝗘𝗻𝘁𝗶𝘁𝘆 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻 (𝗙𝗴𝗡𝗘𝗥) 𝗦𝘆𝘀𝘁𝗲𝗺 𝘄𝗶𝘁𝗵 𝗚𝗼𝗼𝗴𝗹𝗲 𝗠𝘂𝗥𝗜𝗟

Named Entity Recognition (𝗡𝗘𝗥) is one of the core tasks in 𝗡𝗮𝘁𝘂𝗿𝗮𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 (𝗡𝗟𝗣), enabling AI systems to identify and classify real-world entities such as people, organizations, locations, products, events, and more.

Most existing Tamil NER systems focus only on three coarse-grained categories—𝗣𝗘𝗥𝗦𝗢𝗡, 𝗟𝗢𝗖𝗔𝗧𝗜𝗢𝗡, and 𝗢𝗥𝗚𝗔𝗡𝗜𝗭𝗔𝗧𝗜𝗢𝗡. While suitable for basic information extraction, these categories are often insufficient for advanced document intelligence, semantic search, and knowledge graph applications.

As part of ongoing research at 𝗖𝗧𝗡𝗟𝗣𝗥, we explored a different direction by fine-tuning 𝗚𝗼𝗼𝗴𝗹𝗲 𝗠𝘂𝗥𝗜𝗟 on the 𝗦𝗮𝗺𝗽𝘂𝗿𝗡𝗘𝗥 Fine-Grained Tamil NER dataset to enable richer semantic understanding of Tamil documents.

━━━━━━━━━━━━━━━━━━━━
𝗪𝗵𝘆 𝗙𝗶𝗻𝗲-𝗚𝗿𝗮𝗶𝗻𝗲𝗱 𝗡𝗘𝗥?
━━━━━━━━━━━━━━━━━━━━

Instead of assigning broad labels like PERSON or ORGANIZATION, Fine-Grained NER recognizes much more specific entity types.

Examples include:

✓ Government Organizations
✓ Educational Institutions
✓ Hospitals
✓ Buildings
✓ Products
✓ Events
✓ Diseases
✓ Creative Works
✓ Athletes
✓ Artists
✓ Writers
✓ Scholars
✓ Biological Entities

This richer semantic representation significantly improves downstream AI systems such as 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻, 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗚𝗿𝗮𝗽𝗵𝘀, 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗦𝗲𝗮𝗿𝗰𝗵, and 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹-𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚).

━━━━━━━━━━━━━━━━━━━━
𝗪𝗵𝘆 𝗠𝘂𝗥𝗜𝗟?
━━━━━━━━━━━━━━━━━━━━

Unlike general multilingual transformer models, 𝗠𝘂𝗥𝗜𝗟 was specifically pretrained for Indian languages using:

✓ Large-scale Indian language corpora
✓ Parallel translated sentence pairs
✓ Transliterated document pairs
✓ Multilingual Masked Language Modeling

Because Tamil is one of MuRIL's primary target languages, it provides a strong contextual representation for Fine-Grained Tamil Named Entity Recognition.

━━━━━━━━━━━━━━━━━━━━
𝗠𝗼𝗱𝗲𝗹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲
━━━━━━━━━━━━━━━━━━━━

The overall system follows a transformer-based token classification pipeline:

𝗦𝗮𝗺𝗽𝘂𝗿𝗡𝗘𝗥 𝗗𝗮𝘁𝗮𝘀𝗲𝘁
↓
MuRIL Tokenizer
↓
WordPiece Tokenization
↓
BIO Sequence Labeling
↓
Transformer-Based Token Classification
↓
Fine-Grained Entity Prediction

Rather than modifying the transformer architecture, the model specializes MuRIL through supervised fine-tuning for fine-grained semantic classification.

━━━━━━━━━━━━━━━━━━━━
𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀
━━━━━━━━━━━━━━━━━━━━

During experimentation, several challenges were observed:

✓ High-cardinality semantic label space
✓ Agglutinative Tamil morphology
✓ Long multi-token entity spans
✓ Fine-grained semantic ambiguity
✓ Context-dependent classification

These challenges require the model to rely on contextual semantic understanding rather than simple lexical matching.

━━━━━━━━━━━━━━━━━━━━
𝗠𝗼𝗱𝗲𝗹 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝘁𝗶𝗼𝗻𝘀
━━━━━━━━━━━━━━━━━━━━

The fine-tuned MuRIL model demonstrated strong performance across multiple entity categories, including:

✓ Person
✓ Organization
✓ Government Organization
✓ Educational Institution
✓ Buildings
✓ Hospitals
✓ Products
✓ Creative Works

The multilingual representations learned during MuRIL pretraining transferred effectively to Fine-Grained Tamil NER through supervised task adaptation.

━━━━━━━━━━━━━━━━━━━━
𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀
━━━━━━━━━━━━━━━━━━━━

Fine-Grained Tamil NER enables richer semantic understanding for:

✓ Information Extraction
✓ Entity Linking
✓ Semantic Search
✓ Knowledge Graph Population
✓ Retrieval-Augmented Generation (RAG)
✓ Digital Libraries
✓ Document Intelligence
✓ Tamil Question Answering
✓ Large-Scale Archive Processing

━━━━━━━━━━━━━━━━━━━━
𝗞𝗲𝘆 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁
━━━━━━━━━━━━━━━━━━━━

One of the key findings is that the limitation of conventional Tamil NER is not entity detection itself, but semantic granularity.

Identifying an entity simply as PERSON or ORGANIZATION provides limited value for downstream reasoning. Fine-Grained NER preserves domain-specific semantic distinctions, enabling richer knowledge representation and more expressive AI applications.

━━━━━━━━━━━━━━━━━━━━
𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗮𝘁 𝗖𝗧𝗡𝗟𝗣𝗥
━━━━━━━━━━━━━━━━━━━━

The current implementation provides:

✓ Transformer-based token classification using MuRIL
✓ BIO sequence labeling
✓ Fine-grained semantic entity classification
✓ High-cardinality entity taxonomy
✓ Context-aware multilingual representations
✓ Integration-ready outputs for Information Extraction and Knowledge Graph Construction

━━━━━━━━━━━━━━━━━━━━
𝗙𝘂𝗹𝗹 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗕𝗹𝗼𝗴
━━━━━━━━━━━━━━━━━━━━

🔗 https://www.ctnlpr.com/2026/07/08/building-a-fine-grained-tamil-named-entity-recognition-system-with-muril/

━━━━━━━━━━━━━━━━━━━━

24/06/2026

🚀 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿-𝗕𝗮𝘀𝗲𝗱 𝗧𝗮𝗺𝗶𝗹 𝗥𝗲𝗹𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗳𝗼𝗿 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗚𝗿𝗮𝗽𝗵 𝗖𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻

Knowledge Graph construction from unstructured text remains one of the fundamental challenges in 𝗡𝗮𝘁𝘂𝗿𝗮𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 (𝗡𝗟𝗣).

For low-resource languages such as 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹, this challenge is even more significant due to limited annotated datasets, domain-specific resources, and production-ready information extraction systems.

As part of ongoing research at 𝗖𝗧𝗡𝗟𝗣𝗥, we designed and implemented a 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿-𝗯𝗮𝘀𝗲𝗱 𝗧𝗮𝗺𝗶𝗹 𝗥𝗲𝗹𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 capable of converting unstructured Tamil documents into structured knowledge triples.

━━━━━━━━━━━━━━━━━━━━
𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗠𝗼𝘁𝗶𝘃𝗮𝘁𝗶𝗼𝗻
━━━━━━━━━━━━━━━━━━━━

Consider the Tamil sentence:

சாம்.ஏ.சபாபதி யாழ்ப்பாணத்தில் வாழ்ந்தார்.

Humans can easily infer:

(சாம்.ஏ.சபாபதி, LIVED_IN, யாழ்ப்பாணம்)

However, an NLP system must solve multiple sub-problems before arriving at this representation:

✓ Sentence Segmentation
✓ Named Entity Recognition
✓ Entity Typing
✓ Entity Pair Generation
✓ Relation Classification
✓ Structured Triple Construction

These tasks form the foundation of modern 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 and 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗚𝗿𝗮𝗽𝗵 systems.

━━━━━━━━━━━━━━━━━━━━
𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲
━━━━━━━━━━━━━━━━━━━━

The system follows a modular pipeline design:

𝗗𝗼𝗰𝘂𝗺𝗲𝗻𝘁
→ Sentence Splitter
→ Named Entity Recognition
→ Pair Generator
→ Entity Marker
→ Relation Extraction
→ Triple Builder

Unlike end-to-end extraction architectures, a pipeline-based approach was selected due to:

✓ Modularity
✓ Interpretability
✓ Ease of debugging
✓ Suitability for low-resource NLP environments

Each component can be independently evaluated, improved, or replaced without affecting the overall system.

━━━━━━━━━━━━━━━━━━━━
𝗖𝘂𝗿𝗿𝗲𝗻𝘁 𝗖𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀
━━━━━━━━━━━━━━━━━━━━

The current implementation supports:

✓ Sentence Segmentation

✓ Named Entity Recognition

Entity Types:

• PERSON
• LOCATION
• ORGANIZATION

✓ Entity Pair Generation

Supported Semantic Pair Constraints:

• PERSON ↔ LOCATION
• PERSON ↔ ORGANIZATION
• ORGANIZATION ↔ LOCATION

✓ Transformer-Based Relation Extraction

Current Relation Labels:

• LIVED_IN
• WORKED_AT
• CONTRIBUTED_TO
• PART_OF
• BORN_IN

✓ Knowledge Triple Construction

━━━━━━━━━━━━━━━━━━━━
𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵
━━━━━━━━━━━━━━━━━━━━

The current system uses a multilingual 𝗡𝗮𝘁𝘂𝗿𝗮𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 (𝗡𝗟𝗜) Transformer model in a 𝘇𝗲𝗿𝗼-𝘀𝗵𝗼𝘁 setting for relation classification.

This approach enables rapid experimentation without requiring a dedicated Tamil Relation Extraction dataset.

Entity mentions are explicitly marked before classification:

[E1] சாம்பசிவம் [/E1]

[E2] யாழ்ப்பாணத்தில் [/E2]

This helps the transformer focus on target entities when predicting semantic relationships.

━━━━━━━━━━━━━━━━━━━━
𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀
━━━━━━━━━━━━━━━━━━━━

During experimentation, several practical challenges were observed:

✓ Tamil Unicode combining-character boundary issues

✓ Tokenization inconsistencies

✓ Entity span normalization

✓ Candidate relation filtering

✓ Low-resource language constraints

To address these issues, an entity post-processing layer was introduced to improve boundary consistency and downstream extraction quality.

━━━━━━━━━━━━━━━━━━━━
𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗢𝘂𝘁𝗰𝗼𝗺𝗲𝘀
━━━━━━━━━━━━━━━━━━━━

The pipeline was evaluated on:

✓ Personal biographies

✓ Academic affiliations

✓ Institutional relationships

✓ Historical narratives

✓ Sri Lankan Tamil named entities

The experiments demonstrate that the architecture can successfully extract meaningful knowledge triples from Tamil text and provide a strong foundation for future Knowledge Graph construction.

━━━━━━━━━━━━━━━━━━━━
𝗙𝘂𝘁𝘂𝗿𝗲 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗗𝗶𝗿𝗲𝗰𝘁𝗶𝗼𝗻𝘀
━━━━━━━━━━━━━━━━━━━━

Coreference Resolution
↓
Entity Canonicalization
↓
Ontology Validation
↓
Relation Filtering
↓
Knowledge Graph Construction

Additional research areas include:

✓ MuRIL-based NER

✓ XLM-R-based NER

✓ Fine-tuned Tamil Relation Extraction Models

✓ Document-Level Relation Extraction

✓ Ontology-Aware Triple Validation

✓ Knowledge Graph Population from Noolaham Archives

━━━━━━━━━━━━━━━━━━━━
𝗙𝘂𝗹𝗹 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗕𝗹𝗼𝗴
https://www.ctnlpr.com/2026/06/23/building-a-transformer-based-tamil-relation-extraction-pipeline-for-knowledge-graph-construction/
━━━━━━━━━━━━━━━━━━━━

15/06/2026

🚀𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗧𝗮𝗺𝗶𝗹 𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗥𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗦𝘆𝘀𝘁𝗲𝗺 𝗳𝗼𝗿 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻

Modern Information Extraction systems rely on more than Named Entity Recognition (NER). While NER can identify entities such as people, locations, and organizations, it does not explain how references to those entities evolve throughout a document.

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗥𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 (𝗖𝗥) 𝗯𝗲𝗰𝗼𝗺𝗲𝘀 𝗲𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹.

Coreference Resolution is the task of determining whether multiple mentions within a document refer to the same real-world entity. It acts as a critical bridge between entity recognition and structured knowledge extraction.

𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀:

• Knowledge Graph Construction
• Relation Extraction
• Semantic Search
• Document Intelligence
• Retrieval-Augmented Generation (RAG)
• Conversational AI

Accurate coreference resolution is often the difference between fragmented information and coherent knowledge.

𝗧𝗵𝗲 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 𝗣𝗿𝗼𝗯𝗹𝗲𝗺

During our work at CTNLPR, we observed a common challenge in Tamil document processing.

Documents rarely repeat the full entity name in every sentence. Instead, they rely on:

• Pronouns
• Possessive references
• Descriptive noun phrases
• Location references

Humans resolve these references naturally using context. Machines do not.

When we use coreference resolution The extracted knowledge becomes meaningful and directly usable within downstream systems.

𝗪𝗵𝘆 𝗪𝗲 𝗗𝗶𝗱 𝗡𝗼𝘁 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗡𝗲𝘂𝗿𝗮𝗹 𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗠𝗼𝗱𝗲𝗹𝘀

Most modern coreference systems rely on:

• Transformer-based architectures
• Mention-ranking models
• End-to-end neural systems
• Span-ranking approaches

While highly effective for English and other high-resource languages, they typically require:

• Large annotated datasets
• Extensive model training
• Significant computational resources
• Language-specific supervision

Tamil currently lacks large-scale publicly available coreference corpora.

Instead of waiting for benchmark datasets, we explored a different direction:

𝗕𝘂𝗶𝗹𝗱 𝗮 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰, 𝗲𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗹𝗲, 𝗮𝗻𝗱 𝘁𝗮𝘀𝗸-𝗼𝗿𝗶𝗲𝗻𝘁𝗲𝗱 𝗰𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗳𝗼𝗿 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻.

Our goal was not to compete with neural benchmarks.

Our goal was to improve relation extraction quality in real-world Tamil document processing.

𝗦𝘆𝘀𝘁𝗲𝗺 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲

Tamil Document
↓
Text Normalization
↓
Sentence Segmentation
↓
Mention Detection
↓
Entity Normalization
↓
Entity Memory
↓
Coreference Resolution
↓
Coreference Chain Construction
↓
Visualization Layer

Each layer contributes toward discourse-level entity understanding.

𝗠𝗲𝗻𝘁𝗶𝗼𝗻 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻

The mention detection layer combines multiple strategies:

• Named Entity Recognition (PERSON, LOCATION, ORGANIZATION)
• Pronoun Detection
• Location Reference Detection
• Rule-Based Noun Phrase Detection

These mentions become candidates for resolution.

𝗘𝗻𝘁𝗶𝘁𝘆 𝗡𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻

Tamil's rich morphology creates multiple surface forms for the same entity.

𝗘𝘅𝗮𝗺𝗽𝗹𝗲:

┌────────────────┐
யாழ்ப்பாணம்

யாழ்ப்பாணத்தில்

யாழ்ப்பாணத்தின்

யாழ்ப்பாணத்திற்கு
└────────────────┘

Using Stanza lemmatization:

┌─────────────────┐
யாழ்ப்பாணத்தில்
↓
யாழ்ப்பாணம்
└─────────────────┘

This reduces entity fragmentation and improves linking consistency.

𝗧𝗵𝗲 𝗘𝗻𝘁𝗶𝘁𝘆 𝗠𝗲𝗺𝗼𝗿𝘆 𝗟𝗮𝘆𝗲𝗿

One of the key design decisions was introducing a lightweight discourse memory.

Instead of neural antecedent scoring, the system maintains contextual entity state:

last_person
last_location
last_org

Whenever a new entity is detected, the corresponding memory state is updated.

This memory acts as the document's discourse context.

𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗥𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻

Once discourse memory has been established, the resolver links newly encountered mentions to previously observed canonical entities.

The system performs:

• Person Pronoun Resolution
• Possessive Resolution
• Location Resolution
• Rule-Based Noun Phrase Resolution

By maintaining discourse state across sentence boundaries, fragmented references are transformed into consistent entity representations.

This significantly improves downstream relation extraction quality.

𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗖𝗵𝗮𝗶𝗻 𝗖𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻

Rather than replacing mentions individually, the system groups related mentions into entity clusters.

consider this example:-

சாம்.ஏ.சபாபதி ஒரு பிரபல சமூக சேவகர். அவர் யாழ்ப்பாணத்தில் பல கல்வித் திட்டங்களை முன்னெடுத்தார். இந்த சமூகசேவகர் பல விருதுகளை பெற்றுள்ளார். அவர் யாழ்ப்பாணத்தில் பிறந்தார். அங்கு அவருக்கு பெரும் மதிப்பு இருந்தது. இவர் யாழ்ப்பாண நூலகத்தின் உருவாக்கத்தில் முக்கிய பங்காற்றினார்.

┌─────────────────────┐
Entity: சாம்.ஏ.சபாபதி

├── சாம்.ஏ.சபாபதி
├── அவர்
├── இந்த சமூகசேவகர்
├── அவர்
└── இவர்

Entity: யாழ்ப்பாணம்

├── யாழ்ப்பாணம்
└── அங்கு
└─────────────────────┘

These chains provide a document-level view of entity references.

Useful for:

• Debugging
• Evaluation
• Knowledge Graph Construction
• Relation Extraction

𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 & 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀

To support experimentation and validation, we developed a Streamlit-based visualization layer.

Users can:

• Submit Tamil documents
• Inspect generated coreference chains
• Analyze entity clusters
• Validate resolution decisions

This provides transparency into the resolution process and helps identify weaknesses in rule design.

𝗞𝗲𝘆 𝗜𝗻𝘀𝗶𝗴𝗵𝘁

𝗖𝗼𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗥𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗶𝘀 𝗻𝗼𝘁 𝗺𝗲𝗿𝗲𝗹𝘆 𝗮 𝗽𝗿𝗼𝗻𝗼𝘂𝗻-𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝘁𝗮𝘀𝗸.

It is an entity consistency layer that connects:

• Named Entity Recognition
• Relation Extraction
• Knowledge Graph Construction
• Semantic Search
• RAG Systems

Without coreference:

┌─────────────────┐
(இவர், பங்காற்றினார்
, யாழ்ப்பாண நூலகம்)
└─────────────────┘

With coreference:

┌───────────────────────┐
(சாம்.ஏ.சபாபதி, பங்காற்றினார்
, யாழ்ப்பாண நூலகம்)
└───────────────────────┘

The second representation is immediately usable within structured knowledge systems.

𝗦𝘆𝘀𝘁𝗲𝗺 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗮𝘁 𝗖𝗧𝗡𝗟𝗣𝗥

Current capabilities include:

✅ Named Entity Recognition
✅ Entity Normalization
✅ Pronoun Resolution
✅ Possessive Resolution
✅ Location Resolution
✅ Rule-Based Noun Phrase Resolution
✅ Coreference Chain Construction
✅ Streamlit-Based Visualization

The system acts as a foundational layer between entity extraction and knowledge graph generation.

𝗙𝘂𝘁𝘂𝗿𝗲 𝗪𝗼𝗿𝗸

• Multi-entity discourse memory
• Entity salience tracking
• Advanced noun phrase resolution
• Relation-aware coreference resolution
• Knowledge graph integration
• Hybrid neural-rule architectures

𝗖𝗼𝗻𝗰𝗹𝘂𝘀𝗶𝗼𝗻

Building effective Tamil Information Extraction systems requires more than Named Entity Recognition.

By introducing a dedicated coreference resolution layer, we can maintain entity consistency across documents, improve relation extraction quality, and generate more reliable structured knowledge.

For low-resource languages such as Tamil, carefully designed rule-based systems remain a practical and effective pathway toward document-level semantic understanding while larger neural approaches continue to mature.

03/06/2026

⚡️𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗜𝗻𝗱𝗶𝗰𝗡𝗘𝗥 𝗳𝗼𝗿 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗮𝗺𝗲𝗱 𝗘𝗻𝘁𝗶𝘁𝘆 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻

Transformer-based multilingual NLP systems have significantly improved Named Entity Recognition (NER) across many languages. However, low-resource language variants such as Sri Lankan Tamil still face substantial challenges due to limited domain-specific datasets and linguistic underrepresentation.

At CTNLPR, we fine-tuned 𝗮𝗶𝟰𝗯𝗵𝗮𝗿𝗮𝘁/𝗜𝗻𝗱𝗶𝗰𝗡𝗘𝗥 specifically for Sri Lankan Tamil using a custom annotated NER corpus.

𝗢𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲

Improve entity recognition for:

• Sri Lankan Tamil linguistic patterns
• Local person, location, and organization names
• Morphology-aware contextual variations

𝗪𝗵𝘆 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥 𝗶𝘀 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗶𝗻𝗴

Most multilingual NER systems are trained primarily on:

• General web corpora
• Indian Tamil datasets
• Multilingual benchmark datasets
• Formal textual sources

When applied to Sri Lankan Tamil, they often struggle with:

• Regional naming conventions
• Local organization terminology
• Morphological suffix complexity
• OCR-induced token inconsistencies
• Subword tokenization fragmentation
• Ambiguous entity boundaries

These limitations directly affect downstream systems such as:

• Semantic Search
• Document Intelligence
• Knowledge Graph Construction
• Tamil Chatbots
• RAG Systems
• Government Document Processing

𝗠𝗼𝗱𝗲𝗹 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄

𝗕𝗮𝘀𝗲 𝗠𝗼𝗱𝗲𝗹

→ ai4bharat/IndicNER

𝗘𝗻𝘁𝗶𝘁𝘆 𝗧𝘆𝗽𝗲𝘀

• PERSON
• LOCATION
• ORGANIZATION

𝗞𝗲𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀

✅ Tamil-safe Tokenization
✅ Unicode Normalization
✅ BIO Tagging
✅ Proper Subword Label Alignment
✅ Morphology-aware Training
✅ OCR-aware Preprocessing

𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀

1️⃣ 𝗧𝗮𝗺𝗶𝗹 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻

Tamil is morphologically rich. Incorrect tokenization can cause:

• Broken entity spans
• Incorrect BIO labels
• Fragmented predictions

2️⃣ 𝗦𝘂𝗯𝘄𝗼𝗿𝗱 𝗟𝗮𝗯𝗲𝗹 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁

Transformer tokenizers frequently split Tamil words into multiple subword units.

Without proper alignment:

• Entity spans become corrupted
• BIO labels mismatch
• Training instability increases

3️⃣ 𝗢𝗖𝗥 𝗡𝗼𝗶𝘀𝗲

Tamil OCR systems still generate:

• Grapheme inconsistencies
• Merged tokens
• Invalid Unicode combinations
• Punctuation corruption

Therefore OCR-aware normalization was integrated before training.

𝗠𝗼𝗱𝗲𝗹 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻

𝗢𝘃𝗲𝗿𝗮𝗹𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲

• F1 Score → 0.650
• Precision → 0.602
• Recall → 0.707
• Accuracy → 96.04%

𝗘𝗻𝘁𝗶𝘁𝘆-𝘄𝗶𝘀𝗲 𝗙𝟭

• PERSON → 0.721
• LOCATION → 0.698
• ORGANIZATION → 0.484

PERSON and LOCATION categories achieved relatively strong performance, while ORGANIZATION entities remain the most challenging category.

𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗘𝘅𝗮𝗺𝗽𝗹𝗲𝘀

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟭

Sentence:
"பாரதிதாசன் எழுதிய நூலை பாரதி பதிப்பகம் வெளியிட்டது."

Output:
👤 PERSON → பாரதிதாசன்
🏢 ORGANIZATION → பாரதி பதிப்பகம்

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟮

Sentence:
"வடமராட்சி தொழில்நுட்ப நிறுவனம் மாணவர்களை சேர்த்தது."

Output:
🏢 ORGANIZATION → வடமராட்சி தொழில்நுட்ப நிறுவனம்

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟯

Sentence:
"நவமணி கிராமம் வெள்ளத்தால் பாதிக்கப்பட்டது."

Output:
📍 LOCATION → நவமணி

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟰

Sentence:
"ஜே.ஏ.எஸ்.பீ. ஜயசிங்க நவமணி கிராமத்திற்கு சென்றார்."

Output:
👤 PERSON → ஜே.ஏ.எஸ்.பீ. ஜயசிங்க
📍 LOCATION → நவமணி

𝗞𝗲𝘆 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝘁𝗶𝗼𝗻

One of the most important findings from this work is:

"Better preprocessing and domain-specific data can be as important as model architecture."

For low-resource languages like Sri Lankan Tamil:

• High-quality annotations matter
• OCR normalization matters
• Tokenizer alignment matters
• Linguistic preprocessing matters

Large transformer architectures alone are not sufficient without carefully prepared language-specific datasets.

𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀

• Tamil NER Systems
• Semantic Search
• RAG Pipelines
• OCR Information Extraction
• Knowledge Graph Construction
• Tamil Chatbots

This work is part of ongoing Tamil NLP research at CTNLPR aimed at building stronger NLP infrastructure for low-resource Tamil language technologies.

25/05/2026

🌟𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐒𝐫𝐢 𝐋𝐚𝐧𝐤𝐚𝐧 𝐓𝐚𝐦𝐢𝐥 𝐍𝐚𝐦𝐞𝐝 𝐄𝐧𝐭𝐢𝐭𝐲 𝐑𝐞𝐜𝐨𝐠𝐧𝐢𝐭𝐢𝐨𝐧 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐋𝐨𝐰-𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐍𝐋𝐏

The growth of Large Language Models (LLMs) and multilingual NLP systems has significantly improved language technologies across major global languages. However, low-resource languages such as Sri Lankan Tamil still face a severe lack of high-quality annotated datasets—especially for foundational tasks like Named Entity Recognition (NER).

To address this gap, we developed the Srilankan-Tamil-NER Dataset, a Tamil NER dataset designed specifically for Sri Lankan Tamil linguistic and contextual usage.

This dataset is intended to support:

• Tamil NER research
• Indic language fine-tuning
• Information extraction systems
• Retrieval-Augmented Generation (RAG)
• Tamil LLM adaptation
• Domain-specific AI systems for Sri Lanka

𝗪𝗵𝘆 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥 𝗠𝗮𝘁𝘁𝗲𝗿𝘀

Named Entity Recognition (NER) is a core NLP task that identifies and classifies entities such as:

• Person names
• Locations
• Organizations
• Dates
• Miscellaneous entities

NER acts as a foundational layer for many downstream NLP systems including:

• Question answering
• Search systems
• Chatbots
• Document intelligence
• Machine translation
• Knowledge graph generation

For Tamil — particularly Sri Lankan Tamil — publicly available annotated corpora remain extremely limited. Existing multilingual datasets often underrepresent regional linguistic variations, local named entities, and culturally contextual terminology.

Most existing NER systems for Tamil are trained on datasets originating from Indian Tamil corpora, leaving significant gaps in handling:

• Sri Lankan Tamil vocabulary
• Local organization names
• Sri Lankan place names
• Government and institutional terminology

Our dataset aims to bridge this gap.

𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮𝘀𝗲𝘁

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗡𝗮𝗺𝗲:
Srilankan-Tamil-NER Dataset

The primary goal of this dataset is to create a high-quality manually curated Named Entity Recognition corpus for Sri Lankan Tamil under CTNLPR.

The dataset is structured to support fine-tuning transformer-based multilingual models such as:

• IndicNER
• mBERT
• XLM-RoBERTa
• MuRIL
• IndicBERT

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝘀

• B-PER (Person): 4,533
• B-LOC (Location): 8,110
• B-ORG (Organization): 3,369
• Total Entities: 16,012

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗣𝗿𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲

Creating a Tamil NER dataset involves significantly more than simple annotation.

The preparation workflow included multiple stages:

1. 𝑫𝒂𝒕𝒂 𝑪𝒐𝒍𝒍𝒆𝒄𝒕𝒊𝒐𝒏

The raw Tamil text corpus was collected from the Noolaham corpus and other relevant publicly available Sri Lankan Tamil textual sources.

Special attention was given to:

• Local linguistic relevance
• Entity diversity
• Sentence quality
• Contextual richness

The objective was to capture realistic Sri Lankan Tamil usage patterns rather than synthetic or translated text.

2. 𝑶𝑪𝑹 𝒂𝒏𝒅 𝑻𝒆𝒙𝒕 𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒂𝒕𝒊𝒐𝒏

Tamil NLP pipelines often begin with scanned or image-based documents.

As part of our broader Tamil document intelligence workflow, OCR-extracted Tamil text underwent:

• Unicode normalization
• Punctuation cleaning
• Whitespace normalization
• Invalid character filtering
• OCR noise reduction

OCR-related preprocessing becomes extremely important because Tamil script errors can propagate heavily into token classification systems.

3. 𝑵𝒂𝒎𝒆𝒅 𝑬𝒏𝒕𝒊𝒕𝒚 𝑨𝒏𝒏𝒐𝒕𝒂𝒕𝒊𝒐𝒏

The dataset was manually annotated using BIO tagging format.

Entity Types:

• B-PER — Beginning of person entity
• I-PER — Inside person entity
• B-LOC — Beginning of location entity
• I-LOC — Inside location entity
• B-ORG — Beginning of organization entity
• I-ORG — Inside organization entity
• O — Non-entity token

Example:

இராமநாதன் → B-PER
யாழ்ப்பாணம் → B-LOC
பல்கலைக்கழகம் → B-ORG

𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗶𝗻 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥

Building a Tamil NER dataset introduced several language-specific challenges.

• Morphological complexity
• OCR noise
• Unicode inconsistencies
• Token boundary detection
• Subword alignment
• Limited benchmark corpora

𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀

This dataset can support:

• Tamil NER
• OCR post-processing
• Semantic search systems
• RAG pipelines
• Tamil chatbots
• Government document AI
• Knowledge graph generation

The Srilankan-Tamil-NER Dataset, developed under CTNLPR, represents an important step toward strengthening the Sri Lankan Tamil NLP ecosystem through high-quality entity annotation and linguistically relevant corpus preparation.

#𝑆𝑟𝑖𝐿𝑎𝑛𝑘𝑎𝑛𝑇𝑎𝑚𝑖𝑙 #𝑇𝑎𝑚𝑖𝑙𝑁𝐸𝑅 #𝑇𝑎𝑚𝑖𝑙𝑁𝐿𝑃 #𝑁𝑎𝑚𝑒𝑑𝐸𝑛𝑡𝑖𝑡𝑦𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛 #𝐿𝑜𝑤𝑅𝑒𝑠𝑜𝑢𝑟𝑐𝑒𝑁𝐿𝑃 #𝐼𝑛𝑑𝑖𝑐𝑁𝐿𝑃 #𝑀𝑢𝑅𝐼𝐿 #𝑚𝐵𝐸𝑅𝑇 #𝑋𝐿𝑀𝑅 #𝐵𝐼𝑂𝑡𝑎𝑔𝑔𝑖𝑛𝑔 #𝐿𝐿𝑀 #𝑅𝐴𝐺 #𝑆𝑒𝑚𝑎𝑛𝑡𝑖𝑐𝑆𝑒𝑎𝑟𝑐ℎ #𝐾𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒𝐺𝑟𝑎𝑝ℎ𝑠 #𝐴𝐼𝐸𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑖𝑛𝑔 #𝐷𝑜𝑐𝑢𝑚𝑒𝑛𝑡𝐼𝑛𝑡𝑒𝑙𝑙𝑖𝑔𝑒𝑛𝑐𝑒 #𝐸𝑛𝑡𝑖𝑡𝑦𝐸𝑥𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 #𝑂𝐶𝑅 #𝑇𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚𝑒𝑟𝑀𝑜𝑑𝑒𝑙𝑠 #𝐶𝑜𝑚𝑝𝑢𝑡𝑎𝑡𝑖𝑜𝑛𝑎𝑙𝐿𝑖𝑛𝑔𝑢𝑖𝑠𝑡𝑖𝑐𝑠 #𝐶𝑇𝑁𝐿𝑃𝑅

15/05/2026

⚡️ 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐓𝐚𝐦𝐢𝐥 𝐂𝐨𝐫𝐞𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐑𝐞𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐒𝐲𝐬𝐭𝐞𝐦: 𝐙𝐞𝐫𝐨-𝐒𝐡𝐨𝐭 𝐂𝐨𝐧𝐭𝐞𝐱𝐭𝐮𝐚𝐥 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐌𝐮𝐑𝐈𝐋

Coreference Resolution (CR) is a critical NLP task for identifying whether multiple mentions in a document refer to the same real-world entity. It plays a major role in:

• Knowledge Graph Construction
• Relation Extraction
• Semantic Search
• RAG Systems
• Conversational AI
• Document-Level Understanding

While English NLP already has mature coreference systems and libraries, Tamil remains a highly challenging low-resource language for discourse-level semantic modeling.

At CTNLPR, we explored how modern multilingual coreference architectures can be adapted for Tamil using zero-shot contextual semantic modeling instead of heavily supervised pipelines.

Tamil introduces several difficult linguistic challenges:

• Agglutinative morphology
• Free word order
• Pronoun dropping
• Implicit subject references
• Rich inflectional structures
• Long-distance discourse dependencies
• Noun-to-noun semantic references

Additionally:

• No dedicated Tamil coreference libraries currently exist publicly
• Large annotated Tamil CR datasets are unavailable
• Most multilingual systems remain heavily English-biased

⚙️ Our Architecture

The architecture currently being explored at CTNLPR uses:

• MuRIL-based contextual embeddings
• Span-based mention detection
• Contextual span representations
• Cosine similarity-based semantic linking
• Agglomerative clustering

Instead of manually defining antecedents, the system automatically generates semantic mention spans from Tamil text and groups semantically related mentions into discourse-level entity chains using contextual similarity.

🧠 Key Technical Direction

Traditional supervised coreference systems depend heavily on:

• Large annotated corpora
• Expensive training pipelines
• Language-specific supervision
• Antecedent ranking architectures
• High computational cost

For Tamil, these resources are extremely limited.

Our approach avoids heavy annotation dependency while still leveraging multilingual transformer-based semantic understanding learned from Indian-language pretraining.

Each candidate span is encoded using contextual embeddings generated from MuRIL, and span-level semantic representations are constructed using contextual token pooling. The system then performs:

• Heuristic span pruning
• Semantic similarity computation
• Similarity-driven clustering

to generate discourse-level coreference chains.

🚀 Key Advantages

• Zero-shot inference
• Low-resource scalability
• Context-aware semantic reasoning
• Better adaptation to Tamil morphology
• Lightweight unsupervised inference
• Reduced annotation dependency

This architecture is being explored at CTNLPR as a foundation for:

• Tamil discourse understanding
• Entity-aware semantic linking
• Knowledge Graph Construction
• Ontology-aware NLP
• Multilingual semantic reasoning
• Advanced RAG systems

🔬 Building document-level semantic understanding for Tamil is one of the next major steps toward scalable low-resource AI systems.

04/05/2026

⚡𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐌𝐨𝐫𝐩𝐡𝐨𝐥𝐨𝐠𝐲-𝐀𝐰𝐚𝐫𝐞 𝐓𝐚𝐦𝐢𝐥 𝐍𝐄𝐑 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞: 𝐅𝐫𝐨𝐦 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫 𝐄𝐱𝐭𝐫𝐚𝐜𝐭𝐢𝐨𝐧 𝐭𝐨 𝐂𝐚𝐧𝐨𝐧𝐢𝐜𝐚𝐥 𝐄𝐧𝐭𝐢𝐭𝐲 𝐑𝐞𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧

Named Entity Recognition (NER) is a critical layer in our Tamil NLP stack (search, indexing, knowledge graph construction, and RAG).
However, for Tamil, extracting entities is only half the problem — canonicalizing them is the real challenge.

💥 What We Built in CTNLPR

We designed a Tamil-aware NER pipeline by extending a transformer-based model with morphological normalization:

• ai4bharat/IndicNER → baseline entity extraction
• Custom span merging → IOB consolidation
• Prefix-based grouping → variant clustering
• Morphological normalization layer → canonical entity resolution

Since Indic NER models are not morphology-aware, we explicitly evaluated and integrated normalization strategies.

🔰Method Exploration (What We Tried)

• IndicNER (Transformer baseline)
✅ Strong recall across entity types
❎ Produces multiple inflected variants of the same entity

• Prefix-based grouping
✅ Fast heuristic clustering
❎ Not linguistically grounded

• UoM Thamizhi Morphological Normalizer (University of Moratuwa)
✅ Linguistically motivated rule-based approach
❎ Limited effectiveness on real-world data
❎ Struggled with:

* Noisy OCR text
* Complex suffix chains
* Unseen word forms

• Tamil Lemmatizer (final approach)
✅ Consistent root-form extraction
✅ Robust across inflected variants
✅ Best empirical performance in our pipeline

🔬 Key Design Decision

Transformer models do not enforce canonical forms.

👉 Surface forms like:
• இலங்கையில்
• இலங்கையிலும்
• இலங்கையிலே

are extracted as separate entities

👉 After normalization:
• இலங்கை

This enables many-to-one mapping, critical for system consistency.

🧩 Our Setup

Pipeline:

• Document → chunking
• Transformer inference (IndicNER)
• IOB span merging + filtering
• Variant aggregation (prefix-based)

• Morphological normalization (UoM explored → Lemmatizer selected)
• Entity re-indexing

Example:

இலங்கையில் → இலங்கை
இலங்கையிலும் → இலங்கை
இந்தியாவில் → இந்தியா

➡️ System-Level Challenges We Solved

• Agglutinative suffix handling
• Variant explosion in entity outputs
• OCR/noisy input robustness
• Canonical entity consistency across documents

📊 What We Observed

• Transformer NER → high recall, low canonical consistency
• UoM morphological normalizer → linguistically sound but limited robustness
• Lemmatizer → best normalization performance in practice

🌟 Final system:
IndicNER + Lemmatization (hybrid architecture)

✳️ Key Insight

In Tamil NER, the challenge is not detection —
it is morphological normalization.

NER output ≠ final entity

🌟 Canonicalization is essential for:
• Indexing
• Entity linking
• Knowledge graphs
• RAG systems

🚀 Outcome

We built a production-ready Tamil NER system that:

• Resolves inflected entity variants
• Produces stable canonical forms
• Improves downstream retrieval and analytics
• Scales across multi-document pipelines

🔬 This work is part of ongoing Tamil NLP system development at CTNLPR

Center for Tamil Natural Language Processing Research

10/07/2026

24/06/2026

15/06/2026

03/06/2026

25/05/2026

15/05/2026

04/05/2026

Address

Opening Hours

Telephone

Website

Alerts

Contact The School

Shortcuts

Share

Category

Listing takedown request

Monday	09:00 - 17:00
Tuesday	09:00 - 17:00
Wednesday	09:00 - 17:00
Thursday	09:00 - 17:00
Friday	09:00 - 17:00