Eko Samuel working at his software development workstation

AI (Machine Learning) Engineer

I’m Eko Samuel, a software engineer with over ten years of experience building full-stack applications, backend systems, and cloud infrastructure. My work combines AI and machine learning with practical engineering to solve real-world problems.

Connect with Eko Samuel

All AI Projects

Case study / 05

C++ Information Retrieval Engine with Text Parsing, Tokenization, and Inverted Indexing

Builds a document-processing and retrieval pipeline in C++ that parses TREC records, normalizes tokens, removes stopwords, and applies stemming. It creates document and term dictionaries alongside forward and inverted indexes for term lookup.

  • C++17
  • CMake
  • Tokenization
  • Porter stemming
  • Inverted indexes
Project evidence / Visual exploration01 / 02

01 / Context & purpose

The project in detail

Search systems need a reliable way to transform raw documents into structures that make term lookup possible. This C++ project implements that foundation by parsing TREC documents and building dictionaries, forward indexes, and inverted postings.

The pipeline normalizes tokens, removes stopwords, and applies stemming before writing indexed output. The committed output files make the mapping between terms, documents, and frequencies inspectable without relying on a black-box retrieval service.

02 / Engineering perspective

How the project works

  1. Read document records and separate content from the surrounding corpus structure.

  2. Apply tokenization, lowercase normalization, stopword filtering, and stemming to construct consistent searchable terms.

  3. Persist dictionaries and indexes, then use lookup components to retrieve postings for indexed terms.

03 / Engineering perspective

Results & Outcomes

  1. Produces persisted parser output, document and term dictionaries, forward indexes, inverted postings, and lookup offsets.

  2. The committed output demonstrates term-to-document postings with term frequencies; no benchmark latency or retrieval-accuracy claim is supplied.

04 / Engineering perspective

Why this project matters

  1. Builds the lexical retrieval foundations used by many larger search and question-answering systems.

  2. Makes preprocessing decisions explicit, helping explain why a query term does or does not match a document.

  3. Provides reusable systems-programming components for document ingestion and indexing.

05 / Potential next steps

Advancing AI & Future Directions

Research opportunities and proposed extensions, rather than completed achievements.

  1. Add BM25 ranking and evaluate retrieval quality with a relevance-judged benchmark.

  2. Combine lexical retrieval with dense embeddings to study hybrid search for retrieval-augmented generation.

  3. Measure index size, build time, query latency, and result relevance as the corpus grows.

Explore the next projectMachine Learning-Based Email Threat Detection and Classification System | 2025