Research Service centralizes provider-level communication across ONTIC and acts as the FedRAMP boundary for research integrations.

Research Service is a global level microservice that communicates to all research providers on behalf of ONTIC.

Pre-requisite

• If you want to know more about ONTIC infrastructure, see this first: Ontic Architecture [ Infra ]

Intro

• The basic ask was to make ONTIC Research FedRAMP compliant. As a security company, achieving FedRAMP compliance was one step forward for ONTIC.
• A simple way to think about it:

If your company provides a cloud-based SaaS product or service to the US government,
you must go through FedRAMP authorization.

This means your specific implementation on AWS needs to be assessed and approved
by the government.

• Since we were adding a new client DEA, which is a US government client, we had to make sure our app is FedRAMP compliant.
• AWS as a cloud provider is already FedRAMP compliant, but that is not enough.
• How ONTIC accesses it, manages data, stores controls, and handles access patterns also has to be compliant.
• Reference: AWS FedRAMP Overview

Fun Fact: The Problem

• If you are calling a research provider's API or integrating with an external research service, then that research provider or external service also has to be FedRAMP compliant.

If your system handles government data and sends it to a non-FedRAMP provider, it could be a compliance issue.

• ONTIC is basically a collection of several research providers.
• Making every provider FedRAMP compliant is not feasible.
• It is also expected that ONTIC will keep adding more research providers, so we needed a middle ground.

Solution

• The solution leadership came up with was to extract a separate research-service.
• This service handles all provider-level communication on behalf of ONTIC and becomes the thing we make FedRAMP compliant.
• The practical benefit, apart from compliance, is that all provider-level communication is now centralized in one microservice.
• That also makes things like IP whitelisting and webhook handling much easier to manage across environments.

We as developers or DevOps engineers cannot directly access the DEA environment, which is gc1np I guess. We usually have to ask someone with access to check logs or environment-specific state for us.

Tech

Enough product context. Let’s talk about the system.

HLD

HLD Source Link..

• Research Service is a global microservice.
• It is a separate microservice with its own Mongo in a separate environment.
• Global here means different ONTIC infrastructures communicate with the same Research Service on the Global Data Prod infrastructure.
• Yes, that means research data from multiple infrastructures can land in the same Research Service Mongo.
• Let’s look at how that works in practice.

Workflow [Sync Searches]

• User A in Prod wants information about Elon Musk.
• Assume the user has purchased an offering like Social Net, which can do social profile searches based on alias, phone, email, and similar inputs.
• User A hits search.
• Our backend receives the request and checks the property NEW_RESEARCH_ENABLED.
• If enabled, after request adaptation, we send the request over HTTP to Research Service.
• From there, Research Service is responsible for talking to Social Net, executing the search, and sending the response back to the originating environment.

All provider-level communications are abstracted out to Research Service. To achieve FedRAMP compliance, we just have to make Research Service compliant.

Since we do not have a separate global data environment for each ONTIC infrastructure, all research load lands on this microservice. Right now that is manageable because we do not have enough clients for scale to become a major issue.

• The sync flow is straightforward because the response comes back immediately.
• But many research providers are asynchronous.
• In those cases, keeping a persistent socket open would be a poor design, so we handle them differently.

Async Researches

• A user initiates a search that takes time to complete.
• We still check whether the new research service path is enabled.
• If yes, we call Research Service and pass performAsync=true.
• Research Service receives the request and checks the performAsync flag.
• If the flag is not enabled, it falls back to the sync flow.
• If the flag is enabled:
  • It publishes an event to Kafka on the TASK_CREATED topic.
  • Research Consumer consumes that message from Kafka.
  • Research Consumer calls the provider and fetches the result.
  • The shared provider-calling logic lives in research-core, and research-consumer depends on it, so the provider interaction code is reused.
  • Once the response is fetched, Research Service sends a push message to SQS or whatever push config is configured on Global Data for that environment.
  • For Prod, we have an SQS push config in Global Data, so the message is pushed into Prod's SQS.
• Back in Prod, we have a consumer polling that queue.
• The message is processed and then a pusher event is sent to the UI.
• That pusher is important because it tells the UI that the response is ready and can now be fetched.

Hurray, we got our response.

LLD

The purpose here is to understand the higher-level ONTIC system design, not to go deep into code-level detail. You learn design principles more by reading and writing code than by drawing boxes.

• Honestly, I think LLD is often very organization-specific.
• We can talk about higher-level system behavior here, but for real LLD I would still expect the code to be the source of truth.
• If needed, this can be expanded later with code-path level notes.