grisuno/estorides: open-source intelligence (OSINT) aggregator and correlation engine

E.S.T.E.R.O.I.D.E.S. (acronym)

• E — Entities (identification of entities, aliases, emails, IPs).
• S — Signals (capture of digital footprints and metadata).
• T — Targeted (focused on specific objectives).
• E — Extraction (automated extraction from web sources).
• R — Reconnaissance (recon and footprinting).
• O — Open-source (the nature of the OSINT engine).
• I — Intelligence (processing and correlation of data).
• D — Data (massive ingestion of unstructured records).
• E — Engine (the central engine that orchestrates queries).
• S — Scraper (automated and persistent collection).

Open-source intelligence (OSINT) aggregator and correlation engine inspired by Palantir, Bellingcat, Maltego, and Citizen Lab workflows. A pure open-source re-imagining of the original fucklantir / osint_palantir toolchain, with a much bigger source catalogue, a proper knowledge graph, structured parsers, and a multi-backend LLM analyst.

No payloads. No active scanning. Just 99+ free public OSINT sources, fanned out in parallel, fused into a single intelligence picture.

                       +--------------------------+
   query "example.com" |  Estorides Orchestrator  |   -> STIX 2.1 bundle
---------------------> |  - async fanout          |   -> MISP event JSON
                       |  - 99+ free sources      |   -> GraphML for Gephi
                       |  - structured parsers    |   -> JSONL for training
                       |  - entity resolution     |
                       |  - knowledge graph       |
                       |  - ontology engine       |   <- OFAC SDN cross-check
                       |  - MITRE ATT&CK mapper   |   <- technique auto-tagging
                       |  - SSRF guard            |   <- blocklist at egress
                       |  - audit log + RL        |   <- per-IP trail
                       |  - multi-LLM analyst     |   <- BLUF / tactical / system
                       +--------------------------+
                                 |
                                 v
                          Web UI: map / graph / timeline / results

Architecture highlights (state-level)

Estorides is structured around small, single-responsibility registries so adding a new source, backend, inferer, or feed never requires touching the central orchestrator. The five plug-in surfaces are:

What you get that the original does not

What v1.1 adds on top

v1.1 architecture

                       +--------------------------+
   query "example.com" |  Estorides Orchestrator  |   -> STIX 2.1 / MISP / GraphML / JSON
---------------------> |  + async fanout          |
                       |  + 99 free sources       |
                       |  + 7 Osiris-style probes |
                       |  + SSRF guard + audit     |
                       |  + ontology engine       |
                       |  + MITRE ATT&CK mapper   |
                       |  + multi-LLM analyst     |
                       |  + cross-feed resolver   |   <- Wikidata SPARQL + OFAC + IP-API + NVD
                       |  + fuzzy entity cluster  |   <- difflib SequenceMatcher
                       +-----------+--------------+
                                   |
                  +----------------+-----------------+--------------------+
                  v                                  v                    v
        +------------------+              +------------------+    +------------------+
        | Kùzu graph DB    |              | SQLite case store|    | In-memory NX     |
        | (Cypher queries) |              | (FK observations)|    | (per-run working)|
        | 99 node labels   |              | search by entity |    | per-run edges    |
        | 9 REL types      |              | search by query  |    |                  |
        +------------------+              +------------------+    +------------------+
                  ^                                  ^
                  +---------/api/intel/resolve-------+
                  +---------/api/cases/...-----------+

v1.1 API additions

v1.1 install

pip install -r requirements.txt

The only new required dep is kuzu>=0.11. The orchestrator falls back to in-memory NetworkX if Kùzu is not importable, but a persistent cross-run graph only happens with Kùzu present.

Quickstart

1. Install (no extra packages needed; the project uses Flask + NetworkX + requests)

cd estorides
python3 -m pip install flask networkx requests pyyaml

Optional, for a real LLM:

# pick one
ollama serve && ollama pull llama3.1:8b
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export OPENROUTER_API_KEY=sk-or-...

2. CLI

# 99 sources, 12 categories
python3 estorides_cli.py status

# run a query (free sources only)
python3 estorides_cli.py run 8.8.8.8

# enable sources that need an API key
python3 estorides_cli.py run [email protected] --include-paid

# only a subset of sources
python3 estorides_cli.py run example.com \
    --only-sources crt_sh_certificates,shodan_internetdb,ipapi_free

# export the latest run as STIX 2.1 or MISP
python3 estorides_cli.py stix --out my_bundle.json
python3 estorides_cli.py misp --out my_event.json

3. Web UI

python3 estorides_cli.py serve --port 5050
# open http://127.0.0.1:5050

UI features:

• 2D map (Leaflet) of every geolocated result
• D3.js force-directed knowledge graph (drag, zoom, hover)
• Timeline of source acquisition
• Source results panel with per-source parsed output
• Filterable entity list
• LLM analysis with backend / model badge
• One-click export: STIX 2.1, MISP, GraphML, JSON

Graph intelligence (Maltego-style)

The Graph canvas turns observations into an interactive intelligence workbench:

• Clusters — nodes are grouped into communities (translucent hulls) and coloured by cluster. Inter-cluster links are dashed/highlighted; click one to see a cross-reference tooltip explaining how two clusters relate (the bridge entities + relation).
• Click to enrich — left-click a node to resolve it (cross-feed + VirusTotal relationships) and merge the new nodes/links into both the graph and the map. Each new node is itself clickable, so exploration is recursive.
• Intelligence tiers — every node carries an auto-computed level shown as a coloured ring: data → information (≥2 corroborating sources) → intelligence (cross-cluster / resolved) → counter-intelligence (sanction / threat / VirusTotal-malicious). Override any node's level from the right-click menu or the inspector (persists in the browser).
• Transforms — right-click a node (or use the side inspector panel) for transforms grouped by tier: data → information → intelligence → counter-intelligence.

VirusTotal

VirusTotal is integrated both as a source (vt_ip, vt_domain, vt_file) and as the relationship engine behind graph expansion (resolved domains/IPs, communicating/dropped files, contacted infra). It needs a free API key; without it VirusTotal stays inactive and the rest of the platform is unaffected:

export VT_API_KEY=...   # https://www.virustotal.com/gui/my-apikey

99 sources, 12 categories

1. DNS Intelligence (9) - Google DoH, Cloudflare DoH, HackerTarget, crt.sh, Cert Spotter, RDAP, DNS Dumpster, host search
2. IP & Infrastructure (13) - ip-api, ipinfo, ipapi.co, ipwho.is, Shodan InternetDB, GreyNoise, ipwhois, Robtex, RDAP, AS lookup, AbuseIPDB, MAC OUI, RIPE Stat
3. Web Intelligence (10) - urlscan, Wayback CDX, Wayback availability, HTTP headers, whois, geoip, traceroute, nping, Microlink, Google cache
4. Social Media (13) - GitHub, Reddit, Mastodon, Keybase, HackerNews, Telegram, Pinterest, WordPress, Medium, DEV.to
5. Threat Intelligence (13) - ThreatFox, URLhaus, payloads, PhishTank, OpenPhish, OTX (+passive domain/IP, no key), MalwareBazaar, Feodo, SSLBL, Emerging Threats, blocklist.de
6. Breach Intelligence (6) - HIBP breaches, HIBP pastes, Phonebook email, Phonebook domain, DeHashed, IntelligenceX
7. Geolocation (5) - Nominatim search + reverse, OpenWeather geocoding, TimeZoneDB, Wikidata
8. Knowledge (12) - Wikipedia, summary, DuckDuckGo IA, OpenAlex, Crossref, arXiv, GitHub advisories, NVD CVE, cve.circl, ExploitDB, Reddit subreddit search
9. Wireless (5) - WiGLE, IEEE OUI, OpenSky, MarineTraffic, N2YO
10. Blockchain (5) - blockchain.info (balance + tx), Blockstream, Ethplorer, mempool.space
11. Paste & Leaks (4) - psbdmp, GitHub gist search, TGStat, LeakCheck
12. Visual (4) - ScreenshotMachine, Microlink, TinEye, EXIF

Sources are addons: one YAML file per source, organised into category subdirectories under sources/ (lazyaddons-style). The loader recurses, so add a new source by dropping sources/<NN_category>/<name>.yaml — no central registry to edit. Grouped multi-document files still load if present. Point ESTORIDES_SOURCES_DIR at another tree to use your own addon set. The schema is documented at the top of estorides_core/source_loader.py.

sources/
  01_dns/
    dns_google.yaml
    crt_sh_certificates.yaml
  02_ip_infra/
    shodan_internetdb.yaml
  ...

tools/split_sources.py migrates legacy grouped files into this layout.

Architecture

sources/                  one YAML per addon, grouped by category dir (99 addons)
estorides_core/
    config.py             every tunable (env-overridable)
    source_loader.py      registry, validation, lookup
    async_client.py       aiohttp + circuit breaker + SQLite cache
    parsers.py            50+ structured parsers (ipapi, dns_json, crtsh…)
    entity_extraction.py  regex-based entity finder with dedup
    knowledge_graph.py    NetworkX MultiDiGraph + GraphML export
    orchestrator.py       glues everything, infers higher-level relations
estorides_llm/
    manager.py            multi-backend LLM (Ollama → OpenRouter → Anthropic → OpenAI → stub)
estorides_export/
    stix.py               STIX 2.1 bundle export
    misp.py               MISP event JSON export
estorides_cli.py          argparse CLI
estorides_web.py          Flask app
templates/index.html      UI
static/{css,js}/estorides.*  UI styles + D3 controller

Tips for a real run

1. Start with the free-tier sources (default) — that is 80+ endpoints.
2. Set ESTORIDES_PARALLEL=16 for faster fanout.
3. Set ESTORIDES_TIMEOUT=20 if your network is slow.
4. Disable paid sources you don't have keys for by setting ESTORIDES_DISABLE_BACKENDS=openai,anthropic (or by leaving --include-paid off in the CLI).
5. The SQLite cache lives in data/estorides_cache.sqlite — delete it to force fresh fetches.
6. The LLM stage needs a generative model. Ollama auto-selects an installed model if ESTORIDES_OLLAMA_MODEL is missing, but an embedding-only model (e.g. *:e2b) returns no text and the run falls back to the stub. ollama pull llama3.1:8b for real analysis.

Performance knobs (bounds that keep a run from stalling)

Passive recon & operator OPSEC (bug-bounty mode)

For attack-surface scoping the two things that matter are: never let the target observe a probe attributable to your recon window, and never let a queried broker tie the lookups back to your real IP. Estorides enforces both at the engine level.

Contact classification

Every source declares how its traffic reaches the target:

An unknown/typo class is treated as active, so a passive-only run can never be silently widened. --passive-only is enforced even for an explicit --only-sources list. Sources that log your lookups also carry logs_queries: true (surfaced in status).

# scope a domain without ever touching its infrastructure
python3 estorides_cli.py discover example.com --passive-only --out-json surface.json
python3 estorides_cli.py run example.com --passive-only

Egress anonymisation

Route every outbound request through a proxy so brokers never see your real IP. SOCKS (Tor) needs aiohttp_socks; HTTP/HTTPS proxies work with stock aiohttp and a comma-separated pool rotates per request.

python3 estorides_cli.py run example.com --passive-only --tor
python3 estorides_cli.py run example.com --proxy socks5://127.0.0.1:9050
export ESTORIDES_HTTP_PROXY_POOL="http://p1:8080,http://p2:8080"

Fail-closed: if a SOCKS proxy is requested without aiohttp_socks installed, the client refuses to run rather than fall back to a deanonymising direct connection. When proxying, the SSRF guard's local DNS-resolution leg is skipped (ESTORIDES_PROXY_REMOTE_DNS=1, default) so your resolver never learns which targets you are investigating — the literal-host guard still runs and the exit node resolves the name.

Scope classification

Turn a discovered surface into in/out-of-scope flat lists you can pipe into the active phase. Out-of-scope always wins, so an excluded asset is never targeted by accident.

python3 estorides_cli.py scope \
    --assets surface.json \
    --scope program_scope.txt \
    --out scope_result.json \
    --flat-dir ./scope_out
# -> scope_out/in_scope_hosts.txt, in_scope_ips.txt, unknown.txt

Rules file grammar (one per line, # comments; a ## out-of-scope divider separates the two lists):

*.example.com            wildcard host suffix (apex + subdomains)
api.example.com          exact host
192.0.2.0/24             CIDR (IPv4 or IPv6)
re:^staging-[0-9]+\.ex   regex (prefix re:)

## out-of-scope
blog.example.com
192.0.2.200/32

Hard rules

• This is a passive intelligence tool. It does not probe, exploit, or interact with the target beyond what the public sources allow.
• All API keys stay in environment variables; they are never written to disk.
• Respect the rate limits of the upstream services. The circuit breaker will back off automatically when a host starts returning errors.
• Output is for legitimate OSINT, threat intelligence, journalism, academic research, and defensive security work.

Security & operations

Intelligence features

Ontology engine — OFAC SDN cross-check

estorides_core/ontology.py loads the OpenSanctions OFAC SDN list (CC-BY 4.0) once, indexes it by normalised name + alias, and stamps every observation with {sanctioned, hits, fields}. The LLM analyst stage then writes a "SANCTIONED — OFAC SDN match on …" line into the brief so sanctions exposure is impossible to miss in the report.

Index characteristics:

• ~7 MB, low-tens-of-thousands of entries
• 24h lazy refresh
• Single-flight: concurrent first-loads share one fetch
• Best-effort disk cache at data/ontology_sdn.json
• Stale-on-error: keeps the previous snapshot if a refresh fails

MITRE ATT&CK auto-tagging

estorides_core/mitre_attack.py maps every observation to the ATT&CK techniques it might support, by both source-keyed table (40+ techniques across the threat-intel, breach, and web sources) and keyword scan (catches malware families: mimikatz, cobalt strike, lockbit, …). Aggregated techniques are exposed at the top of the orchestrator result as result.mitre.techniques.

Real-time feeds

estorides_core/feeds.py ships three keyless feeds that the map UI can layer on top of OSINT results:

Endpoint: GET /api/feeds?bbox=min_lon,min_lat,max_lon,max_lat&no_cache=1.

LLM prompt flavours

estorides_llm/intelligence_prompts.py ships three prompt styles:

• system — the default Palantir-grade analyst with BLUF + confidence-graded findings.
• bluf — single-paragraph BLUF only, for time-critical briefs.
• tactical — adds THREAT PICTURE + COA-1/2/3 + IMMEDIATE ACTION.

Backend priority is configurable: ESTORIDES_BACKEND_PRIORITY=openai,ollama or via the LLMManager constructor.

Tests

# All tests, ~10s
python3 _validate.py

# Individual suites
python3 _test_ssrf.py        # 20 SSRF cases
python3 _test_validation.py  # 16 input-validation cases
python3 _test_feeds.py       # 3 real-time feeds
python3 _test_encryption.py  # age encryption + graceful degradation
python3 _test_routes.py      # Flask route table
python3 _multi_test.sh       # end-to-end: 5 query types through the orchestrator

The validator exits 0 only when every check passes. CI runners can grep FAIL to surface regressions.