AIL framework – Framework for Analysis of Information Leaks

August 11, 2019 Comments Off

AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information (e.g. data leak prevention).

Features

Modular architecture to handle streams of unstructured or structured information
Default support for external ZMQ feeds, such as provided by CIRCL or other providers
Multiple feed support
Each module can process and reprocess the information already processed by AIL
Detecting and extracting URLs including their geographical location (e.g. IP address location)
Extracting and validating potential leak of credit cards numbers, credentials, …
Extracting and validating email addresses leaked including DNS MX validation
Module for extracting Tor .onion addresses (to be further processed for analysis)
Keep tracks of duplicates (and diffing between each duplicate found)
Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
A full-text indexer module to index unstructured information
Statistics on modules and web
Real-time modules manager in terminal
Global sentiment analysis for each providers based on nltk vader module
Terms, Set of terms and Regex tracking and occurrence
Many more modules for extracting phone numbers, credentials and others
Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
Detect and decode encoded file (Base64, hex encoded or your own decoding scheme) and store files
Detect Amazon AWS and Google API keys
Detect Bitcoin address and Bitcoin private keys
Detect private keys, certificate, keys (including SSH, OpenVPN)
Detect IBAN bank accounts
Tagging system with MISP Galaxy and MISP Taxonomies tags
UI paste submission
Create events on MISP and cases on The Hive
Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags
Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
Graph relationships between decoded file (hashes), similar PGP UIDs and addresses of cryptocurrencies
Tor hidden services crawler to crawl and parse output
Tor onion availability is monitored to detect up and down of hidden services
Browser hidden services are screenshot and integrated in the analysed output including a blurring screenshot interface (to avoid “burning the eyes” of the security analysis with specific content)
Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
Generic web crawler to trigger crawling on demand or at regular interval URL or Tor hidden services

Installation

Type these command lines for a fully automated installation and start AIL framework:

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh

cd ~/AIL-framework/
cd bin/
./LAUNCH.sh -l

The default installing_deps.sh is for Debian and Ubuntu based distributions.

There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.

Requirement:

Python 3.5+

Installation Notes

In order to use AIL combined with ZFS or unprivileged LXC it’s necessary to disable Direct I/O in $AIL_HOME/configs/6382.conf by changing the value of the directive use_direct_io_for_flush_and_compaction to false.

Starting AIL

cd bin/
./LAUNCH -l

Eventually you can browse the status of the AIL framework website at the following URL:

https://localhost:7000/

The default credentials for the web interface are located in DEFAULT_PASSWORD. This file is removed when you change your password.

Training

CIRCL organises training on how to use or extend the AIL framework. AIL training materials are available at https://www.circl.lu/services/ail-training-materials/.

HOWTO

HOWTO are available in HOWTO.md

Privacy and GDPR

AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.

Research using AIL

If you write academic paper, relying or using AIL, it can be cited with the following BibTeX:

@inproceedings{mokaddem2018ail,
  title={AIL-The design and implementation of an Analysis Information Leak framework},
  author={Mokaddem, Sami and Wagener, G{\'e}rard and Dulaunoy, Alexandre},
  booktitle={2018 IEEE International Conference on Big Data (Big Data)},
  pages={5049--5057},
  year={2018},
  organization={IEEE}
}