• Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Menu
  • Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Search
Close
  • Home
  • 2019
  • August
  • 11
  • AIL framework – Framework for Analysis of Information Leaks

AIL framework – Framework for Analysis of Information Leaks

August 11, 2019 Comments Off on AIL framework – Framework for Analysis of Information Leaks
ail analysis ail framework leak analysis

AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information (e.g. data leak prevention).

Dashboard

Finding webshells with AIL

Features

  • Modular architecture to handle streams of unstructured or structured information
  • Default support for external ZMQ feeds, such as provided by CIRCL or other providers
  • Multiple feed support
  • Each module can process and reprocess the information already processed by AIL
  • Detecting and extracting URLs including their geographical location (e.g. IP address location)
  • Extracting and validating potential leak of credit cards numbers, credentials, …
  • Extracting and validating email addresses leaked including DNS MX validation
  • Module for extracting Tor .onion addresses (to be further processed for analysis)
  • Keep tracks of duplicates (and diffing between each duplicate found)
  • Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
  • A full-text indexer module to index unstructured information
  • Statistics on modules and web
  • Real-time modules manager in terminal
  • Global sentiment analysis for each providers based on nltk vader module
  • Terms, Set of terms and Regex tracking and occurrence
  • Many more modules for extracting phone numbers, credentials and others
  • Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
  • Detect and decode encoded file (Base64, hex encoded or your own decoding scheme) and store files
  • Detect Amazon AWS and Google API keys
  • Detect Bitcoin address and Bitcoin private keys
  • Detect private keys, certificate, keys (including SSH, OpenVPN)
  • Detect IBAN bank accounts
  • Tagging system with MISP Galaxy and MISP Taxonomies tags
  • UI paste submission
  • Create events on MISP and cases on The Hive
  • Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags
  • Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
  • Graph relationships between decoded file (hashes), similar PGP UIDs and addresses of cryptocurrencies
  • Tor hidden services crawler to crawl and parse output
  • Tor onion availability is monitored to detect up and down of hidden services
  • Browser hidden services are screenshot and integrated in the analysed output including a blurring screenshot interface (to avoid “burning the eyes” of the security analysis with specific content)
  • Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
  • Generic web crawler to trigger crawling on demand or at regular interval URL or Tor hidden services

Installation

Type these command lines for a fully automated installation and start AIL framework:

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh

cd ~/AIL-framework/
cd bin/
./LAUNCH.sh -l

The default installing_deps.sh is for Debian and Ubuntu based distributions.

There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.

Requirement:

  • Python 3.5+

Installation Notes

In order to use AIL combined with ZFS or unprivileged LXC it’s necessary to disable Direct I/O in $AIL_HOME/configs/6382.conf by changing the value of the directive use_direct_io_for_flush_and_compaction to false.

Starting AIL

cd bin/
./LAUNCH -l

Eventually you can browse the status of the AIL framework website at the following URL:

https://localhost:7000/

The default credentials for the web interface are located in DEFAULT_PASSWORD. This file is removed when you change your password.

Training

CIRCL organises training on how to use or extend the AIL framework. AIL training materials are available at https://www.circl.lu/services/ail-training-materials/.

HOWTO

HOWTO are available in HOWTO.md

Privacy and GDPR

AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.

Research using AIL

If you write academic paper, relying or using AIL, it can be cited with the following BibTeX:

@inproceedings{mokaddem2018ail,
  title={AIL-The design and implementation of an Analysis Information Leak framework},
  author={Mokaddem, Sami and Wagener, G{\'e}rard and Dulaunoy, Alexandre},
  booktitle={2018 IEEE International Conference on Big Data (Big Data)},
  pages={5049--5057},
  year={2018},
  organization={IEEE}
}

Screenshots

Tor hidden service crawler

Tor hidden service

Trending charts

Trending-Web Trending-Modules

Extracted encoded files from pastes

Extracted files from pastes Relationships between extracted files from encoded file in unstructured data

Browsing

Browse-Pastes

Tagging system

Tags

MISP and The Hive, automatic events and alerts creation

paste_submit

Paste submission

paste_submit

Sentiment analysis

Sentiment

Terms manager and occurence

Term-Manager

Top terms

Term-Top Term-Plot

AIL framework screencast

Command line module manager

Module-Manager

You can read more and download this tool over here: https://github.com/CIRCL/AIL

Post navigation

Attack Surface Mapper – Tool to Automate Reconnaissance
Theo – Ethereum Recon and Exploitation Tool

Related Articles

PatrOwl – Smart and Scalable Security Operations Orchestration Platform

- Threat Intelligence
October 7, 2019

Sampler – A Tool For Shell Commands Execution, Visualization And Alerting

- Threat Intelligence
August 17, 2019

ThreatHunting – A Splunk App Mapped To MITRE ATT&CK To Guide Your Threat Hunts

- Threat Intelligence
August 14, 2019
hacker gadgets
hacker phone covers

Recent Posts

DefaScan: Defacement Scan and Alert

DefaScan: Defacement Scan and Alert

February 2, 2023
curio: finds risks and vulnerabilities in your code

curio: finds risks and vulnerabilities in your code

February 1, 2023
Monomorph - MD5-Monomorphic Shellcode Packer - All Payloads Have The Same MD5 Hash

Monomorph – MD5-Monomorphic Shellcode Packer – All Payloads Have The Same MD5 Hash

February 1, 2023
A Guide to Crypto Self-Custody

A Guide to Crypto Self-Custody

February 1, 2023
CVE-2023-23924: Critical-Severity RCE Flaw Found in Popular Dompdf Library

CVE-2023-23924: Critical-Severity RCE Flaw Found in Popular Dompdf Library

February 1, 2023
wa-tunnel: TCP Tunneling through Whatsapp

wa-tunnel: TCP Tunneling through Whatsapp

February 1, 2023

Social Media Hacking

SocialPath – Track users across Social Media Platforms

SocialPath – Track users across Social Media Platforms

- Social Media Hacking
October 16, 2019October 16, 2019

SocialPath is a django application for gathering social media intelligence on specific username. It checks for Twitter, Instagram, Facebook, Reddit...

SocialScan – Check Email Address and Username Availability on Online Platforms

SocialScan – Check Email Address and Username Availability on Online Platforms

June 17, 2019
Shellphish – Phishing Tool For 18 Social Media Apps

Shellphish – Phishing Tool For 18 Social Media Apps

June 10, 2019July 27, 2019
WhatsApp Hacking using QRLJacking

WhatsApp Hacking using QRLJacking

May 2, 2019May 19, 2019
How to Hack any Facebook Account with Z-Shadow

How to Hack any Facebook Account with Z-Shadow

April 26, 2019June 29, 2020
hacker buffs

About Us

Haxf4rall is a collective, a good starting point and provides a variety of quality material for cyber security professionals.

Join Our Community!

Please wait...
Get the latest News and Hacking Tools delivered to your inbox.
Don't Worry ! You will not be spammed

Active Members

Submit a Tool

Hackers Handbook 2018


Grab your copy here

ABOUT US

Haxf4rall is a collective, a good starting point and provides a variety of quality material for cyber security professionals.

Our primary focus revolves around the latest tools released in the Infosec community and provide a platform for developers to showcase their skillset and current projects.

COMPANY
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Menu
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Live Chat
RESOURCES
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Menu
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Get Started
TOOLBOX
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Menu
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Tools Directory

2014 – 2020 | Haxf4rall.com               Stay Connected:

Facebook Twitter Google-plus Wordpress
Please wait...

Join Our Community

Subscribe now and get your free HACKERS HANDBOOK

Don't Worry ! You will not be spammed
SIGN UP FOR NEWSLETTER NOW