• Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Menu
  • Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Search
Close
  • Home
  • 2019
  • June
  • 27
  • FreshOnions TorScraper – TOR Hidden Service Crawler

FreshOnions TorScraper – TOR Hidden Service Crawler

June 27, 2019 Comments Off on FreshOnions TorScraper – TOR Hidden Service Crawler
fresh onions hidden wiki list hidden wiki scraper latest onions list of onions new onion services osint onion links tor scraper

This is a copy of the source for the http://zlal32teyptf4tvi.onion hidden service, which implements a tor hidden service crawler / spider and web site.

Features

  • Crawls the darknet looking for new hidden service
  • Find hidden services from a number of clearnet sources
  • Optional fulltext elasticsearch support
  • Marks clone sites of the /r/darknet superlist
  • Finds SSH fingerprints across hidden services
  • Finds email addresses across hidden services
  • Finds bitcoin addresses across hidden services
  • Shows incoming / outgoing links to onion domains
  • Up-to-date alive / dead hidden service status
  • Portscanner
  • Search for “interesting” URL paths, useful 404 detection
  • Automatic language detection
  • Fuzzy clone detection (requires elasticsearch, more advanced than superlist clone detection)
  • Doesn’t fuck around in general.

Dependencies

  • python
  • tor
Also read: ToRat - Remote Administation Tool using Tor as a Transport Mechanism

pip install:

pip install -r requirements.txt

Install

Create mysql db from schema.sql

Edit etc/database for your database setup

Edit etc/proxy for your TOR setup

script/push.sh someoniondirectory.onion 
script/push.sh anotheroniondirectory.onion

Edit etc/uwsgi_only and set BASEDIR to wherever torscraper is installed (i.e. /home/user/torscraper)

Run:

init/scraper_service.sh # to start crawling
init/isup_service.sh # to keep site status up to date

Optional ElasticSearch Fulltext Search

The torscraper comes with optional elasticsearch capability (enabled by default). Edit etc/elasticsearch and set vars or set ELASTICSEARCH_ENABLED=false to disable. Run scripts/elasticsearch_migrate.sh to perform the initial setup after configuration.

if elasticsearch is disabled there will be no fulltext search, however crawling and discovering new sites will still work.

Official TOR Browser App released for Android

cronjobs

# harvest onions from various sources
1 18 * * * /home/scraper/torscraper/scripts/harvest.sh

# get ssh fingerprints for new sites
1 4,16 * * * /home/scraper/torscraper/scripts/update_fingerprints.sh

# mark sites as genuine / fake from the /r/darknetmarkets superlist    
1 9 * * 1 /home/scraper/torscraper/scripts/get_valid.sh

# scrape pastebin for onions (needs paid account / IP whitelisting)                 
*/5 * * * * /home/scraper/torscraper/scripts/pastebin.sh

# portscan new onions               
1 */6 * * * /home/scraper/torscraper/scripts/portscan_up.sh

# scrape stronghold paste
32 */2 * * * /home/scraper/torscraper/scripts/stronghold_paste_rip.sh

# detect clones
16 3 * * * /home/scraper/torscraper/scripts/detect_clones.sh

Infrastructure

Fresh Onions runs on two servers, a frontend host running the database and hidden service web site, and a backend host running the crawler. Probably most interesting to the reader is the setup for the backend. TOR as a client is COMPLETELY SINGLETHREADED. I know! It’s 2017, and along with a complete lack of flying cars, TOR runs in a single thread. What this means is that if you try to run a crawler on a single TOR instance you will quickly find you are maxing out your CPU at 100%.

The solution to this problem is running multiple TOR instances and connecting to them through some kind of frontend that will round-robin your requests. The Fresh Onions crawler runs eight Tor instances.

Debian (and ubuntu) comes with a useful program “tor-instance-create” for quickly creating multiple instances of TOR. I used Squid as my frontend proxy, but unfortunately it can’t connect to SOCKS directly, so I used “privoxy” as an intermediate proxy. You will need one privoxy instance for every TOR instance. There is a script in “scripts/create_privoxy.sh” to help with creating privoxy instances on debian systems. It also helps to replace /etc/privoxy/default.filter with an empty file, to reduce CPU load by removing unnecessary regexes.

Additionally, this resource https://www.howtoforge.com/ultimate-security-proxy-with-tor might be useful in setting up squid. If all you are doing is crawling and don’t care about anonymity, I also recommend running TOR in tor2web mode (required recompilation) for increased speed

CIA sets up Untraceable Dark Web Website on the Anonymous, Encrypted Tor Network

Screenshot on Freshonions

 

 

Visit their site on TOR here or view source below on github

 

Post navigation

Former AlphaBay Moderator Pennismith Alias Botah under Rico Charges
TorBot – OSINT tool for Deep and Dark Web

Related Articles

Smogcloud - Find Cloud Assets That No One Wants Exposed

Smogcloud – Find Cloud Assets That No One Wants Exposed

- Hack Tools
March 26, 2021
Gitrecon - OSINT Tool To Get Information From A Github Profile And Find GitHub User'S Email Addresses Leaked On Commits

Gitrecon – OSINT Tool To Get Information From A Github Profile And Find GitHub User’S Email Addresses Leaked On Commits

- Hack Tools
March 25, 2021
mediator

mediator: end-to-end encrypted, extensible reverse shell

- Hack Tools
March 25, 2021
hacker gadgets
hacker phone covers

Recent Posts

Microsoft Exchange Server Remote Code Execution Vulnerability Alert

Microsoft Exchange Server Remote Code Execution Vulnerability Alert

April 15, 2021
Chrome vulnerability

Chrome 89.0.4389.128 releases: fix two critical security vulnerability

April 14, 2021
September Patch Tuesday

Microsoft Patch Tuesday 2021 April: fix 114 security vulnerabilities

April 14, 2021
COVID Unemployment Scammer Sentenced to Prison

COVID Unemployment Scammer Sentenced to Prison

April 13, 2021
Researcher Publishes Chrome 0-Day Remote Code Execution Vulnerability

Researcher Publishes Chrome 0-Day Remote Code Execution Vulnerability

April 13, 2021
Former Police Employee Denies Selling Guns on the Darkweb

Former Police Employee Denies Selling Guns on the Darkweb

April 12, 2021

Social Media Hacking

SocialPath – Track users across Social Media Platforms

SocialPath – Track users across Social Media Platforms

- Social Media Hacking
October 16, 2019October 16, 2019

SocialPath is a django application for gathering social media intelligence on specific username. It checks for Twitter, Instagram, Facebook, Reddit...

SocialScan – Check Email Address and Username Availability on Online Platforms

SocialScan – Check Email Address and Username Availability on Online Platforms

June 17, 2019
Shellphish – Phishing Tool For 18 Social Media Apps

Shellphish – Phishing Tool For 18 Social Media Apps

June 10, 2019July 27, 2019
WhatsApp Hacking using QRLJacking

WhatsApp Hacking using QRLJacking

May 2, 2019May 19, 2019
How to Hack any Facebook Account with Z-Shadow

How to Hack any Facebook Account with Z-Shadow

April 26, 2019June 29, 2020
hacker buffs
ABOUT US

Haxf4rall is a collective, a good starting point and provides a variety of quality material for cyber security professionals.

Our primary focus revolves around the latest tools released in the Infosec community and provide a platform for developers to showcase their skillset and current projects.

COMPANY
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Menu
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Live Chat
RESOURCES
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Menu
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Get Started
TOOLBOX
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Menu
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Tools Directory

2014 – 2020 | Haxf4rall.com               Stay Connected:

Facebook
Twitter
Google-plus
Wordpress
Please wait...

Join Our Community

Subscribe now and get your free HACKERS HANDBOOK

Don't Worry ! You will not be spammed
SIGN UP FOR NEWSLETTER NOW