• Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Menu
  • Home
  • Become a Hacker
    • Get Started
    • Hacker Mindset
    • Roadmap
    • Simple Setup – Hacker 101
    • Types of Hackers
    • Recommended Courses
  • Boot People Offline
  • Courses
    • All Hacking Courses
    • Cyber Security School
  • CTF
    • Beginners to Advanced Guide
    • Create your own CTF box
    • Field and Resources Guide
    • Platforms & Wargames
    • Tools Used for Solving CTF
    • Writeups
  • Dark Web
    • Beginners Guide
    • Darknet Markets
    • Darkweb 101 (Anonymity Guide)
    • Dark Web OSINT Tools
    • Hacking Forums
    • Latest News
    • Onion Links
  • Hacker Gadgets
  • Hacking Books
  • Tools Directory
Search
Close
  • Home
  • 2019
  • January
  • 12
  • Realtime Scrapper: Scrap all pasties,github,reddit..etc in real time

Realtime Scrapper: Scrap all pasties,github,reddit..etc in real time

January 12, 2019July 27, 2019 Comments Off on Realtime Scrapper: Scrap all pasties,github,reddit..etc in real time
realtime scrapper scrap search terms

RTS (Realtime scrapper) is a tool developed to scrap all pasties,github,reddit..etc in real time to identify the occurrence of search terms configured.

Upon the match, an email will be triggered. Thus allowing the company to react in case of leakage of code, any hacks tweeted..etc.. and harden themselves against an attack before it goes viral.

The same tool in malicious user hands can be used offensively to get the update on any latest hacks, code leakage etc..

List of sites which will be monitored are:

  • Non-Pastie Sites
    • Twitter
    • Reddit
    • Github
  • Pastie Sites
    • Pastebin.com
    • Codepad.org
    • Dumpz.org
    • Snipplr.com
    • Paste.org.ru
    • Gist.github.com
    • Pastebin.ca
    • Kpaste.net
    • Slexy.org
    • Ideone.com
    • Pastebin.fr

Architecture 

Realtime scrapper

Configuration

Before using this tool is is neccessary to understand the properties file present in scrapper_config directory.

      • consumer.properties: Holds all the neccessary config data needed for consumer (Refer apache Kafka guide for more information). The values present here are default options and does nto require any changes
      • producer.properties: Holds all the neccessary config data needed for Producer (Refer apache Kafka guide for more information).The values present here are default options and does nto require any changes
      • email.properties: Configure SMTP server with email id’s.
      • scanner-configuration.properties: This is the core configuration file. Update all the config for enabling search on twitter/github(To get tokens and key refer respective sites). For pastie sites and Reddit, there is no need for any changes in config.

    Note: However in all cases make sure to change “searchterms” to values of our choice to search. If there are multiple search terms then add them separate by comma as shown with example terms in the config file.

    Understanding more about scanner-configuration.properties file.

      • For any pastie site configuration is as below:

        • scrapper.(pastie name).profile=(Pastie profile name)
        • scrapper.(pastie name).homeurl=(URL from where pastie ids a extracted)
        • scrapper.(pastie name).regex=(Regex to fetch pastie ids)
        • scrapper.(pastie name).downloadurl= (URL to get information about each apstie)
        • scrapper.(pastie name).searchterms=(Mention terms to be searched seperated by comma)
        • scrapper.(pastie name).timetosleep=(Time for which pastie thread will sleep before fetching pastie ids again)
      • For github search configuration is as below:

        • scrapper.github.profile=Github
        • scrapper.github.baseurl=https://api.github.com/search/code?q={searchTerm}&sort=indexed&order=asc
        • scrapper.github.access_token=(Get your own github access token)
        • scrapper.github.searchterms=(Mention terms to be searched seperated by comma)
        • scrapper.github.timetosleep=(Time for which github thred should sleep before searching again)
      • For reditt search configuration is as below:

        • scrapper.reddit.profile=Reddit
        • scrapper.reddit.baseurl=https://www.reddit.com/search.json?q={searchterm}
        • scrapper.reddit.searchterms=(Mention terms to be searched seperated by comma)
        • scrapper.reddit.timetosleep=(Time for which github thred should sleep before searching again)
      • For Twitter search configuration is as below:

        • scrapper.twitter.apikey=test
        • scrapper.twitter.profile=Twitter
        • scrapper.twitter.searchterms=(Mention terms to be searched seperated by comma)
        • scrapper.twitter.consumerKey=(Get your own consumer key)
        • scrapper.twitter.consumerSecret=(Get your own consumerSecret)
        • scrapper.twitter.accessToken=(Get your own accessToken)
        • scrapper.twitter.accessTokenSecret=(Get your own accessTokenSecret)

Download 

Post navigation

H8Mail – Email OSINT and password breach hunting.
Arch Linux 2019.01.01 released: Linux Kernel 4.20.0

Related Articles

Boomerang: expose multiple internal servers to web/cloud

Boomerang: expose multiple internal servers to web/cloud

- Hack Tools
March 8, 2021
Procrustes - A Bash Script That Automates The Exfiltration Of Data Over Dns In Case We Have A Blind Command Execution On A Server Where All Outbound Connections Except DNS Are Blocked

Procrustes – A Bash Script That Automates The Exfiltration Of Data Over Dns In Case We Have A Blind Command Execution On A Server Where All Outbound Connections Except DNS Are Blocked

- Hack Tools
March 7, 2021
packetStrider - A Network Packet Forensics Tool For SSH

packetStrider – A Network Packet Forensics Tool For SSH

- Hack Tools
March 7, 2021
hacker gadgets
hacker phone covers

Recent Posts

Boomerang: expose multiple internal servers to web/cloud

Boomerang: expose multiple internal servers to web/cloud

March 8, 2021
Procrustes - A Bash Script That Automates The Exfiltration Of Data Over Dns In Case We Have A Blind Command Execution On A Server Where All Outbound Connections Except DNS Are Blocked

Procrustes – A Bash Script That Automates The Exfiltration Of Data Over Dns In Case We Have A Blind Command Execution On A Server Where All Outbound Connections Except DNS Are Blocked

March 7, 2021
packetStrider - A Network Packet Forensics Tool For SSH

packetStrider – A Network Packet Forensics Tool For SSH

March 7, 2021
OWASP Mutillidae II

OWASP Mutillidae II 2.8.24 releases: OWASP Mutillidae II Web Pen-Test Practice Application

March 7, 2021
kics: Find security vulnerabilities, compliance issues, and infrastructure misconfigurations

kics: Find security vulnerabilities, compliance issues, and infrastructure misconfigurations

March 7, 2021
Teen Faces Trafficking Charges After Ecstasy Interception

Teen Faces Trafficking Charges After Ecstasy Interception

March 7, 2021

Social Media Hacking

SocialPath – Track users across Social Media Platforms

SocialPath – Track users across Social Media Platforms

- Social Media Hacking
October 16, 2019October 16, 2019

SocialPath is a django application for gathering social media intelligence on specific username. It checks for Twitter, Instagram, Facebook, Reddit...

SocialScan – Check Email Address and Username Availability on Online Platforms

SocialScan – Check Email Address and Username Availability on Online Platforms

June 17, 2019
Shellphish – Phishing Tool For 18 Social Media Apps

Shellphish – Phishing Tool For 18 Social Media Apps

June 10, 2019July 27, 2019
WhatsApp Hacking using QRLJacking

WhatsApp Hacking using QRLJacking

May 2, 2019May 19, 2019
How to Hack any Facebook Account with Z-Shadow

How to Hack any Facebook Account with Z-Shadow

April 26, 2019June 29, 2020
hacker buffs
ABOUT US

Haxf4rall is a collective, a good starting point and provides a variety of quality material for cyber security professionals.

Our primary focus revolves around the latest tools released in the Infosec community and provide a platform for developers to showcase their skillset and current projects.

COMPANY
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Menu
  • Contact Us
  • Disclaimer
  • Hacker Gadgets
  • LANC Remastered
  • PCPS IP Puller
  • Privacy Policy
  • Sitemap
  • Submit your Tool
Live Chat
RESOURCES
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Menu
  • Attack Process
  • Become a Hacker
  • Career Pathways
  • Dark Web
  • Hacking Books
  • Practice Your Skills
  • Recommended Courses
  • Simple Setup – Hacker 101
Get Started
TOOLBOX
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Menu
  • Anonymity
  • Bruteforce
  • DoS – Denial of Service
  • Information Gathering
  • Phishing
  • SQL Injection
  • Vulnerability Scanners
  • Wifi Hacking
Tools Directory

2014 – 2020 | Haxf4rall.com               Stay Connected:

Facebook
Twitter
Google-plus
Wordpress
Please wait...

Join Our Community

Subscribe now and get your free HACKERS HANDBOOK

Don't Worry ! You will not be spammed
SIGN UP FOR NEWSLETTER NOW