Open Source Tool Categories
- AI Frameworks
- AI Libraries
- Application Performance Monitoring
- Caching / Reverse Proxy
- Container Security
- Continuous Delivery
- Data Pipelines
- Data Serialization
- Data Storage
- Distributed Tracing
- Intrusion Detection
- IP Scanners
- Java Build Tools
- Kubernetes Dev Tools
- Load Balancers
- Log Management
- Network Emulator
- Packet Analysis
- Packet Processors
- Packet Filter
- Pen Testing
- Service Discovery
- Service Mesh
- Stream Processing
- Subdomain Scanner Tools
- UDP File Transfer
- Vulnerability Scanning Tools
Tools A – Z
- TensorFlow: Developed by Google, TensorFow is a comprehensive deep learning open source platform.
- PyTorch: Developed by Facebook, PyTorch is a machine learning framework developed in Lua which includes modules to extend functionality.
- Scikit-learn: Python library used for data analysis which uses NumPy.
- Caffe: Deep learning framework with large community of supporters. Ideal for image classification and image segmentation.
- Torch: Scientific computing framework for machine learning developed in Lua. Good for computer vision, image classification, video, and voice.
- NumPy: Python library used in the scientific computing. Supports multi-dimensional arrays and is the foundation for other programs such as Scikit-learn.
- SciPy: Collection of libraries used in scientific computing. Works with NumPy and Matplotlib.
- Pandas: Python library built on top of NumPy. Great program for data analysis and maniupulation. One of the best tools for data preparation.
- Matplotlib: Python library used for 2D plotting, histograms, bar charts, errorcharts, and more. Has a MATLAB-like interface.
- Kong: The “world’s most popular open source API gateway.” Works in multi-cloud, hybrid, and distributed environment, and supporting microservices.
- Tyk: Lightweight, open source API gateway that stores detailed data on user interation with the API.
- KrakenD: High performance API gateway that supports +18k request/second on commodity hardware.
Application Performance Monitoring
- Pinpoint: APM tool for large-distributed systems. Written in Java/PHP. Tools helps you understand application topology and structure of the system, showing componentes interconnectedness, tracing transactions across the application. Minimal impact on performance. Install agent without changing code.
- inspectIT: APM tools for monitoring and analyzing Java(EE) applications. Remote sensors capture data from every request from the end user to business tier to the backend. Creates traces and correlates calls between JVM nodes in application.
- stagemonitor: APM tool for Java server apps. Supports distributed tracking using Open Tracing API. Use in development, QA, and production. Metrics are displayed in tables and graphs.
- MoSKito: Monitoring tool for Java applications. Collects performance data from threads, memory, caches, storage registrations, payments, conversions, and more. Supports apps such as Tomcat, jetty, Glassfish, Weblogic, and Websphere. Records user actions inside the web app.
BGP (Border Gateway Protocol)
- Quagga: Routing suite that supports OSPFv2/v3, RIO, RIPng, and BGP-4 on FreeBSD, Solaris, Linux, and NetBSD. Fork of GNU Zebra. Components include a daemon and zebra, an abstraction layer underlying the Unix kernal. Daemons configured va CLI.
- Bird: Internet routing daemon. Works on Linux, FreedBSD, and others. Supports IPv4, IPv6, BGP, RIP, SPF, BFD, Babel, inter-table protocol, and static routes. Developed as school project at Charles University in Prague.
- ExaBGP: Based on SDN application, this tool is popular among ISP’s. It provides cross data center failover and mitigation against network attacks. The healthcheck feature anycasted DNS service failures can be detected.
Caching and Reverse Proxy
- Nginx: The most popular cache in the world. The softwar can be used as a reverse proxy, load balancer, and/or and HTTP cache. Although F5 acquired Nginx, they are commiitted to the continued development of the open source product.
- Varnish: Software was designed as an HTTP accelerator for dynamic websites. Its highly customizable using VCL, a domain-specific language and each connection is handeled by a worker thread. CDNs like Fastly and StackPath use Varnish.
- Apache Traffic Server: Software can be used as a reverse proxy and forward proxy. It is one of the oldest caching products in the market, developed by Inktomi, acquired by Yahoo, then donated to Apache Foundation.
Container Security Tools
- Anchore: Tool offers inspectoins, analysis, and certification of images. Provided as docker image and can be run on Kubernetes, Swarm, Rancher, and other platforms. Works as standalone product or in CI/CD setup.
- CoreOS Clair: Tool that offer static analysis on vulnerabilities in application containers. Ingest vulnerability metadata from various sources and stores it in a database. An API allows queries to the database to find vulnerabilities.
- Dagda: Tool that provides static analysis on known vulnerabilities including viruses, malware, trojans, and other threats in docker images and containers. Tool pulls data from software installed on docker image.
- OpenSCAP: Security policy written in SCAP from that consist of detailed rules used for remediation scripts. Implements policy recommended by DSS, PCI, STIG, and USGCB.
- Sysdig Falco: Behavior activity monitor. Continuously monitors container, host, application, and network activity. Detects and alerts can be set up based on system calls, call arguments, and calling properties.
- Spinnaker: Developed by Netflix, this tool is used for continuous deliver of software releases to multi-cloud environments. Can create deployment pipelines to run system test and monitor rollouts.
- Jenkins: Leading automation server used to automate task for “building, testing, delivering and deploying software.” Written in Java and runs in servlet containers like Apache Tomcat. Supports Subversion, Git, Performce, Mercurial, and other version control tools.
- Redis: Data structure server that provides “mutable data structures” through commands using a server-client model. Stores data structures on disk, while it is served from memory, making it performant. Features included replication, HA, clustering, and more.
- PostgreSQL: Relational database that has been around for decades. It’s reliable, performant, and feature rich. Used in simple set ups and also large scale environmnets like a data warehouse.
- Luigi: Developed by Spotify, the tool is a Python package the helps in building sophisticated pipelines needed for features such as tasks as recommendations, top list, and more. Supports MapReduce jobs in Hadoop, Pig, and Hive.
- Azkaban: Workflow manager developed by LinkedIn. Helps in scheduling time-based dependencies for Hadoop batch jobs. The web server component takes care of project management, scheduling, authentication, and monitoring executions.
- Oozie: Workflow scheduling system developed in Java used for Hadoop jobs. Workflows are arranged in DAGs. Control nodes define when jobs start, stop, fork, and join other nodes along execution path.
- Apache Airflow: Developed by Airbnb, the tool is used for authoring, scheduling, and monitoring workflows as DAGs. The rich CLI enables end users to see dependencies, logs, process, and when tasks are completed.
Data Serialization / RPC
- Protocol Buffers: Protobuf, as it is known was developed by Gooogle and is a protocol that enables serialization and deserialization of structured data. The primary aim was to make communication over wire better than XML. It supports Objective -C, Python, Java, and other languages.
- Apache Thrift: Facebook developed this RPC framework its known for being a “scalable cross-language services” that works with a code generation engine to support services between Python, PHP, Java, C++, Perl, Erland, and many more.
- Apache Avro: RPC tools that is a more recent serialization system that works on a schema-based system. and uses JSON for defining various data types. It provides rich data structures in compact binary form.
- Jaegar: Developed by Uber. End-to-end distributed tracking tool. Product can store elements, visualize, and filter traces. Supports Python, Ruby, and PHP. Provides an agent on every host to collect data.
- Zipkin: Developed by Twitter. One of the 1st of its kind. They described it as “gather timing data for all the disparate services involved in managing a request to the Twitter API”. Also describe the tool as “performance profiler” like Firebug.
File Systems (Distributed)
- Lustre: Most widely used FS in the HPC community, at least the Top 500 HPC sites worldwide. Can be used over SAN and NFS file systems. Mainly used on Linux HPC clusters. Ideal for distributing very large files over many nodes.
- GlusterFS: Network file system idea for media streaming and cloud storage. Uses block storage, where it stores chunks of data in open space connected to Linux computers, like scale-out NAS and object stores.
- Ceph: Linux-based distributed file system. Built in replication and fault tolorance – POSIX compatable. Provides traditional file system interface. Uses object storage, thus storing data in binary objects.
- ClearOS: Works on ClearOS. A linux firewall the comes with many features. Rules can be created to allow external connections to the OS and/or block an IP address or network from accessing the OS.
- pfSense: Community edition is open source. Rules can be set up to block or allow certain types of traffic. Can also be used as a router. Under Apache License 2.0
- OPNsense: FreeBSD firewall and routing platform. Its a fork of pfSense product. Comes with traffic shaper, forward caching proxy, two-factor auth, netflow monitoring, and much more.
- IPFire: Linux based firewall and router. Comes with a management console. Started as a fork from IPCop but its been completely rewritten. Fetures include state packet inspection firewall, QoS, GeoIP filtering, and much more.
- VyOS: Routing platform that can also be used as a firewall and VPN gateway. GRE and IPIP tunneling protocols supported, as well as L2TPv3 and VXLAN. Runs on virtual machines, bare metal hardware, and various cloud platforms.
- Smoothwall Express: Popular firewall that is simple to use. Can be set up on web browser and supports lots of different netwokrs cards. Supports DMZ, LAN, and wireless. Features includes QoS, portsfoward, IDS, and web proxy.
- csf (ConfigServer): Firewall config script that provides advanced security. The login failure daemon tracks user activity for excessive logins. Easy to use interface helps in managing the settings.
- ModSecurity: One of the most popular web application firewalls (WAF) in the world. Works with Apache HTTP Server, Nginx, and Microsoft IIS. Other companies like Verizon and Cloudflare have rewritten for Sailfish and Nginx.
- Snort: One of the most popular IDS tools in the community. It can be used as a packet sniffer, packet logger, or IDS. Features include real time traffic analysis, protocol analysis, and content matching. Can also detect probes such as fingerprinting attemps, buffer overflows, stealth port scanes, semantic URL attacks, server message block probes, and more .
- Suricata: Mature IDS/IPS system. Supports real time intrusion detection, network security monitoring, and offline pcap processing. Some features include Gzip compression, DNS logger, IP reputation, multi-threading, and much more.
- OSSEC: Host intrusion detection system. Features include log analysis, rootkit detection, time-based alerting, and integrity checking. Provides detection Linux, OpenSD, FreeBSD, Windows, and Solaris.
- OpenDLP: Centrally managed data loss prevention tool. Identifies sensitive data at rest on thousands of systesm including Microsoft, Unix, and MySQL. Able to perform scans on Windows files systems, Microsoft SQL, and more.
- Zeek: Network analysis framework. Event engine analyzes recorded network traffic and live traffic. Tool performs anomaly detection, application layer decoding, signature matching, and connection analysis.
- Samhain Labs: Host-based IDS. Tool provides file integrity checking, log file monitoring, log file ananlysis, centralized loggin, and monitor multiple host systems.
- Nmap Scan: Network scanning tool. Useful for pen testing. User friendly for beginners. Simple CLI command nmap <target> can start the process. Converts <target> to IPv4 address using DNS.
- ARP Scan: Tool that displays every IPv4 device on the subnet even if protect by firewall. Non-routable so it works only on a LAN. Very fast. Arp packets are better for scanning because they cannot be hidden like ping.
- Angry IP Scanner: Tool that works on Linux, Windows, and MAC. Light weight and fast. Supports NetBIOS and favorite IP address ranges. Plugins are available. Plugins can be written in Java.
- Advanced IP Scanner: Network scanner for LAN. User friendly and fast. Able to located systems on wireless and wired networks. 40 million users. Provide access to computers remotely.
- Fping: Tools the provides the ability to send ICMP echo probes. Much better than ping when it involves pinging to multiple host.
Java Build Tools
- Gradle: Build-automation framework. Uses domain-specific language called Groovy instead of XML. Uses DAG’s to determine the order of tasks. Builds upon Ant and Maven.
- Apache Maven: Build-automation tool for Java projects. Provides build-process management and project processing features. Supports a variety of plugins.
- Apache Ant: Java library and CLI tool. Used to build Java applications.
Kubernetes Dev Tools
- CodeReady: Red Hat CodeReady Studio provides and a comprehensive set of tools and frameworks to support varoius programming modles such as EJB, JSF, OpenShift, and many more.
- Skaffold: CLI tool for building Kubernetes applications. Works on CI/CD basis where developers can create code locally, then deploy it anywhere. Tools manages the workflow from build to deploy.
- Draft: Product makes it easy to build application for Kubernetes. Works in a CI/CD basis where developers can build apps, then commit and push to version control. Constructing CI pipelines is simple since it builds upon Kubernetes Helm and Kubernetes Chart format.
- Squash: Tool for debugging microservices. Its a modern tool built specifically for debugging microservices processes across multiple systems. Whereas OpenTracking and Istio are fine tools, they are “passive and feedback loop is slow.” Also, they can’t monitor apps at runtime. Squash fixes all this.
- Telepresence: Local development environment of building Kubernetes services. Tool allows you to develop locally, then run code in a remote Kubernetes cluster easily.
- Ksync: Helps programmers build applications for Kubernetes quickly. It does so by updating all containers in a cluster from your local station. Developers can use their IDE’s of choice like Sublime or Atom inside the cluster.
- HAProxy: Most popular load balancer in the market. Suited for high traffic environments. Load balancer and proxy for HTTP-based applications and TCP. Features include multi-threading, on-the-fly server additions and removals.
- Seesaw: Linux Virtual Server that provides basic load balancing capabilities. Supports physical and virtual machines. Each node requires two interfaces-one for the host and other for cluster VIP.
- Neutrino: eBay developed the product and open sourced it. Created in Scala language. Does L7 routing and has highly extensible architecture. New modules and functionality can be added to product easily.
- Katran: Developed by Facebook. HIgh performance L4 load balancer. Built in C++. Uses XDP to provide very fast packet processing. Scales linearly. Doesn’t need to wait for TTL to redirect. Better than anycast solution for large envirionments.
- Graylog: Popular log management tool. Horizontally scales. Analyze TB’s of data. Perform search, aggregate, visualize, analyze, and report on data. Simple dashboards. Build queries in minutes and also sophisticated ones.
- Logstash: Tool that helps collect, parse, and store logs. Many different inputs are supported. Built on Elasticsearch. Server-side processing pipeline. Ingest data and transforms it.
- Fluentd: Data collection project. Written in Ruby. Unifies data collection. Works on semi and unstructured data. Analyzes clickstreams, event logs, and application logs.
- Syslog-ng: Log daemon. Supports syslog, message queues, SQL, NoSQL, unstructured test, and more. High performance. Can scale to 800k messages per second. Normalize and process log as they go through system.
- Logwatch: Customizable log monitoring system. Goes through logs over a certain time frame and reports back results, that are set by you.
- Apache Kafka: Created by LinkedIn. Streaming processing platform. Developed in Java and Scala. High performance. System is based on commit log. Three parts are producers, topics, and consumers.
- RabbitMQ: Most popular messaging broker. Lightweight and easy to deploy. Works on premise and in the cloud. Supports many different operating systems. Supports message queuing, devliery acknowledgement, multiple message protocols, and much more.
- Nagios: Tool that monitors networks and systems, including servers, switches, applications and services. Written in C. At protocol and resource level, monitors POP3, FTP, SSH HTTP, SMTP, processor load, disk usage, system logs, and more.
- Cacti: Front-end tool that monitors networks. Uses poll services to monitor and graph results. Ideal for polling router interface or switch using SNMP.
- Zabbix: Monitors servers, networks, virtual machines, and cloud service. XML templates can be set up to monitor metrics including CPU load, disk space consumption, and network utilization.
- Incinga 2: Monitors network resources. Alerts users when outages occur. Reports on performance data. Can also monitor services such as HTTP, SMTP, SNMP, printers, routers, switches, and other networsk connected services.
- Libre NMS: Network monitoring tool. The supported features include alerting, auto discovery, distributed polling, netflow, syslog, traffic billling, and data collection of protocols such as STP, BGP, and. OSPF.
- Pandora FMS: Monitors firewalls, proxies, web servers, databases, routers, SNMP, TCP, UDP, HTTP, load balancers, swithces, printers, and more.
- Prometheus: One of the most popular event monitoring and alerting tools for Kubernetes. Metrics are stored in a time series database. Written in Go. Comprised of several parts including exporters, Alertmanager, PromQL, and Grafana.
- Grafana: Analytics and monitoring solutions. Referred to as observability platform. Able to query, visualize, and alert. Create histograms and heatmaps. Supports many different databases.
- Wireshark: Most popular packet analyzer runs on multiple OS. For network troubleshooting and analysis. Data captured from the wire. Understands hundreds of protocols. Analyzes VoIP traffic.
- Nmap: Besides being an IP scanner, the tool helps identify hosts and services on a network. Analyzes packet responses.
- Snabb: Packet networking toolkit. Runs on Linxus. Written in Lua and LuaJIT compiler. Three parts including Snabb NFV, IwAFTR, and Packetblaster.
- PacketShader: Routing platform that uses GPU’s. Offloads memory intensive routing services and computation. Avoids CPU performance bottleneck that occur in other routing software stacks.
- Vector Packet Processing: The open source version of Cisco’s VPP technology. Hardware, kernal and deployment agnostic. Feature rich, providing deep switch and routing functionality.
Packet Performance Tools
- eXpress Data Path: Program that lets users control the flow of the packet. Incoporate into the NIC driver between interrup proessing and memory allocations decide packet ate. Drops 26M packets/second on commodity hardware.
- eBPF: Originally developed for packet capturing and filtering. eBPF virtual machien allows instructions to be mapped more efficinetly to hardware ISA to improve performance.
- Pentoo: Gentoo install that includes more customized tools and kernal. XFCE4 wm. Available in binary package. Pentoo-updater keeps tool up to date.
- NodeZero: Ubuntu-based linux tool. Uses Ubuntu repositories, thus system is updated regularly.
- Kali: Linix penentration testing platform. Platform consist of a collection of many pen-test tools. Comes with metapackages for web apps, wireless, forensics, and software-defined radio.
- BackBox: Community project. Contains its own repositories. Network and systems analysis toolkit. Can stimulate various types of attacks. Used for stress test, network analysis, vulnerability assessment, forensics, sniffing, exploitation, and more.
- OpenR: Developed by Facebook, it is a routing platform they use internally. Enables them to develop and deploy apps on the network at scale. Superior to OSPF and IS-IS protocols for large environments.
- CloudRouter: Routing stack designed for the cloud. Runs on virtual, physical, and cloud environments. Supports containers and software-defined networks.
- Consul: Networking service that connects secure services in the public and private cloud and any runtime platform. Supports shift from static to dynamic infrastructure changes.
- Apache Zookeeper: Tool that offers distributed configuration services, synchronization, and naming registry. Highly available. Data is stored in hierarchical name space.
- Apache Flink: Framework written in Java and Scala. Stateful and fault tolerant. Works at scale, supporting thousands of noedes. Flexible windowing. Has checkpoint feature in case of failure.
- Apache Spark: Ideal for in-memory processing of batch data. Works in machine learning environments. Compliments Hadoop, which can extend Spark’s true computing prowess. ML library avaialble for product.
- Apache Storm: real time computation system. Process unbound streams with ease. Works well in machine learning models. Has real time analytics. processes complex streams at different stages of compute.
- Apache Kafka: Ideal for complex setups. Processes inputs as changes and handels out-of-order data. One unique feature – works as a traditional messaging system and distributed system.
Subdomain Scanner Tools
- Amass: OWASP project. Tool helps IT perform network mapping of attack surface. Provides external asset discovery. Gathers info using active recon techniques.
- SubBrute: Fast subdomain enumeration tool. Very high accuracy rates. Has open resolvers that acts as a proxy to bypass DNS rate limiting.
- Knock: Developed in Python. Enumerates subdomains. Perfoms scans of DNS zone transfer. Automatically circumvents wildcard DNS records.
- DNSRcon: Pne testing tool. Enumerates DNS records for certain records including SOA, NA, A, AAAA, SPF, MX, and test. Checks all NS records for zone transfers.
- Sublist3r: Python tool that perfoms fast enumeration of DNS. Assist pen testerscollect subdomains of the target. Emuneration done via Virustotal, ThreatCrowd, Netcraft, ReverseDNS, and DNSdumpster.
- AltDNS: Recon tool. Discover subdomains. Ideal for large datasets. Generates valid subdomains when dataset is 200+ subdomains.
UDP File Transfer
- Tsunami: Leverages TCP control. For data transfer over long distances, uses UDP. Tool was designed to provide much higher throughput than TCP.
- UDT: Ideal for uses cases when TB size files are transferred between locations.
- Enet: Tool provides a simple, lightweight, and flexible communication layer that sits on top of the TCP protocol.
- UFTP: Transfer large files across the global using receivers. Works well when satellite links are involved.
- GridFTP: Product runs over FTP, not UDP. Ideal for scenarios where there are TCP problems.
- QUIC: Developed by Google. Uses UDP. Supports multiplexed connections over UDP. Uses bandwidth estimation feature that helps avoid congestion.