Skip to content

Commit

Permalink
v0.2
Browse files Browse the repository at this point in the history
  • Loading branch information
@hl2guide
hl2guide authored Jun 13, 2024
1 parent 4691c22 commit 3b08524
Show file tree
Hide file tree
Showing 2 changed files with 177 additions and 16 deletions.
31 changes: 22 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,28 @@

An All-In-One AdGuard Home blocklist and allowlist generated locally on your PC using Python.

Version: 0.1.25
Version: 0.2

Updated (AEST): 2024-05-16 02:41:49PM
Updated (AEST): 2024-06-14 01:32:25AM

## Repo Status
## ✨ What's New in 0.2
- Added new must-have requirement of a Python package named [__requests__](https://pypi.org/project/requests/)
- Added new lists
- More cleanup of output rules

## 🔬 Repo Status

- Written in **Python** it now has been updated for 2024 and have new lists added
- Currently around **1.2 million unique rules** are included
- The blocklist will make AdGuard Home's RAM usage increase to around **330MB**
- Currently around **5.2 million unique rules** are included
- The blocklist will make AdGuard Home's RAM usage increase to around **775MB**
- Do not use these lists on older hardware with fewer than **4 CPU cores** or **IOT** devices
- Be prepared to make your own whitelist additions within AdGuard Home, depending on your needs
- I attempted and tested using **GitHub Codespaces** and **github LFS** but moved away from both due to filesize restrictions

## Included Blocklists
## ⛔ Included Blocklists

- [1Hosts (Lite)](https://adguardteam.github.io/HostlistsRegistry/assets/filter_24.txt)
- [AdAway Default Blocklist](https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt)
- [AdGuard DNS filter](https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt)
- [AdGuard Simplified Domain Names filter](https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt) by AdGuard
- [Anudeep's Blacklist](https://github.com/anudeepND/blacklist) - [RAW](https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt) by Anudeep
Expand All @@ -31,7 +37,7 @@ Updated (AEST): 2024-05-16 02:41:49PM
- [Ransom](https://blocklistproject.github.io/Lists/adguard/ransomware-ags.txt)
- [Scam](https://blocklistproject.github.io/Lists/adguard/scam-ags.txt)
- [Tracking](https://blocklistproject.github.io/Lists/adguard/tracking-ags.txt)
- [AdAway Default Blocklist](https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt)
- [Blocklists by sefinek](https://sefinek.net/blocklist-generator/adguard) - ✨ 0.2
- [Dan Pollock's List](https://adguardteam.github.io/HostlistsRegistry/assets/filter_4.txt)
- [Dandelion Sprout's Anti Push Notifications](https://adguardteam.github.io/HostlistsRegistry/assets/filter_39.txt)
- [Dandelion Sprout's Anti-Malware List](https://adguardteam.github.io/HostlistsRegistry/assets/filter_12.txt)
Expand All @@ -55,7 +61,14 @@ Updated (AEST): 2024-05-16 02:41:49PM
- [stamparm's ipsum](https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt)
- [uBlock₀ filters - Badware risks](https://adguardteam.github.io/HostlistsRegistry/assets/filter_50.txt)

## Setup
## ⚒️ Setup

1. Install [Python](https://www.python.org) on your device
2. Install a required Python package "requests" by running:

```
python -m pip install requests
```

- The python script `create_list.py` can be used to generate `aio_blocklist_final.txt` on a local machine that has Python
- I have tested it on Windows 11, however it should work fine on other OSs too (as long as you have Python installed)
Expand All @@ -70,7 +83,7 @@ Updated (AEST): 2024-05-16 02:41:49PM

To regenerate the list simply run `create_list.py` when desired (as a scheduled task etc)

# Archived Old Info
# 📦 Archived Old Info

The below is old info that shouldn't be needed anymore.

Expand Down
162 changes: 155 additions & 7 deletions create_list.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,19 @@

# FOR ADGUARD HOME ONLY

# Version: 0.1.25
# Date: 2024-05-16 02:41:49PM
# Required Packages
# A python package called "requests" is required to handle 403 errors (install that package first by running: python -m pip install requests)

# Version: 0.2
# Date: 2024-06-14 01:20:58AM

# Imports
from time import gmtime, strftime
from urllib.request import urlretrieve
import requests
#from urllib.request import Request, urlopen
#import requests
#from random import seed
import glob
import os

Expand Down Expand Up @@ -92,6 +99,126 @@
"https://raw.githubusercontent.com/borestad/blocklist-abuseipdb/main/abuseipdb-s100-60d.ipv4",
# ipsum : https://github.com/stamparm/ipsum
"https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt",
# Sefinek-Blocklist-Collection (many lists) - https://sefinek.net/blocklist-generator/adguard
"https://blocklist.sefinek.net/generated/v1/adguard/abuse/blocklistproject/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/abuse/urlhaus.abuse.ch/hostfile.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/0Zinc/easylist.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/DandelionSprout.GameConsoleAdblockList.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/FadeMind/UncheckyAds.fork.txt",
#"https://blocklist.sefinek.net/generated/v1/adguard/ads/MajkiIT/SmartTV-ads.fork.txt", # Testing Samsung TV login issues 05.06.2024
"https://blocklist.sefinek.net/generated/v1/adguard/ads/ShadowWhisperer/Ads.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/adaway/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/anudeepND/adservers.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/blocklistproject/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/blocklistproject/youtube.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/craiu/mobiletrackers.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/crazy-max/spy.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/disconnectme/simple-ad.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/AdguardDNS.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Admiral.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Easylist.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Prigent-Ads.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/jerryn70/GoodbyeAds.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/r-a-y/AdguardMobileAds.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/sefinek.hosts.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ads/yoyo/ads-trackers-etc.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/amp/developerdan/amp-hosts-extended.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/amp/ente-dev/google-amp-hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/anime/myanimelist.net.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/anime/shinden.pl.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/apps/whatsapp.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/Snota418/Crypto-streams.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/firebog/Prigent/Crypto.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/hoshsadiq/adblock-nocoin-list.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/zerodot1/CoinBlockerLists-hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/sites/sefinek.hosts.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/dating-services/developerdan/extended.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/dead-domains/jarelllama/dead-domains.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/drugs/blocklistproject/drugs.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/FadeMind/add-2o7Net.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/MajkiIT/adguard-host.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/MajkiIT/easy-privacy-host.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/cbuijs/adult-domains-24733.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/CountryCodesLists-France.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/ParsedBlacklists-EasyList-Liste-FR.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/ParsedBlacklists-EasyList.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/hagezi/pro.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/justdomains/adguarddns-justdomains.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/oisd/big.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/r-a-y/AdguardApps.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/r-a-y/AdguardMobileSpyware.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/fakenews/StevenBlack/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/fakenews/marktron/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/fraud/blocklistproject/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/MajkiIT/gambling-hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/StevenBlack/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/TrustPositif/gambling-indonesia.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/blocklistproject/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/sefinek.hosts.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/games/league-of-legends.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/games/valorant.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/hate-and-junk/developerdan/extended.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/AssoEchap/stalkerware-indicators.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/DandelionSprout-AntiMalwareHosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/RPiList/Malware.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/Spam404/main-blacklist.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/bigdargon/hostsVN.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/blocklistproject/malware.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/digitalside/latestdomains.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/disconnectme/simple-malvertising.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/malware-filter/urlhaus-filter-hosts-online.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/phishing.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/quidsup/notrack-malware.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/reported-by-norton.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/sefinek.hosts1.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/sefinek.hosts2.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/suspicious.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/web-attacks.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/Dogino/Discord-Phishing-URLs-phishing.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/RPiList/Phishing-Angriffe.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/blocklistproject/phishing.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/phishing.army/blocklist-extended.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/4skinSkywalker/hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/Sinfonietta/pornography-hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/StevenBlack/porn.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/blocklistproject/porn.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/chadmayfield/pi-blocklist-porn-all.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/oisd/nsfw.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/porn/sefinek.hosts.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/ransomware/blocklistproject/ransomware.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/redirect/blocklistproject/redirect.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/scam/Dogino/Discord-Phishing-URLs-scam.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/scam/blocklistproject/scam.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/scam/durablenapkin/scamblocklist.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/scam/jarelllama/scam.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/booth.pm.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/gamebanana.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/omegle.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/patreon.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/pinterest.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/pixiv.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/sites/riotgames.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/social/facebook.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/social/instagram.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/social/snapchat.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/social/tiktok.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/social/twitter.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/spam/FadeMind/add-Spam.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/spam/RPiList/spam-mails.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/spam/stopforumspam/toxic-domains-whole.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/suspicious/FadeMind/add-Risk.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/suspicious/firebog/w3kbl.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/test.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/0Zinc/easyprivacy.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/MajkiIT/adguard-mobile-host.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/ente-dev/tv.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/frogeye/firstparty-trackers-hosts.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/mitchellkrogza/INACTIVE.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/neodevpro/host.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/quidsup/trackers-hosts.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/useless-websites/jarelllama/parked-domains.fork.txt",
"https://blocklist.sefinek.net/generated/v1/adguard/useless-websites/sefinek.hosts.txt",

#"https://hosts.anudeep.me/mirror/adservers.txt"
# LostAd [TOO BIG]
#"https://raw.githubusercontent.com/lennihein/LostAd/main/lostad_dns.txt",
Expand All @@ -117,9 +244,26 @@
for URL in URLS:
LIST_INDEX = URLS.index(URL)
FILENAME = CURRENTWORKINGDIRECTORY + "downloaded_lists\\blocklist" + str(LIST_INDEX) + ".txt"
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
urlretrieve(URL, FILENAME)
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
# print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
# urlretrieve(URL, FILENAME)
# NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())

# Uses a workaround for 403 errors by applying a better "User-Agent" header to the request
if URL.startswith("https://blocklist.sefinek.net/"):
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
HEADERS = {
'User-Agent': 'Mozilla 5.0',
}
RESPONSE = requests.get(URL, headers=HEADERS)
CONTENT = RESPONSE.text
if RESPONSE.status_code == 200:
with open(FILENAME, "w", encoding='utf8') as file:
file.write(CONTENT)
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
else:
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
urlretrieve(URL, FILENAME)
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())

# Merges lists into one file
# Source: https://bobbyhadz.com/blog/merge-text-files-in-python#how-to-merge-text-files-in-python
Expand Down Expand Up @@ -158,7 +302,8 @@
with open(CURRENTWORKINGDIRECTORY + "aio_blocklist_final.txt", "w", encoding='utf-8') as new_f:
for line in LINES:
if not line.startswith("!") and not line.startswith("#") and not line.startswith("@@") \
and not line.startswith("$removeparam="):
and not line.startswith("$removeparam=") and not line.startswith("*") \
and not line.startswith("-") and not line.startswith(".") and not line.startswith("/"):
# Removes line prefixes
line = line.replace('0.0.0.0 ','')
line = line.replace('127.0.0.1 ','')
Expand All @@ -169,8 +314,11 @@
line = '||' + line
line = line.replace('||||','||')
line = line.replace('^','^$important')
if not (line == '') and not (line == ' ') and not (line == '||') and not '##' in line:
if not (line == '') and not (line == ' ') and not (line == '||') and not (line == '##'):
line = line.replace('^','^$important')
line = line.replace('||||','||')
line = line.replace('|||','||')
line = line.replace('$important$important','$important')
new_f.write(line)
os.remove(CURRENTWORKINGDIRECTORY + "aio_blocklist.txt")

Expand Down

0 comments on commit 3b08524

Please sign in to comment.