WireHole is a combination of WireGuard, Pi-hole, and Unbound in a docker-compose project with the intent of enabling users to quickly and easily create a personally managed full or split-tunnel WireGuard VPN with ad blocking capabilities thanks to Pi-hole, and DNS caching, additional privacy options, and upstream providers via Unbound.
Interesting... Via Awesome Wireguard.
This is an extension of the very common find ... -exec grep ... {} \;
construct I use almost daily to find which files contain a particular text string.
Now, let's say you're looking for files in all your Ansible roles containing the string ppa:
, because you want to create a new role using a suitable existing role as a template. In this case, I think most recently modified is an excellent proxy for suitability.
Thus, the challenge: can we tack on something to the find ... grep
construct such that the output shows matching files in order of most recently modified?
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" | grep -v "^\\./.*" | sort && cd $OLDPWD
2021-06-09+22:27:20.6889730070 ./roles/public/php-versions/tasks/setup-Debian.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-02-23+15:33:13.2318546100 ./roles/dev/deluge/tasks/install.yml
2022-03-31+04:42:16.5644336650 ./roles/dev/editor-notepadqq/tasks/main.yml
2022-05-09+15:21:01.7974094830 ./roles/dev/qownnotes/tasks/main.yml
2022-06-25+02:46:13.0580097480 ./playbooks/workstation/roles/boot-grub/tasks/main.yml
2022-06-25+02:46:18.2940273650 ./roles/dev/libreoffice/tasks/main.yml
2022-07-03+00:16:44.5563131800 ./roles/dev/shutter/tasks/main.yml
2022-07-04+20:35:15.6128270470 ./roles/dev/magnus/tasks/main.yml
2022-07-14+21:09:01.6502524570 ./roles/dev/flatpak-remote/tasks/main.yml
2022-07-14+22:05:43.3325667290 ./roles/public/firejail/tasks/install.yml
2022-07-16+13:42:04.3756138100 ./roles/public/variety/tasks/main.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-07-21+00:14:51.1137716920 ./roles/public/foliate-ebookreader/tasks/ppa.yml
2022-07-21+03:07:42.2876610030 ./roles/public/graphics-driver-nvidia/tasks/install.yml
2022-07-21+06:05:27.4514643180 ./roles/public/foliate-ebookreader/tasks/flatpak.yml
2022-07-23+18:32:46.8466638700 ./roles/dev/handbrake/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-08-19+15:28:49.5605481840 ./roles/dev/x2goserver/tasks/main.yml
2022-11-04+11:35:14.6208169990 ./roles/public/python3/tasks/python-ppa.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2022-12-31+19:38:32.9105553030 ./roles/public/digikam/tasks/install-ppa.yml
2023-01-01+01:39:08.2045090970 ./roles/public/digikam/tasks/install-appimage.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
2023-01-26+14:10:18.9247087870 ./roles/public/ansible/tasks/main.yml
2023-01-26+16:14:59.9903243110 ./roles/public/sioyek-pdf/defaults/main.yml
2023-05-12+12:01:30.4705549280 ./roles/public/mpv/tasks/install.yml
2023-05-13+00:47:41.1561557100 ./roles/public/nextcloud-desktop/tasks/main.yml
2023-08-27+18:19:44.8291334420 ./roles/public/digikam/defaults/main.yml
Eureka!
cd <path-parent>
ensure that the resulting paths displayed by find
don't contain the <path-parent>
part (to avoid cluttering the output), and the final cd $OLDPWD
just make sure that the bash prompt is not changed to <path-parent>
. -not -path '*/<some-path>/*'
.grep -i
for case insensitive matching, and -l
(that's the letter l
for list) makes grep
print only the filename and not each matching line (this is crucial for this hack to work, we want grep
to produce as little output as possible, in fact, if I could figure out a way to silence grep
altogether I would have, but I couldn't).-printf "%T+ %p\n"
adds the file mtime
to the output (on a new line). Thanks angus@Unix.SE.At this point, an example of the unfinished product is order. Before sorting, and before the final grep -v
, the output looks like this (excerpt):
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
./roles/public/java-openjdk/defaults/main.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-firefox/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
./roles/public/browser-chromium/tasks/install.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
with the grep
output on its own line, followed by the time-stamped output of printf
. Like I said, it would have been better if we could somehow silence the grep
output at this point. If you know a way, feel free to let me know!
As expected, sorting resulted in the non-time-stamped lines dangling about like some unwanted appendage:
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
./roles/public/java-openjdk/defaults/main.yml
At this point I was out of ideas, so grep -v ...
it was, and we end up with the one-liner shown above. It's an ugly hack, but hey, it works :-)
Bash 5.1 on Ubuntu 22.04.3, with GNU find 4.8.0, GNU grep 3.7, and GNU sort 8.32.
This is just a quick-and-dirty one-liner. I was compiling beamer slides, and for some reason the TikZ-generated PDF figures contained two pages under some circumstances. This way we can quickly see whether just some PDF files or all of them suffered a problem:
$ find figure/ -type f -name "*.pdf" -exec pdftk "{}" dump_data \; | grep NumberOfPages | awk '{print $2}'
1
1
1
1
1
1
(in this case they were all one page long).
Passwordstore is a great password manager, and I rely on it also in my Ansible playbooks, where it works by causing the gpg-agent to prompt me for the passphrase of my gpg key.
This prompt is a GUI prompt, which is very suitable when sitting at the computer in question. But a small annoyance is that it does not work at all when working on a remote computer via ssh (the prompt shows up on the remote computer's desktop, and the Ansible playbook in the terminal just freezes until it eventually fails).
It would be so much nicer if those ssh terminal sessions would instead get the gpg-agent prompt in the terminal. So far I have not found a method that achieves this without also sacrificing the GUI desktop prompt for non-remote work.
Desktop and laptop running Ubuntu 22.04 with i3wm desktop.
The relevant parts of my config can be seen in https://codeberg.org/ansible/dotfiles.
I considered the following related Q:s&A:s but did not achieve the desired outcome.
In the SSH session (no effect, unfortunately):
gpg-connect-agent updatestartuptty /bye
Learned that the gpg-agent is running in --supervised
mode, and its ENV variables include DISPLAY=:0
:
solarchemist@desktop:~
$ sudo cat /proc/2652288/environ
HOME=/home/solarchemist LANG=en_US.UTF-8 LC_TIME=sv_SE.UTF-8 LOGNAME=solarchemist
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
SHELL=/bin/bash SYSTEMD_EXEC_PID=2652288 USER=solarchemist
XDG_DATA_DIRS=/home/solarchemist/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/1000 QT_ACCESSIBILITY=1
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus DISPLAY=:0 XAUTHORITY=/home/solarchemist/.Xauthority
MANAGERPID=1532 LISTEN_PID=2652288 LISTEN_FDS=4
LISTEN_FDNAMES=browser:extra:std:ssh INVOCATION_ID=<stuff> JOURNAL_STREAM=<stuff>
(the PID of the gpg-agent process is easily identified with ps aux | grep [g]pg
).
Manual work-around is to set pinentry-program /usr/bin/pinentry-tty
in ~/.gnupg/gpg-agent.conf
and reload the agent gpg-connect-agent reloadagent /bye
.
To revert to the default (GUI) pinentry behaviour, just remove the line and reload the agent again.
How could we go about to draw a straight line (vertical or horizontal) on top of any other window on our Linux desktop? I'm using i3
window manager with picom
compositor at present, so I'm primarily interested in solutions that works for that.
I should get back to this question in the future and implement a nicer solution than pango-view
Produces a vertical line (actually, more of a vertical box of limited width).
Very easy to make it, just issue the command in a terminal. pango-view
was already installed.
$ pango-view --height=99999 --margin=1 --background=red -t ''
gromit-mpx
is a good over-app painter, but does not provide perfectly straight linesWill add more as I learn about them. I only care about FOSS projects.
See my Ansible role that installs and configures LXD and provisions LXC containers.
I have some experience with Docker containers. Prefer LXC if I have the choice.
I have written an Ansible playbook to setup containers with docker-compose as part of a as-yet not-public project for InvenioRDM.
No hands-on experience yet.
Hm, interesting. Could it be better than Docker for sharing something like a thesis with "all batteries included", I wonder?
Singularity can convert Docker containers to Singularity, or can run containers directly from Docker Hub
https://blogs.oregonstate.edu/learningbydoing/2022/01/04/docker-and-singularity-containers-which-one-is-better/
Learn the skills required to sysadmin a remote Linux server from the commandline.
Course restarts every first Monday of the month
I wanted to create an ASCII art "bismillah" for use in the terminal.
Found this beautiful calligraphy by Nuria Garcia Masip. After cropping it, rotating it slightly and making the background monochromatic, I tried the ascii-image-converter by Zoraiz Hassan, and it worked great with the --braille
flag:
This converter tool was really easy to install (just run the binary), and the output was automatically resized to fit the terminal window size, which was a convenient feature.
I discovered some other ASCII art CLI tools (never tested them though):
This is a neat way to quickly generate a QR-code, for example to transfer text from computer to phone (thanks to Solène Rapenne for the original idea and implementation). On the phone, you'll of course need to use a QR-reader app, such as Binary Eye.
xclip -o -selection clipboard | qrencode -o - -t PNG | feh -g 600x600 -Z -
Using this command, whatever is in your clipboard will be encoded to QR and displayed on your monitor. Note that xclip
can pull stuff from different clipboards, and in my case the contents were not picked up with -selection default
but -selection clipboard
did the trick.
Linux (well, really the window managers, so X11 and then, i3, Wayland, etc…) have multiple clipboards. The default ones are the Primary selection one, and the Secondary one. The names are historical accidents, but the “primary” one always has a copy of the last text you selected from anywhere, which can be pasted anywhere by clicking the middle mouse button. You just select some text and that’s it - you don’t have to do anything else and you can then middle-click paste this anywhere. The “secondary” clipboard is the “normal” Cut, Copy, Paste, Ctrl+c, Ctrl+v one.
https://duncanlock.net/blog/2022/04/06/using-windows-after-15-years-on-linux/
There are many more window managers. Add more as I (re)discover them.
This could perhaps be a much more stable alternative to my reverse SSH tunnels for keeping connections with various servers, especially desktops inside FM-NET and other machines inside other LANs.
The main downside is that Tailscale is not FOSS. So perhaps we should look for other solutions built on WireGuard, or perhaps learn to configure WireGuard directly.
The command and its typical output (highlights not shown due to technical limitations in Markdown):
me@host:~/ansible/playbooks
$ find . -type f -name "playbook.log" -exec sh -c 'tac {} | grep -m 1 -A1 "^Playbook last committed by"' \; | grep --color -E "^|git/ansible/[A-Za-z]+?/[A-Za-z-]+?.yml|(19|20)[0-9][0-9]-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01]) [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}"
Playbook last committed by me@host on Fri Apr 17 22:02:15 2020 +0200 (afd13a3b3b3f43d3f84bb16b1c91a6b5bec2cfe1)
2020-04-19 00:40:30,925 p=32526 u=me n=ansible | task path: /home/me/ansible/playbooks/luxor/playbook-host.yml:99
Playbook last committed by me@host on Wed Jan 29 14:34:38 2020 +0100 (5157cd051e276abfe99e93c37a8ad0c79dd4d3dc)
2020-03-29 01:39:34,874 p=14553 u=me n=ansible | task path: /home/me/ansible/playbooks/damietta/playbook-heliopolis.yml:31
Playbook last committed by me@host on Tue Feb 18 17:30:19 2020 +0100 (ae6c02965f4471d8089c5e4d2a427cb0cbfbc6b8)
2020-02-23 19:58:30,188 p=1050 u=me n=ansible | task path: /home/me/ansible/playbooks/abydos/playbook-webserver.yml:35
Playbook last committed by me@host on Sun Jan 5 09:44:27 2020 +0100 (26392ab778deaf86430f36bc7aed942ae04a938c)
2020-01-08 13:26:35,647 p=me u=27195 | changed: [hunan.domain.se -> localhost] => {"changed": true, "cmd": "git log --pretty=\"Playbook last committed by %cn on %cd (%H)\" -1 >> playbook.log", "delta": "0:00:00.003510", "end": "2020-01-08 13:26:35.628135", "rc": 0, "start": "2020-01-08 13:26:35.624625", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
Playbook last committed by me@host on Thu Mar 26 14:57:00 2020 +0100 (2c9aa2030192c2942c5dfb0bcf5976f46fefd774)
2020-04-01 15:44:19,057 p=10821 u=me n=ansible | task path: /home/me/ansible/playbooks/alexandria/playbook.yml:137
The first find
command lists all playbook.log
files below the current directory (recursing into child directories). tac
is the opposite of cat
and lists each file backwards (from last line to first). We use grep
to look for a string ("Playbook last committed") that my Ansible playbooks always insert into the log-file at the end of a run. Note the use of the -A1
flag that gets the matched line and one line after (but because we used tac
, we actually get the line before, which is what we want). The final grep
uses extended regular expressions (-E
) to color highlight several parts of the output (while displaying all of the output, that's what the initial caret does - it effectively matches all lines).
Pretty neat, if I may say so myself.
Some of the refs I consulted to figure out this one-liner:
https://serverfault.com/questions/197123/getting-the-last-match-in-a-file-using-grep
https://unix.stackexchange.com/questions/112159/grep-from-the-end-of-a-file-to-the-beginning
https://stackoverflow.com/questions/307015/how-do-i-include-a-pipe-in-my-linux-find-exec-command
https://superuser.com/questions/914856/grep-display-all-output-but-highlight-search-matches
https://unix.stackexchange.com/questions/366/convince-grep-to-output-all-lines-not-just-those-with-matches
https://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns-with-pattern-having-a-pipe-character
https://en.wikipedia.org/wiki/Regular_expression#POSIX_extended