593 private links
Let's say you are looking for old Ansible code dealing with multiple distros (meaning multiple files in the ./vars/ directory) to draw inspiration from. How can we quickly identify the roles with vars directories containing more than a single file?
$ cd /media/bay/taha/projects/ansible && find . -not -path "*/archived/*" -not -path "*/testing/*" -path "*/vars/*" -printf "%h\n" | sort | uniq -d && cd $OLDPWD
./pub/roles/apache/vars
./pub/roles/common-systools/vars
./pub/roles/desktop-environment/vars
./pub/roles/digikam/vars
./pub/roles/graphics-driver-nvidia/vars
./pub/roles/php/vars
./pub/roles/php-versions/vars
./pub/roles/python2/vars
./pub/roles/python3/vars
./pub/roles/R/vars
./pub/roles/sioyek-pdf/vars
./pub/roles/wallabag/varsAll of the returned paths indeed contain at least two files.
Note how we limited find to only directories named vars.
This is a one-liner that I find useful when deciding a free IPv4 address to assign a new container:
$ lxc network list-leases lxdbr0 | grep STATIC | sort -t "." -n -k4
| taipei | 00:16:3e:9c:dd:28 | 10.252.116.3 | STATIC |
| hunan | 00:16:3e:f7:4f:b2 | 10.252.116.4 | STATIC |
| xian | 00:16:3e:b0:54:12 | 10.252.116.5 | STATIC |
| samarkand | 00:16:3e:e8:79:60 | 10.252.116.6 | STATIC |
| karachi | 00:16:3e:e5:4c:47 | 10.252.116.7 | STATIC |
| lahore | 00:16:3e:f1:73:ab | 10.252.116.8 | STATIC |
| muscat | 00:16:3e:9a:89:05 | 10.252.116.9 | STATIC |
| antioch | 00:16:3e:92:79:ca | 10.252.116.11 | STATIC |
| delhi | 00:16:3e:22:9c:d0 | 10.252.116.13 | STATIC |
| bahrain | 00:16:3e:4b:b6:ee | 10.252.116.14 | STATIC |
| lusail | 00:16:3e:fd:1f:49 | 10.252.116.15 | STATIC |
| pergamon | 00:16:3e:0e:9c:82 | 10.252.116.16 | STATIC |The one-liner is simplified by the fact that only the IP address field contains dots, so we can use them as delimiters for sort without first having to sort on the column delimiters.
This is an extension of the very common find ... -exec grep ... {} \; construct I use almost daily to find which files contain a particular text string.
Now, let's say you're looking for files in all your Ansible roles containing the string ppa:, because you want to create a new role using a suitable existing role as a template. In this case, I think most recently modified is an excellent proxy for suitability.
Thus, the challenge: can we tack on something to the find ... grep construct such that the output shows matching files in order of most recently modified?
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" | grep -v "^\\./.*" | sort && cd $OLDPWD
2021-06-09+22:27:20.6889730070 ./roles/public/php-versions/tasks/setup-Debian.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-02-23+15:33:13.2318546100 ./roles/dev/deluge/tasks/install.yml
2022-03-31+04:42:16.5644336650 ./roles/dev/editor-notepadqq/tasks/main.yml
2022-05-09+15:21:01.7974094830 ./roles/dev/qownnotes/tasks/main.yml
2022-06-25+02:46:13.0580097480 ./playbooks/workstation/roles/boot-grub/tasks/main.yml
2022-06-25+02:46:18.2940273650 ./roles/dev/libreoffice/tasks/main.yml
2022-07-03+00:16:44.5563131800 ./roles/dev/shutter/tasks/main.yml
2022-07-04+20:35:15.6128270470 ./roles/dev/magnus/tasks/main.yml
2022-07-14+21:09:01.6502524570 ./roles/dev/flatpak-remote/tasks/main.yml
2022-07-14+22:05:43.3325667290 ./roles/public/firejail/tasks/install.yml
2022-07-16+13:42:04.3756138100 ./roles/public/variety/tasks/main.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-07-21+00:14:51.1137716920 ./roles/public/foliate-ebookreader/tasks/ppa.yml
2022-07-21+03:07:42.2876610030 ./roles/public/graphics-driver-nvidia/tasks/install.yml
2022-07-21+06:05:27.4514643180 ./roles/public/foliate-ebookreader/tasks/flatpak.yml
2022-07-23+18:32:46.8466638700 ./roles/dev/handbrake/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-08-19+15:28:49.5605481840 ./roles/dev/x2goserver/tasks/main.yml
2022-11-04+11:35:14.6208169990 ./roles/public/python3/tasks/python-ppa.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2022-12-31+19:38:32.9105553030 ./roles/public/digikam/tasks/install-ppa.yml
2023-01-01+01:39:08.2045090970 ./roles/public/digikam/tasks/install-appimage.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
2023-01-26+14:10:18.9247087870 ./roles/public/ansible/tasks/main.yml
2023-01-26+16:14:59.9903243110 ./roles/public/sioyek-pdf/defaults/main.yml
2023-05-12+12:01:30.4705549280 ./roles/public/mpv/tasks/install.yml
2023-05-13+00:47:41.1561557100 ./roles/public/nextcloud-desktop/tasks/main.yml
2023-08-27+18:19:44.8291334420 ./roles/public/digikam/defaults/main.ymlEureka!
Explainer
- the initial
cd <path-parent>ensure that the resulting paths displayed byfinddon't contain the<path-parent>part (to avoid cluttering the output), and the finalcd $OLDPWDjust make sure that the bash prompt is not changed to<path-parent>. - unless you want to exclude some path from the search, there is obviously no need for
-not -path '*/<some-path>/*'. grep -ifor case insensitive matching, and-l(that's the letterlfor list) makesgrepprint only the filename and not each matching line (this is crucial for this hack to work, we wantgrepto produce as little output as possible, in fact, if I could figure out a way to silencegrepaltogether I would have, but I couldn't).-printf "%T+ %p\n"adds the filemtimeto the output (on a new line). Thanks angus@Unix.SE.
At this point, an example of the unfinished product is order. Before sorting, and before the final grep -v, the output looks like this (excerpt):
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
./roles/public/java-openjdk/defaults/main.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-firefox/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
./roles/public/browser-chromium/tasks/install.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.ymlwith the grep output on its own line, followed by the time-stamped output of printf. Like I said, it would have been better if we could somehow silence the grep output at this point. If you know a way, feel free to let me know!
As expected, sorting resulted in the non-time-stamped lines dangling about like some unwanted appendage:
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
./roles/public/java-openjdk/defaults/main.ymlAt this point I was out of ideas, so grep -v ... it was, and we end up with the one-liner shown above. It's an ugly hack, but hey, it works :-)
Bash 5.1 on Ubuntu 22.04.3, with GNU find 4.8.0, GNU grep 3.7, and GNU sort 8.32.
This is just a quick-and-dirty one-liner. I was compiling beamer slides, and for some reason the TikZ-generated PDF figures contained two pages under some circumstances. This way we can quickly see whether just some PDF files or all of them suffered a problem:
$ find figure/ -type f -name "*.pdf" -exec pdftk "{}" dump_data \; | grep NumberOfPages | awk '{print $2}'
1
1
1
1
1
1(in this case they were all one page long).
I wanted to create an ASCII art "bismillah" for use in the terminal.
Found this beautiful calligraphy by Nuria Garcia Masip. After cropping it, rotating it slightly and making the background monochromatic, I tried the ascii-image-converter by Zoraiz Hassan, and it worked great with the --braille flag:
This converter tool was really easy to install (just run the binary), and the output was automatically resized to fit the terminal window size, which was a convenient feature.
I discovered some other ASCII art CLI tools (never tested them though):
Recently installed a new webcam by my office computer. Not long after, a thought materialised: should it not be possible to view the video feed over a remote SSH tunnel?
Indeed, here's one way to do it (I have not bothered connecting to the camera's builtin microphone, but that should be possible):
taha@asks2:~
$ ssh mkem150 "ffmpeg -i /dev/video0 -c:v libx264 -f mpegts -" | mpv -
[file] Reading from stdin...
ffmpeg version 4.4.2-0ubuntu1~18.04.sav1.4 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version='0ubuntu1~18.04.sav1.4' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-crystalhd --enable-libmfx --enable-libsvtav1 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 47325.806919, bitrate: 442368 kb/s
Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1280x720, 442368 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x564e46b7b780] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x564e46b7b780] profile High 4:2:2, level 3.1, 4:2:2, 8-bit
Output #0, mpegts, to 'pipe:':
Metadata:
encoder : Lavf58.76.100
Stream #0:0: Video: h264, yuv422p(tv, progressive), 1280x720, q=2-31, 30 fps, 90k tbn
Metadata:
encoder : Lavc58.134.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
(+) Video --vid=1 (h264 1280x720 30.000fps)ime=00:00:00.90 bitrate=1983.6kbits/s speed=0.198x
VO: [gpu] 1280x720 yuv422pWorked without -c:v libx264, but noticeably poor video quality (pixelation and such).
Does not work if we remove the -f mpegts argument.
This is a neat way to quickly generate a QR-code, for example to transfer text from computer to phone (thanks to Solène Rapenne for the original idea and implementation). On the phone, you'll of course need to use a QR-reader app, such as Binary Eye.
xclip -o -selection clipboard | qrencode -o - -t PNG | feh -g 600x600 -Z -Using this command, whatever is in your clipboard will be encoded to QR and displayed on your monitor. Note that xclip can pull stuff from different clipboards, and in my case the contents were not picked up with -selection default but -selection clipboard did the trick.
Linux (well, really the window managers, so X11 and then, i3, Wayland, etc…) have multiple clipboards. The default ones are the Primary selection one, and the Secondary one. The names are historical accidents, but the “primary” one always has a copy of the last text you selected from anywhere, which can be pasted anywhere by clicking the middle mouse button. You just select some text and that’s it - you don’t have to do anything else and you can then middle-click paste this anywhere. The “secondary” clipboard is the “normal” Cut, Copy, Paste, Ctrl+c, Ctrl+v one.
https://duncanlock.net/blog/2022/04/06/using-windows-after-15-years-on-linux/
I recently got a copy of my "digital covid certificate". (You can get yours at covidbevis.se).
The certificate contains a rather dense-looking QR code, so naturally I was curious as to what data it contained.
With the help of Binary Eye, I could see that whatever the QR code encoded was not stored in clear-text.
Searching the web, I encountered this blog post by Austrian hacker Tobias Girstmair.
He has written a Python script that deconstructs the QR code into its data fields.
I installed it and used it to read out the values from a JPG screenshot of my QR code (I have randomised or hidden some data to protect privacy):
$ git clone https://git.gir.st/greenpass.git/
$ cd greenpass
$ python3.8 -m venv venv,
$ source venv/bin/activate
$ pip3 install flynn base45 PyPDF2 pyzbar Pillow
$ sudo apt install libzbar0
$ python3 greenpass.py myqrcode-screenshot.jpg
QR Code Issuer : SE
QR Code Expiry : 2021-09-29 15:00:00
QR Code Generated : 2021-07-01 15:00:00
Vaccination Group
Unique Certificate Identifier: UVCI : URN:UVCI:01:SE:EHM/V10050020P8X
Country of Vaccination : SE
Dose Number : 1
ISO8601 complete date: Date of Vaccination : 2021-06-01
Certificate Issuer : Swedish eHealth Agency
Marketing Authorization Holder : ORG-100030215
vaccine medicinal product : EU/1/20/1528
Total Series of Doses : 2
disease or agent targeted : 840539006
vaccine or prophylaxis : J07BX03
Date of birth : 1982-02-01
Surname(s), forename(s)
Surname : <>
Forename : <>
Standardised surname : <>
Standardised forename : <>
Schema version : 1.3.0The script worked when supplying a screenshot, but threw an error when I tried supplying the HC1 hash itself. In any case, I congratulate Tobias on a very nice piece of investigative work, allowing citizens all over the EU to inspect the data they share when using their COVID vaccination certificate.
The command and its typical output (highlights not shown due to technical limitations in Markdown):
me@host:~/ansible/playbooks
$ find . -type f -name "playbook.log" -exec sh -c 'tac {} | grep -m 1 -A1 "^Playbook last committed by"' \; | grep --color -E "^|git/ansible/[A-Za-z]+?/[A-Za-z-]+?.yml|(19|20)[0-9][0-9]-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01]) [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}"
Playbook last committed by me@host on Fri Apr 17 22:02:15 2020 +0200 (afd13a3b3b3f43d3f84bb16b1c91a6b5bec2cfe1)
2020-04-19 00:40:30,925 p=32526 u=me n=ansible | task path: /home/me/ansible/playbooks/luxor/playbook-host.yml:99
Playbook last committed by me@host on Wed Jan 29 14:34:38 2020 +0100 (5157cd051e276abfe99e93c37a8ad0c79dd4d3dc)
2020-03-29 01:39:34,874 p=14553 u=me n=ansible | task path: /home/me/ansible/playbooks/damietta/playbook-heliopolis.yml:31
Playbook last committed by me@host on Tue Feb 18 17:30:19 2020 +0100 (ae6c02965f4471d8089c5e4d2a427cb0cbfbc6b8)
2020-02-23 19:58:30,188 p=1050 u=me n=ansible | task path: /home/me/ansible/playbooks/abydos/playbook-webserver.yml:35
Playbook last committed by me@host on Sun Jan 5 09:44:27 2020 +0100 (26392ab778deaf86430f36bc7aed942ae04a938c)
2020-01-08 13:26:35,647 p=me u=27195 | changed: [hunan.domain.se -> localhost] => {"changed": true, "cmd": "git log --pretty=\"Playbook last committed by %cn on %cd (%H)\" -1 >> playbook.log", "delta": "0:00:00.003510", "end": "2020-01-08 13:26:35.628135", "rc": 0, "start": "2020-01-08 13:26:35.624625", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
Playbook last committed by me@host on Thu Mar 26 14:57:00 2020 +0100 (2c9aa2030192c2942c5dfb0bcf5976f46fefd774)
2020-04-01 15:44:19,057 p=10821 u=me n=ansible | task path: /home/me/ansible/playbooks/alexandria/playbook.yml:137The first find command lists all playbook.log files below the current directory (recursing into child directories). tac is the opposite of cat and lists each file backwards (from last line to first). We use grep to look for a string ("Playbook last committed") that my Ansible playbooks always insert into the log-file at the end of a run. Note the use of the -A1 flag that gets the matched line and one line after (but because we used tac, we actually get the line before, which is what we want). The final grep uses extended regular expressions (-E) to color highlight several parts of the output (while displaying all of the output, that's what the initial caret does - it effectively matches all lines).
Pretty neat, if I may say so myself.
Some of the refs I consulted to figure out this one-liner:
https://serverfault.com/questions/197123/getting-the-last-match-in-a-file-using-grep
https://unix.stackexchange.com/questions/112159/grep-from-the-end-of-a-file-to-the-beginning
https://stackoverflow.com/questions/307015/how-do-i-include-a-pipe-in-my-linux-find-exec-command
https://superuser.com/questions/914856/grep-display-all-output-but-highlight-search-matches
https://unix.stackexchange.com/questions/366/convince-grep-to-output-all-lines-not-just-those-with-matches
https://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns-with-pattern-having-a-pipe-character
https://en.wikipedia.org/wiki/Regular_expression#POSIX_extended