593 private links
I have published an Ansible role that installs a Jupyter instance using JupyterHub and JupyterLab.
- JupyterLab 3.0 released (2021-01-05)
- JupyterLab is ready for users (2018)
- How To Set Up a Jupyter Notebook to Run IPython on Ubuntu 16.04
Extending the Jupyter ecosystem
- Voilá
- Voici
- JupyterLab extensions (labextension)
- Classic Notebook extensions (nbextension)
- Notebook Server Extensions (serverextension)
- Jupyter Kernels
- IPython Magics
- IPython Widgets (ipywidgets)
- 99 ways to extend the Jupyter ecosystem
Hosted Jupyter notebook
- Colaboratory Google Jupyter notebook environment that requires no setup to use and runs entirely in Google's cloud.
Sharing data apps
- Streamlit - so like Shiny but for Python
RSS-Bridge
RSS-Bridge is a PHP project capable of generating RSS/Atom feeds for a multitude of sources. You self-host it on your own webserver or run it in stand-alone CLI mode.
Supports a long list of content sources (aka bridges).
- List of public RSS Bridge instances
RSSHub
RSSHub (over 17k stars on Github, wow!). It even has a browser extension and a mobile app. Docs.
Via reddit
"Kill the Newsletter!"
Kill the Newsletter!, by Leandro Facchinetti. Converts email newsletters into Atom feeds.
You can use the hosted service kindly offered by its creator, or self-host it yourself.
Via reddit.
FiveFilters
Web service that lets you create RSS feeds for any web sites that don't support it natively, among other useful content-related services. Offers a free tier, no account necessary. Paying customers also get access to the code and the ability to self-host the service (FiveFilters provide Puppet scripts).
FiveFilters maintains an open-source repo with site-specific content extraction rules that is widely used.
PolitePol
PolitePol is a web service where you can create RSS feed for any page on the internet using
interactive XPath selection. Create either XML or JSON feeds.
Service is not FOSS; pricing appears to depend on number of feeds. Unless you sign up the service is quite hobbled (feed lifetime 14 days, and feed never updates).
Feed43
Web service that lets you create RSS feeds for any web sites that don't support it natively. Offers a free plan, with or without first registering an account, but the code behind the service is proprietary.
RSS Box
RSS Box by Stefan Sundin, supports:
- Twitter (but Nitter may be better, also see this excellent guide by FiveFilters)
- Youtube (YT actually offers RSS feed per channel, also consider Invidious)
- Vimeo
- Instagram (another option is Bibliogram)
- Periscope
- SoundCloud
- Mixcloud
- Twitch
- Speedrun
- Dailymotion
- Imgur
- SVT Play
Feed me up, Scotty!
Feed me up, Scotty!, by Vincent Tunru. Hat-tip to Marie Dubremetz.
Notifier
Notifier, by Kirill Maltsev.
Can generate RSS feeds from emails and Telegram channels.
Offers up to 100 notifications per month for free. For 4 €/month you could receive up to 20,000 notifications.
Via kickscondor.
Where to find e-books
- Standard Ebooks, a curated collection of free and libre ebooks. Via reddit.
- Project Gutenberg, a large collection of free ebooks.
- Litteraturbanken, svenska klassiker. Via egen länknot.
- Z-Lib ,
z-lib.orgdomain seized, access via TOR or "personal domains" instead. - Library Genesis
- Anna's Archive, an open-source search engine for shadow libraries like Sci-Hub, Libgen, and Z-Library that aims to catalog all books in existence. Via TorrentFreak.
- The Internet Archive books collection. Very varying quality, some only for lending (via libraries), not download.
- Open Library, an open, editable library catalog by the Internet Archive. I'm not sure how it relates to IA's book collection.
- OpenStax, a growing collection of chemistry/physics/math textbooks made available under open licenses.
- World Bank's open knowledge repository, over 34000 publications.
- BookFinder, a search engine for new, used, and out-of-print books.
- AbeBooks, another search engine for used and out-of-print books.
- Helda Open Books, monographs from different disciplines written/edited by University of Helsinki faculty.
- OAPEN, Online library of open access academic books, operated by the National Library in The Hague.
- DOAB (Directory of Open Access Books), community-driven, operated by OAPEN.
Managing your ebook collection
- Calibre, the software for managing all aspects of your ebook library. Created by Kovid Goyal, maintained by hundreds of contributors. Project repo.
- For a full-fledged web interface to your Calibre library, look no further than Calibre-web. Just like Calibre's built-in server, Calibre-web also supports OPDS, which makes it easy to integrate with mobile apps such as Librera Reader.
- COPS, a Calibre PHP OPDS server. A less-powerful alternative to Calibre-web, in my opinion. But it served me well for some time before I discovered Calibre-web.
Tools for finding e-books
Reading e-books
Recently discovered Thorium Reader. Works very nicely on the Linux desktop and connects to our Calibre-Web OPDS server. My Ansible role.
Foliate is another nice desktop ebook reader. My Ansible role.
For reading on Android I use Librera Reader.
More links
- A not entirely bad guide to Calibre (desktop only) by MakeTechEasier.
- Library Athena, books from Project Gutenberg neatly displayed for reading in the browser.
- https://reddit.com/r/technology/comments/ynipok/pirated_ebook_site_zlibrary_vanishessending
- https://tenforward.social/@jeffcutsinger/113084913783829362
Any reversible physical process is an energy storage technology.
An interesting take from Technology Connections.
In this one, he reminds us that simple management of heat in our homes can amount to a simple and immediately available method to distribute energy use from the hours of peak demand.
Crooked Timber says: "energy storage is a solvable problem [...] concerns about the variability of wind and solar power will come to nothing in the end".
Moving rocks up and down former mine shafts could be a workable energy storage system.
The SI system
- A comprehensive (SI) units package for LaTeX, by Joseph Wright. I cannot recommend this package enough for all scientific or technical documents. CTAN.
Code examples
- Semiconductor pn-junction diagram in TikZ, by Erwann Fourmond (2016)
- Schematic TEM in TikZ, by Eric Jensen (2012)
- Oxidation of iron surface under a blob of water in TikZ, by Jason Waskiewicz (2010)
- Designing a business card in LaTeX, by Olivier Peters. Github repo.
Templates
- LaTeX packages for CVs, resumés
- PhD thesis template for Cambridge university Engineering dept. Supports LaTeX, XeLaTeX and LuaLaTeX.
Integration with R
The ability to integrate any kind of R output into our LaTeX document is very useful, and the technology has reached a high degree of sophistication with knitr and tikzDevice (on the R side) and TikZ/PGF (on the LaTeX side).
- TikZDevice, an R package for producing graphics output as PGF/TikZ code for use in TeX documents. In other words, we start in R, generate an image, and export it to a TikZ picture. tikzDevice was developed around 2009 (here's an early demo) in concert with pgfSweave (this was before knitr entered the scene).
- dvir, offers the inverse approach to tikzDevice: we start in R, generate a TikZ picture, convert that to DVI, then import (and integrate) the result back into R. dvir is less popular than tikzDevice, and is currently not listed on CRAN (in fact, it clashes with an existing completely unrelated package named
dvir), but appears to have solid technical underpinnings, with LuaTeX support and LaTeX math support.
Integration with Inkscape
- SVG2TikZ, formally known as Inkscape2TikZ, is an Inkscape extension for exporting SVG paths as TikZ/PGF code.
Integration with Microsoft PowerPoint
- IguanaTeX is a PowerPoint add-in which allows you to insert LaTeX equations into your PowerPoint presentation.
Integration with git
- gitinfo2-latexmk, small tool that integrates latexmk and gitinfo2.
Integration with Zotero
- Better BibTeX, a Zotero extension that makes Zotero effective for LaTeX users.
Integration with the web
- TeX commands available in MathJax, a comprehensive list of all commands available in MathJax v2.7.1, by Carol Burns and edited by MathJax creator Davide Cervone. MathJax v2.7 is still widely used, but has technically been superseded by v3. MathJax.
- KaTeX, developed by Khan Academy. Known to be faster than MathJax, but supporting a smaller subset of TeX/LaTeX functions. Differences between the projects appear to be shrinking, though.
Cloud-based LaTeX
Other packages
- Beamer, package for producing presentations and slides. Github.
- FiXme, collaborative annotation tool for LaTeX. Github.
Tips and tricks
Two wheels
Cargo bikes
- Xtracycle, extra sturdy electric bikes with longer-than-normal rear wheelbase to carry two children or lots of cargo.
- Convercycle, a bike that transforms to cargo bike, via Core77
- En informativ presentation av nätverket "Cykla med lastcykel"
Lastcykelguiden, en märkesoberoende guide till lastcyklar och lastkärror.
Peppe Hämeenniemis presentation om lastcyklar (PDF) från Cykelfrämjandets cykelträff aug 2022.
Three wheels or more
- Electric tricycle that leans into corners, a concept from BMW (not produced yet). Cargo area between the two rear wheels, and the front is able to tilt in curves. Chain-based drive mechanism that looks complicated.
Handla begagnade datorer, smartphones & annan IT-utrustning och bidra till en cirkulär ekonomi!
Svenska försäljare
- Godsinlösen Nordic AB (ReturHuset) har fysiska butiker i Staffanstorp och Skogås. Säljer begagnad elektronik.
- Inrego AB köper från företag, säljer begagnad elektronik till privatpersoner.
- Nordway Store AB, en kedja av fristående IT-återförsäljare, säljer begagnad elektronik. Har en LO-"branded" variant.
- Refurbed GmbH, "marknadsplats" som säljer begagnad elektronik. Företaget är baserat i Wien, Österrike.
- Refurbly AB, säljer begagnade mobiler.
- iXPress AB säljer begagnade iPhones, Samsung.
- MobilFynd, Ecommerce Sweden AB, säljer nya och begagnade telefoner från diverse märken.
- Nordic Teknik House AB säljer begagnade mobiler och surfplattor, men även annan elektronik.
- Kimbrer, köper begagnad IT-utrustning, främst från företag.
- IT-Auktion Nordic AB. Säljer ny och begagnad IT-utrustning. (Auktion, på alla vinnande bud tillkommer provision 17.5%). Tar endast emot produkter från företag (ej privatpersoner).
- Teknikdelar (Spares Europe), säljer bland annat begagnade mobiler (mest iPhones, några Samsung). Nyligen uppköpta av ClasOhlson.
International sellers
- eSolutions, sells phones (new or refurbished) preloaded with /e/OS (degoogled Android OS).
- https://buy.gazelle.com (most phones on offer appear to be carrier-locked)
Markdown
- JupyText: Jupyter notebooks as Markdown documents
- Generate Google Slides from Markdown
- CriticMarkup - plaintext editing markup for humans
R Markdown
- https://github.com/rstudio/rmarkdown
- R Markdown: The Definitive Guide by Yihui Xie, J. J. Allaire and Garrett Grolemund (2020)
- Authoring books and technical documents with R Markdown by Yihui Xie (2021)
- Create blogs and websites with R Markdown
- Writing reproducible geoscience papers using R Markdown, Docker, and Gitlab by Daniel Nüst, Vicky Steeves, Rémi Rampin, Markus Konkol, Edzer Pebesma (2018)
pandoc
- https://pandoc.org/extras.html
- https://github.com/LaurentRDC/pandoc-plot
- A pandoc-based layout workflow for scholarly journals by Piero (2018)
Quarto
Quarto, by RStudio, sorry, Posit, a new scientific and technical publishing system.
Something to consider, especially if you are new to the field and aren't already invested in the earlier systems.
Guides and how-tos
- Introduction to Ansible playbooks
- Playbooks vs roles, and explain the difference between task, role, play and playbook
- How to install and configure Ansible on Ubuntu 18.04 - DigitalOcean
- Linux hardening using idempotency with Ansible
Ansible playbooks and roles
- I haven't published all roles that I've written, but all the roles that I do publish are collected at codeberg.org/ansible
- Ansible module (Python code) to install R packages, by yutannihilation
- Ansible role to install TeXLive
- https://ansible.jeffgeerling.com/
- https://robertdebock.nl/ansible.html
Related
Screen capture is useful to create short instruction videos and the like. GIF is the most common output format. But GIF is an awful format. Consider first creating a normal video first, then using something like gifski to encode to GIF without losing too much quality.
- Peek, animated GIF recorder. Linux, FOSS (repo).
- Gifine, record from a directory of frames, or select a region of your desktop. Linux, FOSS (repo).
- SilentCast, silent mkv screencast and animated GIFs. Linux, FOSS (repo).
- Byzanz, part of GNOME. Linux, FOSS.
- ScreenToGif, screen, webcam and sketchboard recorder. Windows only, FOSS (repo, ghacks article)
- Cropper, screen capture utility. Windows only, FOSS.
- GifCam. Windows only. ghacks article.
- GiftedMotion (Java JAR, very old software, ghacks article)
- LICEcap. Windows, OSX and WINE. Github repo.
- On Mac OS X, there's a built-in QuickTime screen recorder.
- OS X screencast to animated GIF using only free tools, instructions here.
- GIF Brewery. MacOS, Mac App Store.
- Kazam, project seems a bit outdated. Recently suggested by Darren Goossens.
Similar lists by others
- ITA Matrix (aggregator, how-to and background)
- Skiplagged (aggregator, also searches for throwaway tickets) prices in USD
- Skyscanner (aggregator)
- Kayak (aggregator, part of Booking Holdings Inc.)
- Google Flights (aggregator)
- Momondo (aggregator, danskt bolag, sköts av Kayak som är en del av Booking Holdings Inc.)
- Expedia (aggregator/OTA) primarily for charter trips, enforces weird booking rules
- CheapOAir (OTA) prices in USD
- Booking.com (aggregator)
- Agoda (aggregator/OTA, part of Booking Holdings Inc.)
- Orbitz (OTA)
- Sembo (OTA, Helsingborg, Stena Line Travel Group AB) [SRF ✔]
- Allaflygbiljetter.se (aggregator)
- Flygresor.se (aggregator, Flightmate AB, part of Etraveli Group AB) [SRF ✔]
- SuperSaver (aggregator, Svenska Resegruppen AB, part of Etraveli Group AB) [SRF ✔]
- Travelstart (aggregator, Svenska Resegruppen AB, part of Etraveli Group AB) [SRF ✔]
- MyTrip (aggregator, Svenska Resegruppen AB, part of Etraveli Group AB) [SRF ✔]
- Seat24 (aggregator, Svenska Resegruppen AB, part of Etraveli Group AB) [SRF ✔]
- Ticket.se (OTA, ägs av Braganza) [SRF ✔]
- Travellink (OTA, ägs av https://edreams.com)
- JetRadar (aggregator) prices in USD/EUR
- Hotwire (OTA) prices in USD
- Travelocity (OTA) priser i USD
- SecretFlying
- BilligaFlygbiljetter.se (aggregator, ägs av Vuelos Baratos Ltd.) unusally poor UX website
- CitizenPlane (a sort of aggregator, not an OTA) ägs av franskt bolag, en société par actions simplifiée. Should probably be avoided.
You can also book tickets directly from many airlines.
Airlines
- Pegasus, Turkish airline, based in Istanbul SAW
- WizzAir, based in Hungary (but their app is horrible, 118 MB, also requests superuser rights?!)
Review articles
https://upgradedpoints.com/best-websites-for-booking-cheap-flights
https://www.frommers.com/slideshows/848046-the-10-best-and-worst-airfare-search-sites
https://www.topfivebuzz.com/20-best-flight-search-engines-for-international-flights-to-find-cheapest-airfares
https://www.forbes.com/sites/johnnyjet/2017/10/04/5-best-online-tools-for-searching-flights
An aggregator is only as good as the OTAs (online travel agencies) it canvasses. There are booking engines that find prices themselves (Expedia, Priceline, Hotwire). And then there are aggregators—sites that do not book tickets but instead search dozens of booking engines, airfare sites, and OTAs and compile the results in one place; you then click through to the one of your choice to make the actual purchase.
https://www.frommers.com/slideshows/848046-the-10-best-and-worst-airfare-search-sites
More links
- Svenska Resebyrå- och arrangörsföreningen (SRF), medlemsförteckning
- SVT artikel om Etraveli Group AB
- Life of an Airline Flight: What Systems Get You From Here to There via the Air, youtube video of talk by Adam Fletcher (2019)
With an eye towards travel agencies that could be useful for our family travel.
Also see my note on OTAs and aggregators.
- Tripp resebyrå Jönköping
- Nygren & Lind resebyrå Surbrunnsgatan 6, Stockholm
- Lingmerths resebyrå (Uppsala universitet använder Lingmerths)
- Resia Vi skräddarsyr resan efter dina behov och erbjuder allt från weekendresor och paketresor med flyg och hotell till sista minuten.
- BigTravel, fullservice resebyrå med 35 års erfarenhet av affärs-, grupp- och konferensresor. Hjälper många små och medelstora företag med deras affärs- och konferensresor.
UD:s "Svensklistan": här kan du lämna uppgifter om var du befinner dig och hur du kan kontaktas under din utlandsvistelse. Uppgifterna kan vid behov användas av UD eller ambassaden för att kontakta dig om en konsulär krissituation skulle inträffa.
IUPAC Periodic Table Challenge
A quiz on the chemical elements with different levels.
Requires registration, but you learn a lot from the questions.
The Periodic Dinner Table, is a cross between a demo lecture, comedy sketch and a game of bingo played on the periodic table (video with additional resources).
General Julia resources
https://en.wikipedia.org/wiki/Julia_%28programming_language%29
https://old.reddit.com/r/Julia/
https://github.com/JuliaLang/
Scattering, which seems it could replace diffractometry in R
Plus, they have extensively documented its development in a series of blog posts:
http://www.yxliu.group/2020/03/scattering-1
http://www.yxliu.group/2020/03/scattering-2
http://www.yxliu.group/2020/04/scattering-3
http://www.yxliu.group/2020/04/scattering-4
http://www.yxliu.group/2020/04/rotation-test
http://www.yxliu.group/2020/04/scattering-5
http://www.yxliu.group/2020/04/scattering-6
Differential equation solver
- In 2013 Google killed Google Reader
- Firefox removed feed discovery features in 2019 (available as plugins now, for example Awesome RSS)
- I've written a linknote on how to subscribe to Youtube channels using RSS
- About Feeds, a primer on web feeds/RSS by Matt Webb.
- You Need Feeds, quickly explains what RSS is and how to use feeds. Via @killyourfm@layer8.space.
- Vad är RSS?, bra intro till RSS av Kamratdataföreningen Konstellationen.
- Youtube channel RSS feeds
- Github RSS feeds
- https://feedcanary.com via https://kevquirk.com/dont-change-your-rss-url
- https://andregarzia.com/2024/05/feed-and-blogrolls-discovery.html
- https://defaults.rknight.me
blog posts
- p1k3, 2020-05-08
- Staying informed without big tech, 2021-07-03, The New Oil
- How to read RSS in 2020, 2020-03-24, Laura Kalbag
- A thorough comparison of three self-hosted RSS readers, 2020-05-31, Luke Singham
- Felicia Day's amazing rant about RSS, 2020-12-16
- Why I still use RSS, 2021-03-03, Marc @0066cc
- Robin Rendle writes about how RSS is the "promised land", via Roy Tang
- How to Subscribe to or Read RSS or ATOM News Feeds, Christopher Heng
- What is RSS?, 2021-09-11, Ali Reza Hayati
- What using RSS feeds feels like, 2022-02-13, Ben Werdmüller
- Styling OPML and RSS with XSLT to XHTML, 2023-06-22, Rubenerd. Something to consider for my own blog.
- RSS for Post-Twitter News and Web Monitoring, 2023-07-06, ResearchBuzz (Tara Calishain). A long, very informative article on RSS and how to use it.
- A well curated RSS feed reader is a wonderful substitute for social media
feed readers
- Tiny Tiny RSS, this is what I've been using for years. Android app(s) on F-Droid, and there's at least one iOS app.
- Miniflux
- Vienna RSS
- FeedReader, a desktop client.
- Fresh RSS
- RSS Guard desktop RSS reader for Linux, BSD, OS/2, macOS, or Windows that supports RSS/ATOM/JSON/iCalendar/Sitemap feeds and also web-based services such as Feedly, Nextcloud News and TinyTinyRSS.
- Winds
- Stringer
- Selfoss
- feedbin
- Leed
- commafeed
- Feedbro is a feature-rich browser plugin feed reader, but it's not open source, so not recommended.
Note: this list is not exhaustive. If you know of a feed reader I've missed, let me know and I'll add it.
tools that make it easier to add sources to your feed reader
- FiveFilters
- https://rssgizmos.com (including feed freshness checker, OPML maker, and OPML peeler).
- Feedsearch, an API service for searching websites for their RSS, Atom or JSON feeds.
- GetRSSFeed. Shows RSS feed URL for Soundcloud or Apple "podcasts".
- FetchRSS. Works well for those Soundcloud "podcasts", but this site requires registration (which suggests that this site is creating their own feed endpoint instead of simply exposing soundcloud's already existing (but well-hidden) RSS endpoint).
- rss discovery engine, pre-alpha, by Brandon Quakkelaar (via Rubenerd)
- https://feedle.world a search engine for blogs and podcasts
- Explore RSS feeds in your own neighbourhood (interesting project)
Is the podcast you want to listen to hosted on Soundcloud? First of all, double-check that the podcast is not also hosted somewhere else (more often than not that is the case, and then you can simply avoid Soundcloud).
Soundcloud makes it really hard to discover the podcast's RSS feed. It is not shown anywhere on the page, nor in the HTML source.
Based on a feed URL we already possess and that channel's Soundcloud landing page, I think the following method is necessary to manually construct the hidden RSS feed endpoint for any Soundcloud "podcast":
- Open the HTML source of the Soundcloud "podcast", for example
https://soundcloud.com/user-268302561. - In the HTML source, find the user ID property, which looks like this
soundcloud://users:819898639(I suggest search forsoundcloud://users). - Copy the numeric part, and construct a URL like this:
https://feeds.soundcloud.com/users/soundcloud:users:819898639/sounds.rss.
Take a moment to reflect on the awfulness of siloed web services and do your best to avoid encouraging them in future.
Cloudflare is user-hostile and site owners using it should beware
I occasionally encounter feeds that my feed reader cannot subscribe to because the site uses Cloudflare DDoS protection, which Cloudflare implements in a manner contrary to the ethics of the open web, in the process making it near impossible for feed readers or read-it-later services to access the content.
I have no simple work-around for such RSS feeds - they are effectively rendered useless by Cloudflare's discriminatory and user-hostile blocking
implementation.
Cloudflare themselves claim doing nothing wrong, and that it is in fact the site operators that have misconfigured their Cloudflare firewall and that users should contact the site operator. Which is simply ridiculous.
If Cloudflare cared at all about the open web, this issue would not be relegated to a few obscure forum posts. They could also easily implement some form of automatic exemption from their page blocking for common feed endpoints, such as /feed, rss.xml and similar.
- Cloudflare considered harmful, 2019-10-23, Hugo Landau
- https://git.nixnet.services/you/stop_cloudflare
- Stay away from Cloudflare, 2017-12-20, unixsheikh.com
- https://news.ycombinator.com/item?id=12646055 (thread on the subject of Cloudflare and RSS)
- https://reddit.com/r/selfhosted/comments/qars38/rss_feed_behind_cloudflare_protection/
- https://stackoverflow.com/questions/11886711/curl-cant-fetch-rss-from-website-because-of-cloudflare
- https://github.com/VeNoMouS/cloudscraper (a work-around in Python, but I don't know how to incorporate with TinyTinyRSS...)
-
Telecommunications metadata
Your ISP learns every website you visit via a few different channels: DNS lookups, the IP address of sites, and TLS Server Name Indication (SNI). Most ISPs log and store this metadata for some time. Mitigation There's no need to use your ISP's DNS server. Run your own. You can't really getting around your ISP learning the IP addresses you visit. But with TLS encryption and the pervasive use of content distribution networks (CDN) by many website providers, the IP address itself does not really say much. The "leaking" of domain names via SNI is harder to get around, but newer versions of TLS improve on the situation. -
Web Site Data
Any Web site that you go to is very likely to keep extensive logs of everything you do on the site, including what pages you visit and what links you click. They may also record what outgoing links you click. For example, Google Search does this. Mitigation Try to use alternative services, that don't track you. Such as Nitter instead of Twitter, Invidious instead of Youtube, DuckDuckGo instead of Google Search, Bibliogram or PixelFed instead of Instagram, Jitsi instead of Zoom, Signal instead of WhatsApp, and so on. -
Browser Sync Data
Although the browsing history stored on your computer may not be directly accessible, many browsers offer a “sync” feature which lets you share history, bookmarks, passwords, etc. between browser instances (such as between your phone and your laptop). This information has to be stored on a server somewhere and so is potentially accessible. Mitigation Make sure that whoever hosts this server allows you to properly encrypt your data (in such a way that the server owner can never decrypt). Or even better, self-host the sync server yourself (Firefox sync can be self-hosted in this manner).
My ideal personal knowledge system would entail a rofi-like search from the desktop that ties into my knowledge repos, such as Zotero, this linkblog, my blog, Wallabag, Calibre server, and more. Even better if the search was powered by a self-hosted LLM. A tall order, to be sure ;-)
Here I'm collecting notes on projects or posts that are along the lines of this vision.
TiddlyWiki (Ness Labs)
TiddlyWiki is (much more than) an open source alternative to Roam Research. You can self-host it. Your entire Wiki (including all notes) is just one HTML file (if you want, it could also be separate markdown files), so sharing should be easy.
You need to choose a method for saving your changes. Many options to choose from, including Node.js, TiddlyDesktop, Beaker browser, WebDAV, TiddlyFox (not so much), or Git.
Via Tom Critchlow
Another good post by Le Cunff: open source and self-hosted alternatives to Roam
There's a TiddlyWiki subreddit and a TiddlyWiki Google group.
A reddit post listing open source alternatives to Roam Research.
A long discussion on HN on pros and cons of Roam based on Roam: Why I Love It and How I Use It by Nat Eliason.
And a post by Jack Baty comparing TiddlyWiki and Roam.
Another blog post, Tiddlywiki for organizing notes and research.
An introduction to TiddlyWiki by Scott Nesbitt.
TiddlyBlink is an adaptation of TiddlyWiki, inspired by Roam's bi-directional links.
For the knowledge graph view, there's TiddlyMap.
- TiddlyMapProject video showing how to setup TiddlyMap on TiddlyWiki
https://news.ycombinator.com/item?id=22104366
I tried it out in Beaker browser. Worked well (TiddlyWiki, Markdown plugin). But I could not install TiddlyBlink, for some reason. Perhaps try a different browser?
There's also TiddlyRoam, which combines TiddlyMap and TiddlyBlink. The announcement reddit post, and source on github.
Roam (and by extension, TiddlyRoam) is a form of Zettelkasten.
You can apply different Tiddly themes (here's one).
Using TiddlyRoam with TiddlyDesktop, I still cannot figure out how to install plugins. If I use the built-in plugin library, and install e.g., Markdown parser, I get prompted to save and reload, do so, and then... nothing. Plugin is not installed, and the Wiki content starts over from scratch, whether I create a new HTML file or replace the existing one. What's up with that? A few edits later, the plugin now works as expected. Maybe it just took some time to apply, or maybe I restarted TiddlyDesktop (not sure).
The TiddlyMap page has an example of mapping a literature collection (using icons set with group styles).
Foam
Foam is another FOSS alternative to Roam.
It's built into VSCode/VSCodium, which is a big selling point for me.
Logseq
Logseq. FOSS.
Emacs org-mode
There's also org-roam (built on Emacs org-mode)
- Video on org-roam setup from Matt Williams
- Visualise Org-Roam Networks With igraph and R, by Peter Prevos, 2021-08-14
Popup IndieWebCamp Session 2020 (video recording of Zoom session), on wikis, blogs and such. Via kickscondor.com.
Obsidian
Obsidian is not libre (not open source) software, but provided for free for personal use. For that reason, although popular, I cannot recommend it.
Also see this blog post on how to create nice tables in Obsidian with plugins.
- https://github.com/anoopkcn/obsidian-reference-map
- https://thesiswhisperer.com/2023/02/01/building-a-second-brain-for-writing-with-obsidian/
ZotHero
Something like ZotHero but for rofi/Linux would be really cool. Too bad ZotHero is only for Alfred (MacOS).
Assorted links
- Knowledge repository, post in the Level1techs forum
- Second Brain Software: From Files to ZettelKasten, Level1Techs, video 22 min, 2022-02-24
- Dreaming of a free mind with tiddlywiki+ freemind mindmaps
- FreeMind to TiddlyWiki integration
- https://github.com/michaeljmcd/mm2tiddlywikitext
- https://github.com/byzheng/rtiddlywiki
- https://github.com/raphaelsty/knowledge (web app that creates your own knowledge graph based on your Github stars, Twitter likes, HN upvotes and Zotero database).
Some guides on the why's and what's of password managers
- The best password manager in 2021, The New Oil
- A technical introduction to password managers, Tavis Ormandy. Via Alan Ralph.
- Wikipedia has an informative article on password managers.
- Internetstiftelsen har publicerat en utmärkt guide för hjälpa dig att komma igång med lösenordshanterare.
- Bra råd hos Myndigheten för samhällsskydd och beredskap (MSB) om hur du säkrar din lösenord.
- https://reddit.com/r/sweden/comments/1abpovo/%C3%A4r_det_inte_riskabelt_att_ha_l%C3%B6senordshanterare
BitWarden
Open source password management solution, can be self-hosted (how-to)
Integrates with Ansible.
1Password
A cloud-hosted password manager. No longer recommended, as it has regressed into an Electron application mandating remote storage of your password vault.
KeePass
Free, open source, light-weight, and easy-to-use password manager.
These slides discuss the why? how? what? where? of KeePass (presented by F. Morsbach at the first CryptoParty in Uppsala, 2019)
KeePass is written in C# and therefore requires Microsoft's .NET platform.
- Plugins: A list of third-party plugins for KeePass
- Android: Keepass2Android
- iPhone: MiniKeePass
- Chrome / Firefox: Tusk
- Web App: KeeWeb
KeePassXC
Open source, cross-platform.
KeePassXC is developed in C++ and runs natively on all platforms.
- Easy to achieve cloud sync by simply storing your KeePassXC database inside your shared cloud folder (using Nextcloud, for example).
AuthPass
AuthPass supports the KeePass database (KDBX format).
Setup instructions.
Pass
Simple GPG/Git password manager. Adheres to the Unix philosophy, which makes it easy to integrate with anything, e.g., Ansible.
This is my password manager of choice :-)
- Excellent in combination with rofi or dmenu on the Linux desktop, and
- on Android, useable with a combination of Password Store and OpenKeychain.
- Rich ecosystem of plugins, tools: pass-rotate.
- Clever uses of pass, by Vikaly Parnas.
Dashlane
"An intuitive password manager with over with over 8 million users worldwide."
Not open source, as far as I know.
Passbolt
Free, open source, self-hosted, extensible, OpenPGP based.
LessPass
Stateless, open source password manager.
Psono
Open source and self-hosted password manager for teams.
Buttercup
Another open source password manager with desktop, mobile, and browser clients.
LastPass
One of the most widely-used password managers, but that does not mean it's the best. Parent company is LogMeIn. Completely cloud-based, and not open source, obviously. On that basis alone, I do not recommend it. LastPass has suffered a major breach, it is now effectively dead.
Password manager apps that integrate with Nextcloud?
More lists of password managers
- https://codeberg.org/alicia/awesome-privacy#password-managers
- https://nextcloud.com/blog/password-managers-for-nextcloud/
- https://github.com/Igglybuff/awesome-piracy#password-vaults
- https://protonmail.com/blog/open-source-password-managers/
- https://www.tomshardware.com/best-picks/best-password-managers
- https://european-alternatives.eu/category/password-managers
I have Zotero installed on three workstations and one laptop. This informs many of the choices I've made when configuring Zotero.
Manual first-time configuration of Zotero
- Install the LibreOffice plugin
- install the Firefox connector
- install the ZotFile plugin and configure its settings (see below for details)
- go into
Preferences -> Advanced -> Config editorand togglezotfile.pdfOutline.getTocfromTRUEtoFALSE(important to not break sync) - go ahead and sync.
The Zotero Connector for Firefox handles proxying nicely (just remember to enable it, for Uppsala university use %h.ezproxy.its.uu.se/%p) and makes the Firefox add-on EZProxy Redirect Foxified redundant.
Storage of PDFs when using Zotero from two or more computers (ZotFile plugin)
Zotero's storage folder for PDFs (Zotero calls them "attachments") is set in
Zotero -> Preferences -> Advanced -> Files and Folders -> Data Directory Location (on Linux, the default location is ~/Zotero).
This is a very inefficient approach when using Zotero from two or more computers, since the stored PDFs and other attachments can easily occupy tens of gigabytes which will thus occupy that space on each and every computer.
A much better solution would be to move the Zotero attachments directory to a remote filesystem that is accessible from all your computers. Zotero itself does not support this use-case, though. This is exactly the premise of the popular ZotFile plugin.
I rely on it on every day - once configured, it has worked flawlessly for me for years.
- http://zotfile.com/
- https://github.com/jlegewie/zotfile
- http://islamicate-dh.github.io/2016-05-27-set-up-zotero-between-multiple-computers/
- http://geekandgirliestuff.blogspot.com/2014/10/new-computer-fresh-zotero-install.html
Download the latest version of the ZotFile XPI, and install it in Zotero: Tools -> Add-ons Manager -> Install Add-on from File. Restart Zotero.
Make sure to uncheck the following choices under File Syncing (since ZotFile takes care of it now):
- Zotero
Preferences -> Sync -> Settings -> File Syncing -> Sync attachment files in My Library - Zotero
Preferences -> Sync -> Settings -> File Syncing -> Sync attachment files in group libraries using Zotero storage
Configure your Zotero "attachments" directory with ZotFile
- In Zotero,
Tools -> ZotFile Preferences -> General Settings -> Location of Files -> Custom Locationand set it to the root of your PDF directory (in my case,/media/bay/taha/chepec/literature/zotero). - Set your naming scheme for subfolders in
Use subfolder defined by. - I suggest you enable
Tools -> ZotFile Preferences -> Advanced Settings -> [X] Remove special chars from filename. - Set ZotFile to always rename new PDFs:
Tools -> ZotFile Preferences -> Advanced Settings -> Automatically rename new attachmentsby setting it toAlways rename.
In Zotero's own preferences, go to Advanced and set your Base directory to the same directory used for your ZotFile custom location.
If you already have items in your Zotero library, highlight all of them in Zotero's library, and select Manage Attachments -> Rename attachments. ZotFile should do its magic and rename and relocate your entire library to the directory you have configured.
Once you have done this on one computer, you then need to repeat it over again for you other computers. But a faster way is to simply copy the entire Zotero profile directory from your already-configured computer and overwrite the same folder on your other computer - and presto, all your settings are immediately in place. But beware that any Zotero configuration you had in place will be wiped out by such an operation. You might want to make a backup copy of your original Zotero profile directory first.
Known issues
Zotero Firefox Connector fails to import anything into Zotero
Seems something broke with v5.0.101. Downgrading to v5.0.100 has resolved the problem for now.
- https://forums.zotero.org/discussion/comment/421625
- https://www.zotero.org/support/troubleshooting_translator_issues
- https://www.zotero.org/support/kb/connector_zotero_unavailable
- https://www.zotero.org/support/translators
Sync breaks with large notes (>250 kB)
If you have large documents with long and complex table of contents, you might observe Zotero sync errors: The note "Contents" is too long to sync. Shorten the note and sync again.
This is because ZotFile tries to extract the TOC into a Zotero note during sync, and runs into Zotero's limit of 250 kB per note. The fix is to disable TOC extraction: Preferences -> Advanced -> Config editor and toggle zotfile.pdfOutline.getToc from TRUE to FALSE.
- https://github.com/jlegewie/zotfile/issues/394
- https://forums.zotero.org/discussion/70958/the-note-is-too-long-to-sync-shorten-the-note-and-sync-again
- https://forums.zotero.org/discussion/comment/295634#Comment_295634
Not all PDFs sync properly between your Zotero instances
It seems the issue lies in the Renaming. A number of PDFs are in fact still residing in the Zotero (not ZotFile) data directory, and those are thus not available on our second instance.
The issue is resolved by manually issuing the Rename command on that file/document.