vanderwal.net

My Automations Over the Ages and Today’s View

For a few decades I’ve converted regular manual tasks into automated tasks for many things. I use short code keyboard conversions with TextExpander (TextExpander), Alfred (Alfred), Keyboard Maestro (Keyboard Maestro), Hazel (Hazel by Noodlesoft), Apple Shortcuts, and a couple other small apps and services. But, I also have quite a few scripts in python and Ruby, but a few pure bash scripts. I have a handful of apps that use Claude Cowork (Claude Cowork | Claude by Anthropic) on a schedule and used where the data is not always standard, so it aggregates and structures the data where it often later gets processed by python.

I started building and running automation scripts in the 1990s to handle tedious tasks with multiple steps, or would likely be victims for typos, and which I was handling daily. A task that would take 3 or more minutes (often longer) and had structured stable data I could search made it viable to become a script and automated. I had them to filter email to folders, filter downloads to common folders / directories, and one to turn off audio for the evening which monitored Twitter every 30 minutes for my nightly signoff (“off to read and sleep - zzzz”) and it would mute audio. I’ve had a few for audio that would capture podcasts I was on for a local copy during recording that took just one button click to set Audio Hijack Pro properly and it just worked.

Moving to the Current Adaptive Flows

The last six months (or a little more) I’ve been building aggregators to capture what I’ve done the day before (and the week prior for a manual review). I’ve also automated my morning Obsidian (Obsidian) setup, so that my daily note (or Daily Dump for my notes, in my vocabulary) templates are created for the week ahead with a python script, rather than creating a new note and use a TextExpander snippet to build the structure each morning. But, doing this I realized that I could take my daily task list and the list of potential tasks I keep at the top of my Daily Dump and script the moving uncompleted items to that day’s Daily Dump from the day prior, but also quickly run a calendar scan for the day with Claude and the BusyCal MCP to have my known items at the top of the note written directly in the note. The completed items and checked off were left on the prior day’s Daily Dump in their sections.

In order to run the scheduled python scripts I need my laptop to be awake in the background (but it still the screen remain locked) so to run early morning tasks. I found PMSET as the key to handle the wake-up for the first run and have a python script handle the remaining wake-up processes for that day.

I now have a mix of Claude Cowork Skills (which if I sort out the data is stable enough I convert them to python for consistency, speed, and sustainability) and python scripts that pull my prior day’s files created and worked on list, my items in a feed of the prior day’s additions to Instapaper, Pinboard, and a list of new items in DevonThink as PDF (with links to the PDF as well as a working URL to the source of the item), a list of Readwise articles and video transcripts, and videos I liked and sent to my personal instance of Slack. These lists are in markdown and placed into the directory Obsidian watches over. Most of them a little later in the morning are transcluded into my Daily Dump note for the day where I read through them.

Morning Process

My morning process is to read through what was done the day before and edit them to add links to other resources in Obsidian and write notes. Often I have some entries that will turn into that day’s list of things to do or one of the “potentials” list. I will take items that need to be their own item note or be included into a concept note. This keeps the items fresh on my brain and connecting things to where I can get back to them more easily.

I write this today, as this morning I found my morning automation hadn’t fully run properly, because my Sunday evening script to create the week’s Daily Dump notes didn’t run as it was a cronjob and I was cooking with the laptop asleep so the job didn’t kick off. This morning I converted the cronjob to a launchd task instead and to run on Sunday during another automated wake open window.

Weekly Aggregation

Sunday’s I also run a weekly wrap-up, that pulls everything done for the week. I’m finding this is a bit too much information for a quick review and a manual assessment for the week where I aim to map out things for the upcoming week. It is a bit too much information overload. I don’t know what I’m going to do about it. Now that I have things automated and aggregated I need to rework my human side of things.

The Value of the Automation

The goal of my automations is to give me time back and remove the manual and grunt work so I can do the thinking work. I’ve done this with personal life and work life (and large systems and projects). The current round of automation and aggregation surfaced that I am doing more than I thought I was doing. Another goal is to give more time to do the human thinking and the projects and work needed. I do this so I can do the creative human work and the human thinking work (I find large errors in summarization in nearly all generative AI and creative work so I keep that at bey and use it for grunt work that I check). The scripted automations don’t need the hand holding and I have some occasional test run to check on them to see if they need attention and honing.

Weeknote Considerations

One of the things I deeply enjoy reading each week is other people’s weeknotes. Well, not everybodies weeknotes. I have 4 to 8 different people who share weeknotes that I check in on. It is like getting the weekly catch-up and it is calm and quaint (in the good way). I really like them and I get concerned if they miss a week.

For a while I was posting weeknotes (found with the tag: Weeknote :: Off the Top :: vanderwal.net). I posted 25 of them. They were recaps of things I found of interest and wanted to share in that prior week (or so). I stopped posting them as the timing was tough at the end of the week, but the creation was getting easier as they just were things I’d pull out of my daily notes to myself in roughly the same categories.

Not Sure If I’ll Return to Posting Weeknotes Again

I am really not sure that I will return to posting weeknotes again, but it is a rather Ouroboros-shaped reason. In the past year or so I set up local (on my laptop) search of my own blog, which moved to the Lab - vanderwal.net, and then on to my blog portion of my site.

In searching my own content to see if I’ve posted on a subject before, I would find what I was looking for in a weeknote post. But, going to the post it was buried in the muddle with other things. I also was finding using categories to find related content when it was surfacing weeknotes and I’d have to dig in the weeknote to find the relevance it was connecting to. Both of these bothered me and from a finding, pointing to, and sharing with others perspective. I really think it is a less than good experience.

The weeknote for me was a means to get back to posting to my blog more regularly. I joked that Twitter ate my blog in 2006 (I’ve seen my data and it sure did). Then moving to a better note tool and note making daily that had me writing again in pretty much my old manner like blogging, but I just wasn’t posting. I didn’t complete the workflow for it to be a blog post. I would write and reuse locally, but the next steps in a blogging / sharing workflow did’t happen. A little over a year ago when I updated my blog management tool, that was part of the backend update, that made posting easier and I was posting a little more. But, when I’d look it was a week or more stretches between posts and I’d been writing like I was blogging, but I was my own reader.

Using Blaugust (Going in on Blaugust :: Off the Top :: vanderwal.net) as impetus to get the workflow back part of my habit of finishing the steps is a good thing for posting, I think.

My Notes to Post Workflow

I’ve kept notes in text in a directory since 2003. It started as plain text along with dash or asterisk bullets. That turned into using Notional and then NVAlt at some point. When NVAlt stopped opening I set Obsidian to watch over the directory in June 2020 and I’ve not looked back. I did a quick modification and the notes that weren’t in markdown (I fully shifted my writing to markdown in 2011 when I had a hard drive crash and access to application that wrapped notes made my notes something I couldn’t open for a few days and with my parents passing and needing notes, a proprietary container was no longer an option – actually it is a bad idea from the start).

I keep a Daily Dump markdown note that is structured with sub-headings as prompts and keep notes in there. One of the things I love about Obsidian is the ability to have transclusions, so when I am writing in my daily note, but it makes sense to have it as a stand alone note to reference, or to become a blog post, I just highlight the segment and right click to export that to a new note and it links to with a backlink (think wiki link), but putting an exclamation mark in front of it brings that note into the daily note to show it.

This means I can create a note that has a “blog-shape” to it that started in my daily note. It is in markdown and I have a title on it and other metadata. It lives in my note directory. I click the ellipses in Obsidian’s right corner of the panel to “Show in Finder”. The finder window for Mac opens with the markdown note highlighted. I right click on it (I don’t use a mouse, so double finger tap is actually what I do) and select to open the file in iA Writer. iA Writer is my longer writing app, but also what I review and edit something that I will post (or transform to a more formal document). When that is done I drag the proxy icon for the file in the titlebar of iA Writer to the icon for the Marked app (Marked 3) in my Dock and it opens and I select “preview toggle” and it shows me very well structured HTML (it matches the scripting I did years ago to do much the same, but is much less hassle) then I highlight and copy and paste into my site’s blog management tool, add the title and select categories and it is posted.

The multiple apps all are working with regular markdown all in the same file. I modify something in one and it changes in all the other interfaces. I own the files and they can be opened in any text app or any markdown focussed app. IA Writer I really like for calm focussed writing, reading what I’ve written, and editing. Marked (now Marked 3) just does a nice markdown conversion to very good standards compliant HTML, plus adds the sub-header IDs for anchor links, which I used to do mostly by hand when I ran my own convert to HTML scripts. Marked also can transform markdown to structured Word, well designed PDF, and more, yet I use it for the one simple task.

In my blog management screen, I manually trigger the build of my RSS and then trigger syndication notifications after I have run a manual visual check and test all the links. Once I’ve finished writing it is 2 to 5 minutes to edit in iA Writer and post.

My Writing Process with Obsidian

A few times lately my writing process has come up as it relies heavily on markdown for portability, longevity access, flexibility, and using the Shift Happened – Part 2: Small Apps Loosely Joined – Personal InfoCloud. The Small Apps Loosely Joined concept is something I use heavily for many things, but for writing the use of markdown I use the concept quite heavily.

Recently in a IndieWeb gathering, James who runs and writes James’ Coffee Blog shared his process and workflow for writing through to posting on his site, which was similar to my own process, but uses different tools along the way. He had something in his process he was looking to improve upon so I walked through my process.

My Process has Morphed Over the Years

My current writing process is an extension and evolution from my initial processes that trace back to college. But, it was in the late 1990s and early 2000s that my current process formed iterated upon. My early writing for blog posts and articles all start in markdown, which years back was well structured text and sometimes HTML for the structure.

My notes prior to formal blogging (started at the tail end of 2000) that I posted to my site were all hand coded HTML (or raw coding). If a note was going to turn into a blog post it quickly was marked up, and often as I was making the note or post.

Automate Early

One of the core elements I learned in the 1990s was to automate early for anything you can automate, use the tools you have at hand to help yourself be more productive. Most of my writing up through around 2010 was text and then quickly turned into HTML markup, as it is simple to do. But, the tough part is connecting related content, which is why I created my personal content management system (CMS) to run this vanderwal.net blog in 2000 and fully put it to use in 2001 (I used Blogger for a bit early on). Taking rote patterns and automating them was a great addition to help my process in 2000 to now.

In 2010 I shifted all of my note taking to markdown as moving across apps and devices made using other note making methods difficult to access and use. This shift to full markdown for notes, helped my writing move from notes to posts and articles much easier.

Markdown Workflow Process

Today and the last 5 years my notes start most often in Obsidian (and on mobile in Drafts, which is great for the good practice of get it out of your head and then sort out what to do with it). I have many notes flagged as “blogfodder” and track those through the writing process and moving them from a stub of an idea, to draft, not posted but ready, and posted (which includes a link to where it was posted). I’ll have another post about my new blogfodder process, which I’m really liking.

The writing starts as a note and gets fleshed out, if it is needed. If it is a short item I may stay in Obsidian and then grab and drag the markdown file in the Finder to the next step (I will get to this shortly).

Quite often I will leave Obsidian and from it click to open the Finder with the markdown file highlighted and then open that file in iA Writer - iA Writer, which is a nice focussed writing app for markdown, with additional capabilities. I often use the old journalism marker for needs attention (tk!) or “to correct” for things that need links, fact checks, or reference notes. Once I’m happy with the writing and structure in iA Writer (or Obsidian) I move to the next step.

Move to Prep for Posting

In years past I would take the markdown and quickly take the markdown structure and convert it to HTML markup by hand. Around 2013 or so I started scripting this transition, but the script was fussy. I’m not sure when, but it wasn’t long after this, I ran across Marked 2 - Marked 2. I think I started using Marked to convert markdown to PDFs and Word documents (for things that need to be sent out for formal article transformation in publications from Word). But, I realized Marked 2 had really good markdown to HTML conversion that was as good as my script, but not fussy. As I moved to Obsidian with properties in front matter and a footer with blogfodder tracking, it can remove all of that with ease, translate external links very nicely, and remove all backlink notation.

It is often in Marked 2 that I find markdown problems or the tk! marks. Marked 2 also includes some light grammar checking, which I appreciate and I’ll work through those suggestions. Then I flip to the HTML markup view and make edits, if needed, there.

Open My Blog Entry Form

Once I’m happy enough with the post I open my blog entry for for vanderwal.net or Personal InfoCloud and paste the HMTL into the form for the body of the post. On vanderwal.net I add in the title, location, type of post (these days everything is a weblog, but in the past it was more diverse), then select the related categories and submit it. Then I just to look at the post and review it again. If it needs an edit (up until July 5th) I would go into the database and make edits to the post there, but now have an edit process in my CMS (after 25 years, I figured it was about time). Once that is done, I go click to generate the RSS feed for the site, and send out alerts to services that share out links and summaries farther.

Wrap-up of the Workflow

This workflow is now done, except for seeing spelling errors or things not right and needing tweaks.

The process that starts in a markdown note, then progresses through to a more formal writing process and flow. I replaced a lot of manual steps that I didn’t think were difficult nor took a lot of time and automated the steps that do exactly what I had been doing with the same level of care, but saving time and reducing errors.

I don’t use AI in any of this writing process. I run across too much AI written content that is lifeless and doesn’t sound like the writer any longer. I’ve stopped reading many colleagues who used to have great ideas and a great personal voice, but are now just bland slop through the use of AI that tilts at, but doesn’t achieve mediocrity.

I have my own quirks and writing patterns, which I am fine with. I don’t write to impress, but to get honest ideas and understandings out. I blog and write to find connection with others of like or similar minds.