vanderwal.net

Automating the Daily Dump Note

In work and personal life I’ve long been an automator. It started in the ’90s, but actually started when I was a kid and talking with my dad about systems, making them more efficient, and removing steps, and automations. Any manual rote step I look at and think about making it a click or “just run”. I’ve used Obsidian since June 2020 and have a lot of things in my markdown notes trigged by TextExpander.

Reducing Steps Setting my My Daily Dump Note

Little by little I’ve been building out some automations for my note taking that sits under Obsidian’s watchful eye, but may start elsewhere or seeds start elsewhere. In the morning I run some automations that sweep up videos of interest I forwarded to myself from the prior evening or that morning that are in a personal instance of Slack. I have one that pulls in notes in Drafts with a specific tag then archives the note. One last piece I’d really like to add is to have my known items from across my calendars dropped in at the top of that Daily Dump note.

Getting the calendar dump is something I’d really like to have before I get to my laptop in the morning. But, each of these automations could and should be automated and dropped in the note before I fully get to it as well.

This takes one thing… having the note.

Creating the note with the series of headers and subheadings is something I usually do in a new markdown note with TextExpander each morning. But, I can’t automate with TextExpander (or I haven’t sorted out how to trigger that from a far).

Automating but Not with Apple Shortcuts

I started a couple days ago thinking that since I had Actions for Obsidian I could easily create a new Daily Dump with the correct future date as a title and have it trigger the Templater template for the note with the correct date. I quickly got the file created properly, but in over 4 hours that included a couple hours the following day it was still not working properly. This also included a few different apps and pieces and if any of them change it breaks.

Since I not only wanted to create one note with a template for the next day, but to have a few days ahead created (say a week at a time). I gave up on Shortcuts and in under 20 minutes I had a python script doing exactly what I wanted. This included setting up a cronjob to run it on Sunday evening and have a week of notes set.

Using the Output

This morning I opened Obsidian and one of the future day’s notes I had setup as tests was a click away and I was capturing an idea. This is a nice change and one I wished I had done long ago.

The next step is automating bringing in the aggregation automations notes and dropping that into my Daily Dump.

My Writing Process with Obsidian

A few times lately my writing process has come up as it relies heavily on markdown for portability, longevity access, flexibility, and using the Shift Happened – Part 2: Small Apps Loosely Joined – Personal InfoCloud. The Small Apps Loosely Joined concept is something I use heavily for many things, but for writing the use of markdown I use the concept quite heavily.

Recently in a IndieWeb gathering, James who runs and writes James’ Coffee Blog shared his process and workflow for writing through to posting on his site, which was similar to my own process, but uses different tools along the way. He had something in his process he was looking to improve upon so I walked through my process.

My Process has Morphed Over the Years

My current writing process is an extension and evolution from my initial processes that trace back to college. But, it was in the late 1990s and early 2000s that my current process formed iterated upon. My early writing for blog posts and articles all start in markdown, which years back was well structured text and sometimes HTML for the structure.

My notes prior to formal blogging (started at the tail end of 2000) that I posted to my site were all hand coded HTML (or raw coding). If a note was going to turn into a blog post it quickly was marked up, and often as I was making the note or post.

Automate Early

One of the core elements I learned in the 1990s was to automate early for anything you can automate, use the tools you have at hand to help yourself be more productive. Most of my writing up through around 2010 was text and then quickly turned into HTML markup, as it is simple to do. But, the tough part is connecting related content, which is why I created my personal content management system (CMS) to run this vanderwal.net blog in 2000 and fully put it to use in 2001 (I used Blogger for a bit early on). Taking rote patterns and automating them was a great addition to help my process in 2000 to now.

In 2010 I shifted all of my note taking to markdown as moving across apps and devices made using other note making methods difficult to access and use. This shift to full markdown for notes, helped my writing move from notes to posts and articles much easier.

Markdown Workflow Process

Today and the last 5 years my notes start most often in Obsidian (and on mobile in Drafts, which is great for the good practice of get it out of your head and then sort out what to do with it). I have many notes flagged as “blogfodder” and track those through the writing process and moving them from a stub of an idea, to draft, not posted but ready, and posted (which includes a link to where it was posted). I’ll have another post about my new blogfodder process, which I’m really liking.

The writing starts as a note and gets fleshed out, if it is needed. If it is a short item I may stay in Obsidian and then grab and drag the markdown file in the Finder to the next step (I will get to this shortly).

Quite often I will leave Obsidian and from it click to open the Finder with the markdown file highlighted and then open that file in iA Writer - iA Writer, which is a nice focussed writing app for markdown, with additional capabilities. I often use the old journalism marker for needs attention (tk!) or “to correct” for things that need links, fact checks, or reference notes. Once I’m happy with the writing and structure in iA Writer (or Obsidian) I move to the next step.

Move to Prep for Posting

In years past I would take the markdown and quickly take the markdown structure and convert it to HTML markup by hand. Around 2013 or so I started scripting this transition, but the script was fussy. I’m not sure when, but it wasn’t long after this, I ran across Marked 2 - Marked 2. I think I started using Marked to convert markdown to PDFs and Word documents (for things that need to be sent out for formal article transformation in publications from Word). But, I realized Marked 2 had really good markdown to HTML conversion that was as good as my script, but not fussy. As I moved to Obsidian with properties in front matter and a footer with blogfodder tracking, it can remove all of that with ease, translate external links very nicely, and remove all backlink notation.

It is often in Marked 2 that I find markdown problems or the tk! marks. Marked 2 also includes some light grammar checking, which I appreciate and I’ll work through those suggestions. Then I flip to the HTML markup view and make edits, if needed, there.

Open My Blog Entry Form

Once I’m happy enough with the post I open my blog entry for for vanderwal.net or Personal InfoCloud and paste the HMTL into the form for the body of the post. On vanderwal.net I add in the title, location, type of post (these days everything is a weblog, but in the past it was more diverse), then select the related categories and submit it. Then I just to look at the post and review it again. If it needs an edit (up until July 5th) I would go into the database and make edits to the post there, but now have an edit process in my CMS (after 25 years, I figured it was about time). Once that is done, I go click to generate the RSS feed for the site, and send out alerts to services that share out links and summaries farther.

Wrap-up of the Workflow

This workflow is now done, except for seeing spelling errors or things not right and needing tweaks.

The process that starts in a markdown note, then progresses through to a more formal writing process and flow. I replaced a lot of manual steps that I didn’t think were difficult nor took a lot of time and automated the steps that do exactly what I had been doing with the same level of care, but saving time and reducing errors.

I don’t use AI in any of this writing process. I run across too much AI written content that is lifeless and doesn’t sound like the writer any longer. I’ve stopped reading many colleagues who used to have great ideas and a great personal voice, but are now just bland slop through the use of AI that tilts at, but doesn’t achieve mediocrity.

I have my own quirks and writing patterns, which I am fine with. I don’t write to impress, but to get honest ideas and understandings out. I blog and write to find connection with others of like or similar minds.