vanderwal.net

Personal Blog Data Analysis - Looking at 25 Years

After adding sparklines to my category lists (Updated Categories with Sparklines and Search is Now in Production) I wanted to have a deeper dive looking at my categories and blog analytics over 25 years.

Category Long Tail

I done a very quick capture of category usage to look at the distribution of use. A question from James about whether my category distribution looked like a long tail distribution and I thought it may, but also looking at the numbers and not having a visualization I wasn’t sure. Charting the use, it really was a very long tail / power law distribution.

I shared it with James and he also ran his and ended up with much the same (Is there a power law of category use? - James’ Coffee Blog). There have been a few discussions of late around category use and some lean into having just a few categories. I have just over 200 categories now as most of my blog post have more than one subject and I use the categories to have an way to jump to related posts that cover the same subject. When I built my site’s CMS I wanted to have the capability to have multiple categories on each post. I have multiple categories for my own purposes, but also I’m cognizant that readers may have other terms.

With the long tail use of categories I know readers may stumble across a post through web search or a link from else where and having a category term that is familiar can get them to other things I may have posted. I view the web as being able to connect with others and blog posts are sharing things I have interests in or curiosity around and being able to connect with others in a similar mindset is the aim. So a handful of categories, particularly across 25 years and over 2,100 posts, doesn’t help build those connections.

More Analysis on Blog Posts and Categories

This distribution of categories really pushed my interest and curiosity of what the last 25 years of blogging looks like. I joke that Twitter ate my blog, but the sparklines sort of show that. I wanted to see the trends on my blog more closely (I have the archive of my Tweets and I’ll analyze them later and then bring the two analysis together).

To prep for the analysis I pulled my site’s database data local and put it into SQLite (it is already on Mac - Apple and quick) to connect locally with Jupyter Notebooks and use Plotly for interactive data visualizations in the notebook. I had a series of questions, somewhat common data analytics questions I’ve used since grad school looking at analysis over time.

Posts per Month

The first analysis is my blog posts by month over 25 years. In my early years I was posting frequently, often in shorter posts (but we will look at that next), and then around 2005 (when started engaging with Twitter) things dropped off. Also at this time I also started blogging more serious subjects at Personal InfoCloud, but those were not all that frequent (I’ll dig into this at some point later too).

My hunch that I posted much more early on and drop off around the time I engaged with Twitter, seem to hold up.

Post Length Over Time

Post length over time also can tell the story of why post volume shifted. I wasn’t posting a lot of short post, but posting longer posts, but less frequently. I’m really curious what I posted in June of 2009 that caused the spike. The spike on the right end in 2020 and beyond are partly attributed to posting weeknotes, which tended to be longer than normal.

I know that my writing muscles went from a few hundred words early on to posts being around 1k and more. I found my comfortable blog post writing length was around 1.2k words. I write to find out and capture what I think, but rarely edit for brevity or other editing benefit, at least on this blog.

Median Categories per Post

This view of the median number of categories per post over time I found interesting and I didn’t know what my expected outcome was going into this analysis. The numbers pretty much are in line with longer posts have more categories to cover slightly more breadth in a post. Again June 2009, not sure. The spike spike on the right aligns with weeknotes, which cover multiple subjects in one post.

Distribution of Categories per Post

This chart groups number of categories on posts. This shows the second bar has the most number of posts (822 posts) have 2 or 3 categories on the posts. The third bar has 408 posts with 4 or 5 categories on them. This lines up well with the frequency and volume of posts early on which were shorter. Looking at the prior chart most posts had 10 or fewer categories on them.

Combined Timeline for Posts, Length, and median

I like this combined chart that reinforces early on with high volume of posts of shorter length and few categories on them. What I find interesting is the correlation of line trends for word count per post and categories per post. This ties closely with the longer posts have more categories.

Seasonal Patterns

The bar chart on the left is total number of posts by month and on the left it is average word count on posts by month.

This was largely a curiosity to see what was there, but also a common analysis trend analysis to see if there are explanations of other trends looking at seasonal comparisons. The posts by month is not surprising to me as summer and early fall months have often been busy. I am not all that sure what the word count by month tells other than the correlation between more posts and shorter post length correlation showing up.

Top Category Activity Over Time

This chart shows the top 30 categories (by use) and their activity over time.

In this heat map Apple categorized posts were sure common, as well as Information Architecture, Information Application Development, Personal, User-Centered Design, and Web Design were also common. Personal and this site’s development.

The heat map being far more dense on the left in early years is skewed by volume of posts and makes activity in the middle and right (more recent years) difficult to see. I need to spend more time on this analysis and chart to separate out the early years and segment things so time outside of the early years can have trends more easily seen. I may want to select a different visualization, but if I can break things out by time that should help. Also running 3 time segements with the same top 30 categories across them and then the top 30 within each time segment could be interesting.

To 40 Co-Occuring Category Pairs

This cart of co-occurring category pairs is in part preparatory work for bringing concurrent tags into the category pages here for understanding and filtering needs for users.

The top 5 pairs are all related to UX, IA, and User-Centered Design and these being the type of concurrence isn’t a surprise to me. The broad UX community had rather divergent use of terms at times and one person’s IA was and other’s UX. For readers who think about these posts in one manner could find other similar content by the term they are familiar with using. Pretty much this whole list is application development, web design and development, web apps, and pan-UX related.

I don’t know how useful this is for broad insights. When I get to adding the concurrent categories on the category pages this will likely be more helpful on a category by category view.

Category Co-occurence Network Graph

This chart looks at the top thirty categories that have 10 or more co-occurrence of categories.

This I find more interesting than the prior in that this has Social Software and Folksonomy showing up and showing its relationships. The largest category in this view is Interaction Design and its multiple connections. I am entertained by the standalone pairing of Apple and Software, that at the scale limited for the data these only connect to each other.

I need to rerun this with higher acceptance to get more included. But, also this graph isn’t interactive in Jupyter, and every time I went to zoom in it collapsed the graph and I couldn’t move a node out of the way was disappointing.

Helpful as a Good First Pass

This analysis and data visualizations were helpful to see into my 25 years of posts. There are some analysis sets and data visualizations that need more work. Most of these are more helpful with Plotly in Jupyter and the ability to interact with the visualizations.

I am really curious with what this will look like when I look at Twitter usage and notes. Obsidian on top of my notes make note making easier and far more helpful with backlinks / wiki links. I started using it on top of my directory with notes in June 2020 that had around 2k notes in it going back to 2003. Now there are around 6k to 7k and in the past about half of these notes would have been on one of my blogs.

Updated Categories with Sparklines and Search is Now in Production

I made a couple of updates I have long wanted to make to this site. I’ve been wanting to see frequency of categories used on my blog for more than 15 years (or pretty much since I’ve had category list pages). I have also wanted to have blog search and the utter mess that Google Search has become in recent years, where my site isn’t showing at all at times has driven this. These additions will likely iterate and adapt a bit going forward.

Updates to Category Lists with Sparklines

I have basically had two category list pages for years: Category List (which is alphabetical sort) and Category List by Use. I have kept these two and added sparklines to them (Sparkline - Wikipedia). Each line now has a small line chart that covers the 25+ years and what periods had used the category and some sense of the volume of use over time. One category list view I wanted and was missing was one to show a view with the focus of most recently used categories, so there is now a Category Recently Used List that not only groups by most recently used (and in the same entry keeps the alphabetical sort) but also shows the date of the last use in the list. Personally, I have been finding this recently used list view the most helpful and interesting. Skimming through the list I know I have more recent posts that have covered or touched on a subject, but it didn’t include the category, and that becomes a quick task to fix that gap.

Sparklines?

I have been a big fan of sparklines to give quick understanding of data’s distributions at a glance, which I learned about in “recommended reading” of Edward Tufte’s book The Visual Display of Quantitative Information | Edward Tufte in grad school in social / policy quant classes. (There are are many Tufte essays and book annotations on sparklines at Edward Tufte Notebooks & Sketches | Art, Science and Sculpture).

Creating the Sparklines

In creating the sparklines for my category lists I looked initially (and have long looked at them) creating static images from the data and bringing the images in (this would mean updating the images and replacing the old with the new ones, which is relatively straight forward programmatically and something I’ve done in the past, but not optimal) and I also looked at JavaScript but it was a bit slow. I poked at using creating SVGs (which work well when printing or zooming in) and often are much quicker and less strain on a browser than JavaScript. I’ve had a few goes at SVG in the past and I get get my mind around simple shapes, but I would need a little help with sparklines. A couple years back on a sparkline spelunking I found Easy SVG sparklines | Alex Plescan which showed the how. But, I have SVGs somewhat in the same category as regex, which is I do it rarely and I’ll just use Claude Code | Anthropic’s agentic coding system \ Anthropic to assist with the creation.

Chunking the Data for a Sparkline

The other part of sparklines is they are intended to be small glimpses and I have 25+ years of posts and a monthly temporal segmentation would make for a long graphic. I played around with breaking things down to quarters, but in the end I went with two segments per year and roughly 50 data points to map out on a line chart. Running a test with the two data points a year was a reasonable enough glimpse to sort out if the category was used recently or what the variation of use was over time.

One of the interesting discoveries with the first lab run of the categories of sparklines was the rather “U shaped” distribution of the use of categories, which pretty much calls out the lull I had in blogging. This softening of blog post rhythm is something I call, “when Twitter ate my blog” (where the interesting things I would discover and want to share and interact around ended up on Twitter rather than my blog(s)). Other patterns that surfaced were limited use a category in a period when I was rather sure I had posted on the subject, some of this was I was not using the term in that way or I didn’t have the category in my system yet. One of the things that helped sort this out was using my blog search.

Search is Now Out of the Lab

One of the things I have been working on and using my my Lab at vanderwal.net is blog search. But, the modifications I made to the Category pages I found I was leaning on my blog search a fair amount to investigate things. But, the categories and blog search are both in the blog section of this site, so making the change from the lab to the production side made sense. One of the things holding back moving search over, was I had an SVG of a magnifying class in the menu bar with “Blog Search”, but no matter how small I made the image it still was messing with the vertical layout of the menubar. In removing the magnifying glass and just using text things kept to the same layout.

Bringing in Search

The search in the menu in the pages in the blog section with “Off the Top” or “random” in the URL which is where there are currently menubar links to Blog Search. I have the menu bar link to a search page to search from rather than a JavaScript drop down or other menu bar convention (again layout of the menu bar was part of the considerations).

When I was working on search in the Lab section I found I needed to make some modifications to the database to have quicker search and I needed to modify the database engine so I could have search include 3 letter terms as a minimum rather than 4 letter words. In working on search I found many of my early posts didn’t (and still don’t) have titles and I was using the title as the link. I initially thought I would just add titles, but there are around 300 posts that don’t have titles (I’m adding some as I touch the posts for other clean-up issues), but I ended up coding the search results to have the results just fill in “Blog Post #…” as a proxy for a proper title.

The initial 170 or so posts are not in the database and are therefore not in the search.

Bringing Search and Categories In

As I went to move the category list pages out of the lab and into the production side I needed to modify a few other templates and pages to add the updated links. In doing this I realized I could also easily update the menu bar to include “Blog Search”. So, I took a little bit of time and made both changes at the same time.

Not all of the links are in yet. If you see something a little off with category lists or missing blog search links let me know.

New Profession Unfolding In Beauty and Geekery

A week or more ago I ran across the incredible video of Blaise Aguera y Arcas presentation of Photosynth and Seadragon at TEDTalks 2007. The video is stunning work of Seadragon and Photosynth bringing a collection of images to life from one or more resources.

While the video and ideas behind the tools are incredible displays of where we are today with technology and where we are heading, this caused some ideas I have been trying to get to gel to finally come together. In this video Blaise states (my own transcription):

So what the point here really is, is we can do things with the social environment taking data from everybody, from the entire collective memory of what the earth looks like, and link all of that together and make something emergent that is greater than the sum of the parts. You have a model that emerges of the entire earth, think of it as the long tail to Stephen Lawlers Virtual Earth work. This is something that grows in complexity as people use it and whose benefits become greater to the users as they use it. Their own photos are getting tagged with metadata that somebody else entered. If somebody bothered to tag all of these saints and say who they all are, then my photo of the Notre Dame Cathedral suddenly gets enriched with all of that data. I can use it as an entry point to dive into that space in that metaverse, using everybody else's photos, and do a cross-modal and cross-user social experience that way. Of course a by product of all of that is an immensely rich virtual models of every interesting part of the earth, collected not just from overhead flight and satellite images, but from the collective memory.

Torrent of Human Contributed Digital Content

What this brought together was the incredible amount of human contributed digital content we are sitting on top of at this moment in time. This is not a new concept, but what is different is the skills, tools, and understanding to make use and sense of all this content are having to change incredibly. The Photosynth team is making use of Flickr content that has been annotated by humans (tags, titles, and descriptions), as well as by devices (date, time, location, etc.). This meta information provides hooks put pull disparate information back from its sole beauty and make an even greater beauty and deeper understanding. The collective is better than the pieces, but pulling to collective together in a manner that is coherent, adds value, and brings deeper appreciation is where get hard.

Much of information understanding and sense making to date has relied on human understanding and we have used tools to augment our understanding. But, we now need to rely on deeper analytics in quantitative methods and advanced algorithms to make sense and beauty out of the bits and bytes. The models of understanding are changing to requiring visualizations methods (much like those of Stamen Design) to begin to grasp and see what is happening in our torrent of information at our finger tips and well as make sense of the social interactions of our digitally networked and digitally augmented lives.

Amalgamation of Designer and Quant Geek

What gelled in my mind watching the Blaise demonstration is there is a skill set missing in the next generation comprised of amalgamated design, information use, analytical foundation, and strong quantitative skills. I have clients in start-up businesses and in enterprise that are confronting these floods of information they need to make sense of from folksonomies and customer generated content (including annotations and regular feedback). The skills needed for building taxonomies are not translating well when the volume of information the information managers are dealing with is orders of magnitude higher than what they dealt with previously. The designer, information architect, and taxonomist who have traditionally have dealt with building the systems of information order, access, and use are missing the quantitative skills to analyze and make sense out of a torrent of loosely structured information and digital objects. Those with the quantitative and strong analytical skills have lacked the design and art skills to bring the understanding into frame for regular people grasp and understand.

This class of designer and quant geek is much like the renaissance men, but today the field of those forging new ground is open to men and women. The need to understand not only broad but deep sets of data and information so to contextualize it into understanding is the realm of few, unfortunately as there is a need for many.

I know of limited pockets of people with the skills to do the hard work of querying the vast array of information, objects, and raw data then make something of value of it. But, there needs to be more of these people getting trained as designers with solid quantitative and analytical skills (or the converse). Design shops are missing the quant geeks and engineering shops are missing the visualization geeks that bring this digital world rich in opportunity into something that makes sense and beauty.

If you know people like this that are bored, please let me know as I am finding opportunities flowing.

More XTech 2006

I have had a little time to sit back and think about XTech I am quite impressed with the conference. The caliber of presenter and the quality of their presentations was some of the best of any I have been to in a while. The presentations got beneath the surface level of the subjects and provided insight that I had not run across elsewhere.

The conference focus on browser, open data (XML), and high level presentations was a great mix. There was much cross-over in the presentations and once I got the hang that this was not a conference of stuff I already knew (or presented at a level that is more introductory), but things I wanted to dig deeper into. I began to realize late into the conference (or after in many cases) that the people presenting were people whose writting and contributions I had followed regularly when I was doing deep development (not managing web development) of web applications. I changed my focus last Fall to get back to developing innovative applications, working on projects that are built around open data, and that filled some of the many gaps in the Personal InfoCloud (I also left to write, but that did get side tracked).

As I mentioned before, XTech had the right amount of geek mindset in the presentations. The one that really brought this to the forefront of my mind was on XForms, an Alternative to Ajax by Erik Bruchez. It focussed on using XForms as a means to interact with structured data with Ajax.

Once it dawned on me that this conference was rather killer and I sould be paying attention to the content and not just those in the floating island of friends the event was nearly two-thirds the way through. This huge mistake on my part was the busy nature of things that lead up to XTech, as well as not getting there a day or two earlier to adjust to the time, and attend the pre-conference sessions and tutorials on Ajax.

I was thrilled ot see the Platial presentation and meet the makers of the service. When I went to attend Simon Willison's presentation rather than attending the GeoRSS session, I realized there was much good content at XTech and it is now one on my must attend list.

As the conference was progressing I was thinking of all of the people that would have really benefitted and enjoyed XTech as well. A conference about open data and systems to build applications with that meet real people's needs is essential for most developers working out on the live web these days.

If XTech sounded good this year in Amsterdam, you may want to note that it will be in Paris next year.

The World in Our Hands

SmartMobs announces It is official, there are more cellphones lines than landlines in the U.S.. I was thinking about this in the past couple weeks. We have already started seeing text and data uses tipping our mobile hands (it is about time we started getting to where much of the rest of the globe has already been).

Now if I could just keep my finger on the number of data enabled phones and the lesser number of laptop/desktop internet connections for the globe. Every time I see this number I forget to mark it or grab it.

[Hat tip Anne]