• 🔗 Django & Celery in production

    DjangoCon JP 2021 発表資料

    Djangoで非同期処理を実現するために、よく使われているCelery。ただDjangoほど知見が共有されていないため、なんとなく使っているという方も多いのではないかと思います。そのような場合Celeryを使えるようにするまでは順調でも、実際に運用がはじまったあとに困ることが出てきます。例えば、ログの保存、リトライの設計、デプロイ戦略など。
    このトークでは、CeleryをDjangoプロジェクトで実際に運用するうえでの役立つTipsをお伝えします。
    Some good tips about Celery Production tips from Django Congress 2021 in Nagano this year (I couldn't attend). Most of the information is in the docs in English, but it's handy to see it condensed, even if it's in Japanese. A couple key points:

    • Use **kwargs for your tasks input. This makes it easier to make updates to your tasks once they're already running in production if your input needs to change. 
    • Reminders about all of the handy kwargs you can pass to tasks in regards to retry. Especially handy autoretry_for where you can pass a tuple of exceptions that will cause the task to automatically retry
    1. Tagged with
    2. python
    3. django
  • Gathering Requirements for Native Crossposting in Tanzawa

    Ru made a feature request for Tanzawa to be able to syndicate / cross post from Tanzawa to other sites natively. I think it's a great idea and spent some time to collect my thoughts about what that could look like.

    My thoughts were posted on GitHub, but I'm copying them here so I always have a copy and in case people who read my blog but don't follow  the Tanzawa GitHub have any input.

    --- 

    Agreed – being able to syndicate content directly from Tanzawa would be ideal. All syndication is currently manual (it doesn't hurt enough to automate it, yet).

    Base Thoughts 


    If we can figure out what the workflow would look like, I think we can define some base requirements for the feature. A couple of general thoughts about syndication:

    • I don't think posting should be syndicated automatically on publish. There's a couple of reasons for this: 
      • Tanzawa doesn't support any kind of background tasks. Posts that have a bunch of links take a while to complete it tries to send webmentions. Adding more external requests could increase the response time to longer than the request timeout allows quite easily. This is especially true because webmentions require 2 requests per save (before content update and after content update). We could probably process webmentions concurrently to speed this up, but that feels like a bit much at this point? It may be easy with asyncio, but I'd need to research it.
      • When I was posting with Wordpress there was a syndication checkbox (via brid.gy) and found it you'd get errors if you saved twice as the post had already been syndicated. Because there also wasn't a preview, it was always a bit of a gamble for how it would post.
    • Syndication is a separate concern from authoring posts themselves. And as such warrants dedicated screens to make it _right_.  Initially I had thought it might work well to handle syndication via a modal on the edit post page, but the more I think about it, it would box us in.
    • Brid.gy is still super handy for backfeeding likes/replies as webmentions. I still think brid.gy is useful / "required" unless we were to build in some kind of polling mechanism, which is difficult without support for cron/background tasks. ( It could be done "easily" with a django management command, I'm hesitant to introduce undue complexity).
    • "Syndication" feels a bit jargony to me. Is "Cross post" (or something similar)  a better term to use in the interface?
    • Each syndication destination is going to require some kind of settings (API keys, mostly). These can / should be managed via the Django admin?

    Workflow


    Create of  Syndication/Cross Posts

    This is assuming that all syndication can be handled with the same form.

    • Creating a new syndication would happen from the edit post button. The button should be on the meta menu (on the right), below the publish  buttons and only display once the post is published (you can't syndicate a draft).
    • Clicking this button would take you to dedicated syndication page.
    • The top of the page would be a form. The form would display the syndication text and up to 4 photos (with the caption).

      The syndication text is free text and pre-filled with either the post title + link or the first 280 chars of the note. Above the text field itself would be a segmented control (tabs? radio button?) that when clicked would let you toggle between the different presets.

      You may not want to syndicate each photo, so perhaps a checkbox is warranted to let you select which photos will be sent.
    • Finally at the bottom there's some checkboxes for which networks to syndicate to. As you can only syndicate a post once per network, posts that have already been syndicated to would be disabled with text explaining that you've already syndicated to that network for this post.
    • Once syndicated they'd store the url of the syndication and create a TSyndication record.

    Syndication List

    I think we'd want a separate section for syndication, like we do with "Posts / Files" on the left for syndications / cross posts. 

    Clicking that would show you a list of your most recent syndication activity ( syndication destination (link to twitter etc..), syndicated text, link to the related post). 

    Selecting a crosspost would show you a (readonly?) version of the syndication Create form.

    ---

    Does that sound about right? Anything you'd change or we should consider (especially regards to Mastodon,  I don't have an account/haven't used it before).
  • TIL: English and GEOS Reference Points Opposite of Each Other

    This post is less a TIL and more of a I knew that and I don't want to forget it again and stems from a bugfix in Tanzawa.

    In English when we refer to a geo-coordinate we usually say it in latitude, longitude order. The reason why we say coordinates in this order is we could measure latitude accurately (via astronomical measurements) before longitude. Frontend mapping libraries like leaflet.js keep this familiar ordering. i.e. plotting points on a map takes a latitude/longitude array and events have a latlng property for referencing points.

    GEOS, the open source geometry library used in most GIS (include GeoDjango) applications doesn't think of points in those terms, but as a graph of x,y coordinates.  As such if you when you're working with data across these boundaries it's important to not mix up your ordering.

    When instantiating a Point it's tempting to just pass in floats directly. But if you do that it's easy to mix up the ordering , so I've started make sure I always use the keyword argument name to reduce mistakes.

    from django.contrib.gis.geos import Point
    
    # Keep our familiar lat/lon ordering without messing up the data point.
    point = Point(y=35.31593281000502, x=139.4700015160363)
  • How to Pipe Python stdout with xargs

    When writing instructions for getting started with Tanzawa, users needed a way to set a unique SECRET_KEY in their environment variable configuration file. Initially I had a secret key entry in the sample file with some instructions to "just modify it". But that felt like I was just passing the buck.

    What I wanted to do was to generate a unique secret_key and output it to the .env file. Outputting just the secret key is simple, you can just use >> and append output to an existing file. But I wanted to use my Python secret key output as an argument to another command.

    I did it as follows:

    python3 -c "import secrets; print(secrets.token_urlsafe())" | xargs -I{} -n1 echo SECRET_KEY={} >> .env

    1. Use the Python secrets module to generate a secure token.
    2. Pipe the output to xargs.
    3. -I is "replace string" and "{}" is the string we want xargs to replace. -n1 limits us to a single argument.
    4. xargs executes and takes our Python output as an argument and replaces the {} with it, giving us our desired string.

    Writing this now, I probably could have just used Python to include the SECRET_KEY= bit and forgone using xargs, but it was good practice anyways.
  • Writing Professionally in a Second Language

    I just finished an article that will be published in this month's Web+DB Press magazine, a monthly web-related programming magazine in Japan. The entire magazine, including my article, is in Japanese. I want to reflect on the process of writing professionally in a second language and contrast this with my first experience when writing the 2nd chapter of Professional Python Programming 3rd edition (PyPro3).


    Picking a Topic


    Unless you're writing for yourself, the general topic is going to be decided for you. The company I work for writes regularly for Web+DB Press. GraphQL's been gaining popularity lately, and since I've built a few projects with it, a co-worker asked if I was interested in writing an article about GraphQL. Remembering how difficult it was to write PyPro3, I was a bit hesitant at first, but thought it'd be a good test to see how much my Japanese had I'd improved over the years. I also wanted share what I'd learned building GraphQL APIs with Django.

    Outlining


    Outlining is the process where you set the scope and decide in in more detail what you want to write about and the general order that you'll write. Before you can start outlining you need to first decide roughly how long of an article you want to write. The length of a magazine article is often decided for you, so it's mostly a matter of figuring out what you can fit into the allocated space.

    My outline was fairly basic, just a bullet list to help guide me along as I wrote and the base idea for what we'd build in the article.

    Writing


    Writing in your first language is difficult enough. Writing in a second language maintains the challenges of writing in your first, but also adds an extra layer on top. Not all idioms or expressions in your native language can be expressed in eloquently or in a similar manner in a different language.

    Having a solid outline of what you're going to write about, including the order and main points you want to cover makes writing much easier. This applies to writing in your first language as well, but it's doubly important in a second language.

    When writing PyPro3, I could write and speak Japanese, but writing a narrative piece was difficult. Many of the words I was using weren't quite yet internalized and reading what I wrote 5 minutes prior was a challenge. So I ended up writing the gist of what I wanted to say in English, and then translating it as best as I could before a native speaker cleaned it up for me.

    This time around, 3 years later, my outline was in Japanese and I didn't write any of the article in English first and then translate. Writing was a lot quicker too as I had become used to explaining code and what it's doing from work. I made the quickest progress when I was focusing on writing for a specific co-worker and explaining GraphQL to them.

    Editing


    Editing in a second language is difficult. I won't say I'm useless, but I don't have that feeling when something sounds "off" like I do with English. Without that sense, it was difficult to know exactly how to fix feedback from internal reviews and the editor at the magazine.

    The actual process of editing this time around was much easier, however. A co-worker and I would pair edit, where I'd pull up the document on my screen via Zoom and we'd work through the comments together. This made it much easier to ask for advice or for clarification or a second opinion. 

    Beyond not having a native feel for the language, skimming is also much more difficult. This makes finding the paragraph a particular comment is referencing a bit slow. I got better as I practiced, but I also found that copying a snippet of the comment and using the Find feature of my editor helped immensely. 

    Conclusion


    Good writing in any language requires clear thinking and proper planning. Being able to stay in the same language for planning, drafting, and editing makes the entire process much easier. Having native speakers check and edit your work  allows you to gain a better understanding of your second language, not only at the grammatical level, but with sentence patterns and flow as well.

    I'm grateful to have been given the opportunity to be published, especially in my non-native language and hope I'll have another chance in the future.
  • How to Process Tailwind / PostCSS with Webpack

    Usually when I work with webpack another tool I'm using generates the appropriate config for me and I can remain blissfully unaware of how it all works.

    With Tanzawa I've only been adding tooling when absolutely necessary. The other day I configured PostCSS and Tailwind with Webpack. It still required a bunch of searching and piecing together blog posts to get something that worked for me.

    Below is a list of facts that helped me figure out how to think about processing CSS with webpack.

    • As wrong as it feels, your entry point for processing your CSS should be a Javascript file.
    • Webpack by default does not output a separate stylesheet file. In order to output a plain CSS file, you must use the MiniCssExtractPlugin.
    • Despite wanting to output only CSS, and specifying the filename in the options (style.css), Webpack will create an empty Javascript file regardless. There isn't a way to prevent this unless you add another plugin. I'm adding it to .gitignore.
    • The "use" plugins have the following roles
      • MiniCssExtractPlugin: Exact built css to a dedicated css file.
      • css-loader: Allow you to import css from your entrypoint Javascript file
      • postcss-loader: Run postcss with your 

    // webpack.config.js
    const path = require('path');
    const MiniCssExtractPlugin = require('mini-css-extract-plugin');
    
    const tailwindConfig = {
      entry: "./css/index.js",
      output: {
        path: path.resolve(__dirname, "../static/tailwind/"),
      },
      plugins: [
        new MiniCssExtractPlugin({ 
            filename: "style.css"
        }),
      ],
      module: {
       rules: [
        {
          test: /\.css$/,
            use: [
              MiniCssExtractPlugin.loader,
              { loader: "css-loader", options: { importLoaders: 1 } },
              "postcss-loader",
            ],
        },
       ],
      },
    }
    module.exports = [tailwindConfig]

    In order for postcss to play well with Tailwind and webpack, I needed to update my config to pass the tailwind plugin the path for the tailwind.config.js. It simply imports (requires) the tailwind config and immediately passes the path.

    // postcss.config.js
    module.exports = {
      plugins: [
        require("tailwindcss")("./tailwind.config.js"),
        require("autoprefixer"),
      ]
    }

    Finally to run this all for production, I execute via webpack as follows. I still need to figure out how to get the NODE_ENV environment variable set via with the webpack --production flag, so for now it's a bit redundant.

    // package.json
    {
      ...
      "scripts": {
        "build": "NODE_ENV=production webpack --mode=production"
      }
      ...
    }
  • 🏔Tanzawa is Public 🎉

    Over the past few months I've been talking about the release of Tanzawa. And over these few months I keep on building up what the first release needs to be in my head. It must: 

    • not have any bugs
    • be easy to use
    • have proper documentation for getting started

    and so on and so forth. I keep on finding new things to add or fix that are "must haves" and so I keep on finding reasons to delay the release. And it's slowly been adding a bit of self-induced stress.

    No more.

    Tanzawa is now open source. It lives on GitHub at jamesvandyne/tanzawa. It has bugs. It has a bunch of edges rough enough to cut you. It has zero documentation for how to install and use it.

    But I'll address them overtime. And eventually, someday,  it will reach my vision of being a simple and sustainable  home on the web.

    If you need help or have questions or want to lend a hand: reply here, ping me on twitter @jamesvandyne, or open an issue.

    Tanzawa is public
  • Response to Developer relations – Marco.org

    Modern society has come to rely so heavily on mobile apps that any phone manufacturer must ensure that such a healthy ecosystem exists as table stakes for anyone to buy their phones.
    I wasn't concerned when the iPhone first came out and third party apps could only be installed via the App Store. Unlike Android, having a single place to go to install apps is arguably a much better customer experience. Having the manufacturer manually approve each app that's installable on your phone seemed warranted as data was super expensive and you didn't want an app misbehaving on your 3G connection giving you surprise bills. This approval process provided some assurance this wouldn't happen.

    However in the years since the AppStore's release, mobile phones have become central to modern society. Even in Japan, a country famous for holding on to fax machines and personal stamps, it's becoming harder to exist without one of their devices.

    Because of the cellphone's new role as the interface for interacting with society, a closed AppStore and closed devices that only allow you to interact with society via a benevolent dictator's approved was feels increasing anti-democratic.

    No matter how benevolent of a dictator they may be, they're still a dictator.
  • Picking a License for Tanzawa

    As Tanzawa is getting closer to being something I can release for other people to use, I've been trying to decide on the best license for it and my goals for the project. The most likely scenario is that Tanzawa will only power my blog or a handful of blogs and that I'll be the only regular contributor. I'd love to be proven wrong, though 😀.

    My main goal for Tanzawa is to provide a system that slightly-technical folks can use to create their own home online. My secondary goal is to explore low-resource computing and using Tanzawa as my proving ground. If I can find a model that would allow me to build a revenue stream around Tanzawa, that would be great, but it's not a primary focus.

    I've considered all of the main open source licenses: Apache, AGPL, MIT, and BSD-3.  Each license is appealing to me for different ideological reasons. 

    Before researching licenses a bit, I had thought I would pick the AGPL. I really like changes would need to be released to the community. Wordpress uses the GPL, too. 

    But then I started thinking about the possibility of building software around Tanzawa to support paid hosting for Tanzawa blogs. This bit wouldn't be open source and it may require some custom hooks into Tanzawa. Picking the AGPL would lock me in needing to release these changes. Being the original author, I could just dual license it to myself, but it gets a bit murkier if anyone contributes to the project.

    I considered the MIT and BSD licenses together. BSD is basically public-domain in my mind. You're free to use Tanzawa how you please and keep all your changes to yourself. I'm not entirely opposed to this, and I'm sure it happens in reality with GPLed code. This said if someone uses Tanzawa to build something, I'd like them to acknowledge that what they've built is built on Tanzawa.  While I think scenario is a low probability, it's not zero.

    The last license I considered is the Apache license. What sets it apart from other licenses is its stance on software patents. Basically is someone contributes to Tanzawa and then sues me (again, low probability), they lose their right to use the software. But not being able to be sued gives me a piece of mind I can't get with the other licenses.

    Now to merge license files into the repository.
  • Re: Ode to my flip phone

    Reading Ode to my flip phone reminded me of Frank in You've Got Mail, singing the praises of his typewriter as everyone is getting computers and getting online. Society, for better or worse, has picked a new technological successor and the old ways – your flip phone, your typewriter, will eventually disappear.

    But this line – this line hit me hard.
    And by privacy I don’t mean cookies or my Social Security number or whatever—I mean the fragile sphere of imagination in which I exist when I’m not diddling about online. I mean what’s left of my nondigital self. When I clack your two halves shut, you glorious techno-mollusk, that’s it. Sauron cannot see me.
Previous 4 of 10 Next