Tag: twitter

Joining the Herd

With the news about what’s happening on the BirdSite… I thought it might be time to explore what the state of the art alternative is these days.

As an aside, this isn’t the first time I’ve looked at Twitter alternatives, back in the very early days of Twitter I build and ran a project called BlueTwit inside IBM. This was ground up clone that was there to see how a micro blogging platform (how quaint a term…) would work in a large organisation. It had a Firefox plugin client, supported 250+ characters long before Twitter. The whole was written in Java ran on a modest AMD64 box under my desk for many years and was even played with by the IBM Research team before similar functionality ended up in IBM/Lotus Connections. (Someday I should do a proper writup about it…)

Anyway back to the hear and now… I have a propensity for wanting to know how things work under the covers (which is why I run my own web, DNS, mail, SIP, finger and, gopher server). So I thought I’d have a go at running my own Mastodon server for a little while to see how it all fits together.

A little digging shows that a Mastodon instance isn’t just one thing, it needs the following:

  • Mastodon Web interface
  • Mastodon Streaming interface
  • Mastodon Job server (sidekiq)
  • A PostgreSQL instance (for persistence)
  • A Redis Instance
  • A Elastic Search Instance (optional)

Given this list of parts trying to run it in a containerised environment made sense. I have both a Docker Compose setup and a Kubernetes setup at home for testing FlowForge on while I’m working, so that wouldn’t be a major problem (though I understand I’m the outlier here). I decided to go with Kubernetes, because that cluster is a bit bigger and I like a challenge.

Deploying to Kubernetes

A bit of Googling turned up that while there isn’t a published Helm chart, there is one included in the project. So I cloned the project to my laptop.

git clone https://github.com/mastodon/mastodon.git

Configuration

I then started to create a local-values.yml file to contain my instances specific configuration details. To get a feel for what values I’d need I started by looking in the chart/values.yml file to see what the defaults are and what I could override.

I also started to read the Mastodon server install guide as it had explanations to what each option means.

The first choice was what to call the instance. I went with a suggestion from @andypiper for the server name, but I’ll have the users hosted at my root domain

This means that the server is called bluetoot.hardill.me.uk and the instance is called hardill.me.uk so users will be identified for example as @ben@hardill.me.uk. These are configured as local_domain and web_domain

Next up was setting up the details of my mail server so that thinks like alerts and password resets can be sent. That was all pretty self explanatory.

The first tricky bit was setting up some secrets for the instance. There are secret keys for authentication tokens and a seed for the OTP setup. The documentation says to use rake secret but that implies you have Ruby environment already setup. I don’t work with Ruby so this wasn’t available. A bit more searching suggested that OpenSSL could be used:

openssl rand -hex 64

The next set of secrets are the vapid public and private keys. Here the documentation again has a rake command, but this time it appears to be a Mastodon specific Gem. To get round this I decided to pull the pre-built docker image and see if I could run the command in the container.

docker pull tootsuite/mastodon
docker run -it --rm --entrypoint /bin/bash tootsuite/mastodon:latest
mastodon@ab417e0a893a:~$ RAILS_ENV=production bundle exec rake mastodon:webpush:generate_vapid_key
VAPID_PRIVATE_KEY=MdgBNkR98ctXtk3xSTbs7-KJBCcykvvw_q1aFGNfMgY=
VAPID_PUBLIC_KEY=BB-g5Lgund3gYi3UhGGn7Z1Yj06gy4DqdozXQXYxeCDJjpEUW9TXYau7Ifv9xK_676MgUE4JSOSh4XSsroBoHmo=

(These keys are purely for demonstration and have not been used)

I made the decision to skip the elastic search instance to start with just to try and limit just how many resources I need to provide.

There are a couple of other bits I tweaked to make things work with my environment (forcing the bitnami charts to run on the AMD64 nodes rather than the ARM64) but the local-values.yml ended up looking like

mastodon:
  createAdmin:
    enabled: true
    username: fat_controller
    email: mastodon@example.com
  local_domain: example.com
  web_domain: bluetoot.example.com
  persistence:
    assets:
      accessMode: ReadWriteMany
      resources:
        requests:
          storage: 10Gi
    system:
      accessMode: ReadWriteMany
      resources:
        requests:
          storage: 10Gi
  smtp:
    auth_method: plain
    delivery_method: smtp
    enable_starttls_auto: true
    port: 587
    server: mail.example.com
    login: user
    password: password
    from_address: mastodon@example.com
    domain: example.com

  secrets:
    secret_key_base: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    otp_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    vapid:
      private_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      public_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
elasticsearch:
  enabled: false

ingress:
  annotations:
  tls:
  hosts:
  - host: bluetoot.example.com
    paths:
    - path: '/'

postgresql:
  auth:
    password: xxxxxxx
  primary:
    nodeSelector:
      beta.kubernetes.io/arch: amd64

redis:
  password: xxxxxxx
  master:
    nodeSelector:
      beta.kubernetes.io/arch: amd64
  replica:
    nodeSelector:
      beta.kubernetes.io/arch: amd64

Deploying

The first part is to run helm

helm upgrade --install  --namespace mastodon --create-namespace mastodon ../mastodon/chart -f ./local-values.yml

The first time I ran this it failed to start most of the Mastodon pods with a bunch of errors around the PostgreSQL user password. I tracked it down to a recent Pull Request so I raised a new issue and a matching PR to fix things. This got merged really quickly which was very nice to see.

Once I’d modified the helm chart templates a little it deployed cleanly.

Proxying

Next up was setting up my Internet facing web proxy. The Ingress controller on my Kubernetes cluster is not directly exposed to the internet so I needed to add an extra layer of proxying.

First up was to setup an new host entry and renew my LetsEncrypt certificate with the new SAN entry.

location / {
    proxy_pass https://kube-three.local;
    proxy_ssl_verify off;
    proxy_ssl_server_name on;
    proxy_redirect off;
    proxy_read_timeout 900;
    proxy_set_header Host $http_host; # host:$server_port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_http_version 1.1;
}

It was important to make sure that the proxy target was over https otherwise the mastodon web app would trigger a redirect loop. the proxy_ssl_verify off; is because I’m using the default https certificate on the kubernetes ingress controller so it will fail the hostname check.

The other bit that needed doing was to add a proxy on the hardill.me.uk server for the .well-known/webfinger path to make sure user discovery works properly.

location /.well-known/webfinger {
    return 301 https://bluetoot.example.com$request_uri;
}

First Login

Now all the proxying and https setup is complete I can point my browser at https://bluetoot.hardill.me.uk and I get the intial sigin/signup screen.

As part of the configuration I created an admin user (fat_controller) but didn’t have a way to login as I don’t know what the generated password is. I tried follwing the lost password flow but couldn’t get it to work so followed the documentation about using the admin cli to do a password reset. I did this by using kubectl to get a shell in the mastodon web app pod in the cluster.

kubectl -n mastodon exec -it mastodon-web-5dd6764d4-mvwnl /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
mastodon@mastodon-web-5dd6764d4-mvwnl:~$ RAILS_ENV=production bin/tootctl accounts modify fat_controller --reset-password
OK
New password: fc20b54eddb312bf9462abfec77a27d8

I could then log in as the administrator, update things like the contact email address, change the default user creation to require approval for new users.

What next?

I’m not sure if I’m going to keep this instance yet. At the moment it’s a testing environment and I’m not sure yet how much I’m going to actually use Mastodon. A lot will depend on what Mr Musk decides to do with his new play thing and if/when the communities I follow move.

But I will probably play with writing a few bots for some of the home automation things I have on the go, e.g. as I write this I’m waiting for a WiFi CO2 monitor to be delivered. Having a bot that Toots the level in my office sounds like a good first project.

Twitter2MQTT bridge

The recent switching off of Basic Auth by Twitter meant I had to rework some of the applications I have running at home. One of these application bridges Twitter to my local MQTT broker so that any DMs and mentions get published to two topics and also a topic that updates my status to what ever is posted.

The original version of the application just made basic http requests and parsed the XML responses, rather than just try and bolt on OAuth support I thought I would be better to start again and use one of the existing libraries that drive the Twitter APIs. There is a list of libraries for different languages http://dev.twitter.com/pages/libraries, I had a bit of a look at a couple of them and settled on Twitter4J.

In order to use OAuth you need to register your application with Twitter, you can do that here http://twitter.com/apps/new. Once registered you will get Consumer Key and a Consumer Secret. Because Twitter are using these keys to help to cut off spammers, keys need to be kept secret in order to prevent spammers knocking legitimate applications off-line, If you want to build the code attached here you will need to apply for your own key.

Before you can use a Twitter application you need to authorise it to act on your behalf, this is a 3 stage process.

  1. The application creates a URL based on it’s Consumer Key and Consumer Secret.
  2. The user follows the URL and signs into Twitter and is then asked if they want to allow the application to access on their behalf. If they allow the application then Twitter supplies a PIN.
  3. The user passes the PIN to the application which uses this to retrieve a Token, This Token is used to authenticate the application when ever it needs to act on the users behalf.

Twitter Application Authentication
Twitter Application Authentication

To do this with Twitter4J you need to do something like this:

    Twitter twitter = new TwitterFactory().getInstance();
    twitter.setOAuthConsumer(consumerKey, consumerSecret);
    try {
        RequestToken requestToken = twitter.getOAuthRequestToken();
        AccessToken accessToken = null;
        BufferedReader reader = new BufferedReader(
                                       new InputStreamReader(System.in));
        while (accessToken == null) {
            System.out.println(requestToken.getAuthorizationURL());
            System.out.println("Please follow the URL above," + 
                      " enter the PIN provided and then press Enter");
            String pin = "";
            try {
                pin = reader.readLine();
            } catch (IOException e) {
                e.printStackTrace();
            }
            accessToken = twitter.getOAuthAccessToken(requestToken, pin);
        }
        String token = accesToken.getToken();
        String secret = accesToken.getTokenSecret();
    } catch (TwitterException e) {
        e.printStackTrace();
    }

The token and the secret should be stored so the user doesn’t need to authorise the application each time it starts. Now the application is authorised you can post status updates and access the DM and Mention streams like this:

    List<DirectMessage> dms = twitter.getDirectMessages();
    List<Status> mentionsList = twitter.getMentions();
    twitter.updateStatus(newStatus);

Now that I can access the updates I need to publish them to topics and listen for when a new status is published. There is a Java library for accessing an MQTT broker provided by IBM known as the IA92 package. This package provides both J2SE and J2ME versions of the libraries. To create a connection to the broker

    IMqttClient client = null;
    try {
        client = MqttClient.createMqttClient("tcp://brocker.loc:1883"
                                                            , null);
        client.connect(clientID, true, (short) 100);
        client.subscribe({"Twitter/send"}, {1});
        client.registerSimpleHandler(new MqttSimpleCallback() {
            public void connectionLost() throws Exception {
            }

            public void publishArrived(String topic, byte[] payload, int qos,
                            boolean retained) throws Exception {
                twitter.updateStatus(new String(payload));
            }
        });

    } catch (MqttException e) {
        e.printStackTrace();
    }

Each client needs a unique id, or the older client will be kicked off when the new one collects, if you want to run multiple versions of the bridge then it may be a good idea to append the OAuth access token or the twtter screen name to the clientID.

Taking all that and knocking some of the rough edges off I have an app that will publish DMs on Twitter/<screenName>/dm & mentions on Twitter/<screenName>/mention and listens on Twitter/<screenName>/send for status updates to publish. There is a second app which subscribes to the mention and DM topics and forwards these messages on to the topic that sends SMS messages to my phone.

Next on the list of additions is a filter that will forward photos attached to tweets as MMS messages and probably some support for bridging search terms as well.

You can download my code from here, remember you will need your own Consumer Key and Consumer Secret to build it yourself, but there is a pre-built version with a valid key included.

Twitter2MQTT.tar.gz

Resources