add mastodon package and module #60788

ghost · 2019-05-02T18:37:43Z

Motivation for this change

This module can already be used to set up a fully functional mastodon instance, but there is still a lot to do:

Discuss if moretea/yarn2nix should be added to nixpkgs again
Don't force users to store secrets in nix store
Improve documentation of module options
Add mastodon user and group ids
Add meta information to the mastodon package
Investigate if it makes sense to allow enabling and disabling the three services seperately, similar to the kubernetes module

Some things would be nice to have but are not strictly required for a first version in my opinion:

Write some tests
Investigate if building streaming, sidekiq and web services seperately would be possible and advantageous
Add more advanced options, i.e. the number of sidekiq threads, support for S3 storage backend, ...
Package mastodon tools like tootctl in a way that makes them easy to use

Any feedback is appreciated :-)

Things done

ghost · 2019-05-02T22:32:24Z

Of course we are facing the same IFD issue with yarn2nix as with riot-desktop and #59111.
It seems like the only solution for now is including yarn.nix and package.json in nixpkgs.

ghost · 2019-05-02T23:55:55Z

Unfortunately it seems like this is a dead end:
moretea/yarn2nix uses builtins.fetchGit, because the yarn.lock file does not contain hashes for git dependencies. builtins.fetchGit doesn't work on Hydra/OfBorg, because it runs at eval-time in restricted mode and network access is not allowed.

I will try to use https://github.com/Profpatsch/yarn2nix tomorrow, which hashes the git dependencies when creating the nix expression, and then uses pkgs.fetchgit. Since we have to include a pregenerated Nix expression for the dependencies anyways (otherwise we get IFD problems), this is a small loss.

alyssais · 2019-05-03T00:05:28Z

Can I do ...? @GrahamcOfBorg eval

As I understand it, you should be able to eval, but not build, unless you're a known or trusted user, in OfBorg terminology.

ghost · 2019-05-03T00:13:58Z

Can I do ...? @GrahamcOfBorg eval
As I understand it, you should be able to eval, but not build, unless you're a known or trusted user, in OfBorg terminology.

That's what I concluded too, but as it says in the OfBorg readme, there is no reason to call eval manually since it happens automatically.

ghost · 2019-05-03T14:24:51Z

After trying to integrate profpatsch/yarn2nix into nixpkgs, I give up on that. There are multiple broken Haskell packages required for profpatsch/yarn2nix to run.

My new plan is to add support for Hydra-enabled git dependencies to moretea/yarn2nix.

ghost · 2019-05-03T15:54:14Z

Hooray /o/

aanderse

I left a few comments about the nixos module which I hope you find helpful. Feel free to use or ignore as you see fit.

nixos/modules/misc/ids.nix

nixos/modules/services/web-apps/mastodon.nix

ghost · 2019-05-06T07:18:21Z

Thanks, I think I have implemented all your suggestions.

zimbatm · 2019-05-06T11:32:44Z

I don't know of a good way to add yarn2nix without forking the code base. When we had it in the tree it started diverging quickly with both sides receiving patches. Before adding it back we need to decide on how to resolve that issue. A few possibilities:

Add yarn2nix as a git submodule to nixpkgs, probably doesn't work with hydra
Use git substree to keep yarn2nix in sync. This needs a bit of a wrapper scripts action.
Archive upstream, declare nixpkgs to hold the canonical version.
?

ghost · 2019-05-06T11:52:56Z

I don't know of a good way to add yarn2nix without forking the code base. When we had it in the tree it started diverging quickly with both sides receiving patches. Before adding it back we need to decide on how to resolve that issue. A few possibilities:

Add yarn2nix as a git submodule to nixpkgs, probably doesn't work with hydra

Use git substree to keep yarn2nix in sync. This needs a bit of a wrapper scripts action.

Archive upstream, declare nixpkgs to hold the canonical version.

?

I agree it has to be discussed and well-thought-through.
Of the solutions you suggested, I prefer 3., but I have another idea:

Actually yarn2nix can be split into two parts, similar to bundix and bundlerEnv.

One part is responsible for generating the nix expression and is maintained in a seperate repository. This command line tool can be provided in nixpkgs just like all other tools with fetchFromGitHub.
The other part contains everything for building a node_modules folder from the generated nix expression and is kept in nixpkgs.

nixos-discourse · 2019-05-07T15:26:38Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/has-anyone-ran-mastodon-on-nixos/1440/9

nyanloutre · 2019-05-07T16:06:31Z

Thanks for the work, I will test this on my server as soon as possible

alyssais · 2019-05-07T17:19:39Z

> I don't know of a good way to add yarn2nix without forking the code base. When we had it in the tree it started diverging quickly with both sides receiving patches. Before adding it back we need to decide on how to resolve that issue. A few possibilities: > > 1. Add yarn2nix as a git submodule to nixpkgs, probably doesn't work with hydra > 2. Use `git substree` to keep yarn2nix in sync. This needs a bit of a wrapper scripts action. > 3. Archive upstream, declare nixpkgs to hold the canonical version. > 4. ? I agree it has to be discussed and well-thought-through. Of the solutions you suggested, I prefer 3., but I have another idea: Actually yarn2nix can be split into two parts, similar to bundix and bundlerEnv.

I like the sound of this.

nyanloutre · 2019-05-07T18:49:48Z

My server is on 19.03 so I will need a bit of work to backport this it seems

ghost · 2019-05-07T18:51:09Z

My server is on 19.03 so I will need a bit of work to backport this it seems

It shouldn't be too much, if you have any questions let me know

nixos/modules/services/web-apps/mastodon.nix

ghost · 2019-05-10T20:57:25Z

My server is on 19.03 so I will need a bit of work to backport this it seems

I just tried to build the package with 19.03. You only have to cherry-pick #60693 and somehow the hashes of the two ruby dependencies that are pulled from git changed depending on what version of nixpkgs I used, so I needed to change those.

kampka · 2019-05-10T21:31:18Z

Can you please make sure the state directory is configurable?
It would help immensely for keeping control over user generated content, such as avatars, media, etc. which needs to be backed up regularly.
A typical name for this option in nix services is dataDir which could easily default to /var/lib/mastodon.

Thanks for your effort. 👍

aanderse · 2019-05-10T21:38:35Z

Can you please make sure the state directory is configurable?
It would help immensely for keeping control over user generated content, such as avatars, media, etc. which needs to be backed up regularly.
A typical name for this option in nix services is dataDir which could easily default to /var/lib/mastodon.

Thanks for your effort. +1

@kampka can you please elaborate on what backup and control needs you have that won't work with /var/lib/mastadon?

aanderse · 2019-10-21T00:14:51Z

nixos/modules/services/databases/postgresql.nix

@@ -22,6 +22,16 @@ let
      ${cfg.extraConfig}
    '';

+  ensureDatabasesOpts = {


@petabyteboy you might be interested in joining our discussion about whether extending the mysql and postgresql database provisioning options is appropriate for NixOS or not.

I see the point of that discussion, but to be honest I don't care. I don't need this functionality, I just added it because people asked for it and I will happily remove it when there's a consensus that we don't want this kind of functionality.
Without this extension of the postgresql module I can not load the schema on creation of the database, which is something multiple people had asked for.

ghost · 2019-10-21T07:41:29Z

... that was me trying a button without a label and clicking it twice because the UI didn't respond.

So far I think I have addressed all comments other than comments that contradict eachother (discussions on other topics). From my side this is ready to go. Sure, I can probably work on this for another 5 months, but this is good enough to be included in nixpkgs in my view.

aanderse

I have lost track... what was the decision on node and yarn? Introduced back into nixpkgs?

Out of people who are following this thread and have commit access: are there (m)any remaining holdups on this?

I'll have to insist at this point the modifications are dropped on postgresql simply because there is a larger conversation happening that isn't resolved. Sorry to be a hassle.

Also... in the future keep in mind that you have been exceptionally accommodating to many people's requests in this PR. Someone stating a feature they think would be nice is not the same thing as a review that is critiquing code that has problems associated with it, or required missing features that you have overlooked. So don't feel bad to push back of you don't see value in a feature because after all it is you doing the work. 😄

nixos/modules/services/web-apps/mastodon.nix

ghost · 2019-10-21T10:38:00Z

I have lost track... what was the decision on node and yarn? Introduced back into nixpkgs?

yarn2nix was introduced back into nixpkgs as yarn2nix-moretea in #60429. node2nix was never really mentioned in this discussion, rather there were multiple competing projects with the name yarn2nix by Profpatsch and moretea.
There were concerns about the maintainability of yarn2nix-moretea in nixpkgs, because in the past we had lost track of upstream. As I understand this is now solved by an update script.

Out of people who are following this thread and have commit access: are there (m)any remaining holdups on this?

I'll have to insist at this point the modifications are dropped on postgresql simply because there is a larger conversation happening that isn't resolved. Sorry to be a hassle.

Okay, I will do the following:

I will create a branch on my fork to preserve the current state with postgresql changes.
I will remove the postgresql changes and all functionality that depends on it on this branch, even though some people will be unhappy about using the migrations on an empty database

Also... in the future keep in mind that you have been exceptionally accommodating to many people's requests in this PR. Someone stating a feature they think would be nice is not the same thing as a review that is critiquing code that has problems associated with it, or required missing features that you have overlooked. So don't feel bad to push back of you don't see value in a feature because after all it is you doing the work.

Except some people do state their (hard to fulfill) wishes as if they were requirements, quoting a comment from this PR: "i would feel bad about merging this if there isn't first-class support for custom forks of mastodon"

ghost · 2019-10-21T10:56:37Z

I have removed the postgresql changes and everything that depended on it. The version with those changes can be found in the branch feature/mastodon-postgres on my fork.

ghost · 2019-10-21T11:19:47Z

Note that the webserver test will not pass without the changes in the postgresql module, because manual intervention is required to initialize the database.

ghost · 2019-10-21T11:29:07Z

Actually it does work by using the migration chain, it's just quite slow.

ashkitten · 2019-10-21T15:57:39Z

to clarify my wishes about custom forks - i meant that more as a blanket statement about making it easy to update mastodon to any release, custom fork or not, because it seemed difficult to assemble all the required components and potentially easy to miss something. i was just advocating for an update script, really. sorry for confusion

happy-river · 2019-11-15T04:47:41Z

Actually it does work by using the migration chain, it's just quite slow.

It's too slow for the amount of patience I have, so I found a way to test whether a postgresql database has no tables yet and use rake db:schema:load instead of rake db:migrate in that case. I've also ported the two mastodon tests to the new Python test driver. Both commits can be found in my mastodon-python-tests branch, which is based on this pull request rebased onto a recent master.

happy-river · 2019-11-15T05:01:08Z

nixos/modules/services/web-apps/mastodon.nix

+          keypair=$(bin/rake webpush:generate_keys)
+          echo $keypair | grep Private | sed 's/^Private -> //' > ${cfg.vapidPrivateKeyFile}
+          echo $keypair | grep Public | sed 's/^Public  -> //' > ${cfg.vapidPublicKeyFile}


This isn't working as expected. With a configuration that doesn't specify the keys, it puts both keys on a single line into both /var/lib/mastodon/secrets/vapid-private-key and vapid-public-key:

Generated VAPID keypair: Public -> BC8xMKgr9Vtx9Sw2RHxnC__9d61l308waEeFS9XRyhtbbH8m5dsUyQ1S-2rxjEyvAfnOdf8AmzbDLHbAu-fCv4g= Private -> Noj-8wTsdT8zSY1ooBpbWGLJ3IPLj4WWkbSYUKikTr8=

Should be fixed, please confirm

Yes, it's fixed.

ghost · 2019-11-15T07:59:47Z

I have pushed your changes to my branch. Thanks!

pkgs/servers/mastodon/default.nix

pkgs/top-level/all-packages.nix

@happy-river

nixos/mastodon: add webserver test nixos/mastodon: add package tests Tests and fixes contributed by @happy-river, thanks!

happy-river · 2019-12-29T21:19:51Z

nixos/modules/services/web-apps/mastodon.nix

+        DB_PASS=$(cat ${cfg.database.passwordFile})
+        SMTP_PASSWORD=$(cat ${cfg.smtp.passwordFile})


To support passwords with spaces:

DB_PASS="$(cat ${cfg.database.passwordFile})" SMTP_PASSWORD="$(cat ${cfg.smtp.passwordFile})"

happy-river · 2019-12-29T21:24:00Z

nixos/modules/services/web-apps/mastodon.nix

+                join pg_namespace s on s.oid = c.relnamespace \
+                where s.nspname not in ('pg_catalog', 'pg_toast', 'information_schema') \
+                and s.nspname not like 'pg_temp%';" | sed -n 3p` -eq 0 ]; then
+          SAFETY_ASSURED=1 rake db:schema:load


Suggested change

SAFETY_ASSURED=1 rake db:schema:load

SAFETY_ASSURED=1 rake db:schema:load

rake db:seed

Mastodon 3.0.0 added an instance actor account which must be seeded into a newly created production database.

happy-river · 2019-12-29T21:34:25Z

nixos/modules/services/web-apps/mastodon.nix

+          Path to file containing the secret key base
+
+          Can be generated by running
+          cd $(nix-instantiate --eval '&lt;nixpkgs&gt;' -A mastodon.outPath); bin/rake secret


nix-instantiate --eval '<nixpkgs>' -A mastodon.outPath just tells you what the path of the Mastodon package is. It doesn't actually build the package, so if the package isn't already in the Nix store, as might be the case if you're just getting started setting up this module, this command won't work.

The best replacement I've come up with is: nix build -f '<nixpkgs>' mastodon; cd result; bin/rake secret

See my branch 60788-tootctl for a commit where I've made this change in the several places where it is needed, added mention that the keys will be generated automatically if they are not supplied, and also edited most of the other option descriptions for formatting and clarity.

happy-river · 2019-12-29T22:28:03Z

I've now successfully created a Mastodon instance using this module and NixOps and a configuration that looks like this:

  tooter = { config, pkgs, ... }: {
    services.mastodon = {
      enable = true;
      configureNginx = true;
      localDomain = "example.com";
      smtp = {
        host = "smtp.example.org";
        port = 587;
        user = "user@mail.example.com";
        fromAddress = "Administrator <admin@mail.example.com>";
      };
    };
    networking.firewall.allowedTCPPorts = [ 80 443 ];
  };

I deployed it, copied my SMTP password over, restarted it, used its web interface to create an account, and then needed to use tootctl to promote the account to an administrator. But tootctl needs Mastodon's environment variables to be set up correctly, or it won't work. My solution to that problem can be found in my 60788-tootctl branch. It creates a shell script mastodon-env which sets up the environment for another command. With that done, making an administrator account became:

$ nixops ssh -d mastodon tooter
[root@tooter:~]# su - mastodon -s /bin/sh
[mastodon@tooter:~]$ mastodon-env tootctl accounts modify myaccount --role admin

This works, but I'm open to better solutions. One feature that might be added to mastodon-env is to make it optionally cd to the Mastodon package directory, which would be helpful if you want to use it to run any rake tasks.

ghost · 2020-01-01T12:45:41Z

This project has caused me too much stress and I'm not really interested in improving it any further. Maybe @happy-river can open a PR and continue this.

Miaourt · 2020-01-01T12:47:01Z

Don't worry, Take care of you @petabyteboy !

happy-river · 2020-01-02T18:11:41Z

Thanks for everything you've done on this @petabyteboy ! I will continue to work on this and will create a new PR in a few days.

bqv · 2020-01-29T19:12:05Z

I actually got this working with a few monkeypatches. Been running it for almost a month. Seems like it shouldn't need much more work.

happy-river · 2020-01-30T05:26:38Z

I've just created #78810 which contains this pull request with improvements.

ofborg bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels May 2, 2019

ghost marked this pull request as ready for review May 2, 2019 22:03

ghost requested a review from infinisil as a code owner May 2, 2019 22:03

ghost mentioned this pull request May 3, 2019

error: Use upstream https://github.com/moretea/yarn2nix nix-community/yarn2nix#98

Open

ofborg bot added 2.status: merge conflict This PR has merge conflicts with the target branch and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels May 3, 2019

ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels May 3, 2019

Mic92 mentioned this pull request May 4, 2019

[RFC 0040] "Ret-cont" recursive Nix NixOS/rfcs#40

Closed

aanderse reviewed May 4, 2019

View reviewed changes

lilyball reviewed May 7, 2019

View reviewed changes

nixos/modules/services/web-apps/mastodon.nix Outdated Show resolved Hide resolved

ghost requested a review from thoughtpolice as a code owner October 20, 2019 16:38

ofborg bot added 2.status: merge conflict This PR has merge conflicts with the target branch and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels Oct 20, 2019

aanderse reviewed Oct 21, 2019

View reviewed changes

ghost requested review from alyssais and aanderse October 21, 2019 07:38

aanderse reviewed Oct 21, 2019

View reviewed changes

nixos/modules/services/web-apps/mastodon.nix Show resolved Hide resolved

nixos/modules/services/web-apps/mastodon.nix Outdated Show resolved Hide resolved

nixos/modules/services/web-apps/mastodon.nix Outdated Show resolved Hide resolved

happy-river reviewed Nov 15, 2019

View reviewed changes

kampka mentioned this pull request Dec 6, 2019

Add trilium server and module #75047

Merged

10 tasks

happy-river reviewed Dec 10, 2019

View reviewed changes

pkgs/servers/mastodon/default.nix Outdated Show resolved Hide resolved

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

Milan Pässler added 2 commits December 14, 2019 22:27

mastodon: init at 3.0.1

c33dcb1

nixos/mastodon: init

b9dc3ee

nixos/mastodon: add webserver test nixos/mastodon: add package tests Tests and fixes contributed by @happy-river, thanks!

happy-river reviewed Dec 29, 2019

View reviewed changes

ghost closed this Jan 1, 2020

alyssais mentioned this pull request Jan 30, 2020

New nixpkgs committers requests #50105

Closed

happy-river mentioned this pull request Jan 30, 2020

Add Mastodon package and module #78810

Closed

12 tasks

This pull request was closed.

		DB_PASS=$(cat ${cfg.database.passwordFile})
		SMTP_PASSWORD=$(cat ${cfg.smtp.passwordFile})

	SAFETY_ASSURED=1 rake db:schema:load
	SAFETY_ASSURED=1 rake db:schema:load
	rake db:seed

add mastodon package and module #60788

add mastodon package and module #60788

Conversation

ghost commented May 2, 2019 • edited by ghost Loading

Motivation for this change

Things done

ghost commented May 2, 2019

ghost commented May 2, 2019 • edited by ghost Loading

alyssais commented May 3, 2019 via email

ghost commented May 3, 2019

ghost commented May 3, 2019

ghost commented May 3, 2019

aanderse left a comment

Choose a reason for hiding this comment

ghost commented May 6, 2019

zimbatm commented May 6, 2019

ghost commented May 6, 2019 • edited by ghost Loading

nixos-discourse commented May 7, 2019

nyanloutre commented May 7, 2019

alyssais commented May 7, 2019 via email

nyanloutre commented May 7, 2019

ghost commented May 7, 2019

ghost commented May 10, 2019

kampka commented May 10, 2019

aanderse commented May 10, 2019

aanderse Oct 21, 2019

Choose a reason for hiding this comment

ghost Oct 21, 2019 • edited by ghost Loading

Choose a reason for hiding this comment

ghost commented Oct 21, 2019

aanderse left a comment

Choose a reason for hiding this comment

ghost commented Oct 21, 2019

ghost commented Oct 21, 2019

ghost commented Oct 21, 2019

ghost commented Oct 21, 2019

ashkitten commented Oct 21, 2019

happy-river commented Nov 15, 2019

happy-river Nov 15, 2019

Choose a reason for hiding this comment

ghost Dec 14, 2019

Choose a reason for hiding this comment

happy-river Dec 29, 2019

Choose a reason for hiding this comment

ghost commented Nov 15, 2019

happy-river Dec 29, 2019

Choose a reason for hiding this comment

happy-river Dec 29, 2019

Choose a reason for hiding this comment

happy-river Dec 29, 2019

Choose a reason for hiding this comment

happy-river commented Dec 29, 2019

ghost commented Jan 1, 2020

Miaourt commented Jan 1, 2020

happy-river commented Jan 2, 2020

bqv commented Jan 29, 2020

happy-river commented Jan 30, 2020

ghost commented May 2, 2019 •

edited by ghost

Loading

ghost commented May 2, 2019 •

edited by ghost

Loading

ghost commented May 6, 2019 •

edited by ghost

Loading

ghost Oct 21, 2019 •

edited by ghost

Loading