RFC: Phone Number Management

[matteosb]

Proposal

To phone number management to the Spoke admin interface similar to phone number provisioning in the Warren fork. We would initially provide support for twilio and fakeservice, but leave the design open to integration with more SMS providers. Unlike the Warren version, the phone numbers will belong to individual organizations, though we may want to eventually add an option to allow transfer between organizations for single-tenant deployments.

Phone number management will be decoupled from how numbers are used, but we will start by integrating it with per-campaign messaging services introduced in #1495 . In the future, we may want to integrate it with per-organization messaging services and other SMS providers.

Twilio numbers provisioned through Spoke will have a configurable twilio voice url, resolving issue #1464

Motivation

Managing numbers in Twilio is a tedious manual process and a barrier to adoption for newcomers and non-technical folks who may be discouraged by a CLI like switchboard-twilio. This gets even more complicated for large organizations that need to scale beyond a single Messaging Service, which Spoke doesn't currently allow. #1495 allows organizations to scale but would require a lot of manual work without the additions described here. Some organizations may also want the added control over what area codes they use for individual campaigns rather than relying on Twilio's geomatch.

Detailed Design

The Warren fork README has a fairly detailed explanation of phone number management combined with per-campaign messaging services: https://github.com/Elizabeth-Warren/Spoke#phone-number-provisioning.

We will depart from the Warren design in a few key ways:

• Phone numbers will belong to an organization, not the entire Spoke instance
• The phone number inventory table will support multiple SMS providers
• Phone numbers will only have two states: "ALLOCATED" or "AVAILABLE". We may want to add a phone number reservation system like the Warren fork, but it isn't necessary and can be added later.

The work will be divided into three tasks:

• Merge the core inventory management feature (admin screen to buy numbers and resolver to see what phone numbers the organization has available) Phone number inventory management #1525 @matteosb
• Merge per-campaign Twilio messaging services Allow campaign to have its own messaging service #1539 @hiemanshu
• Integrate per-campaign Twilio messaging service with the phone number inventory by allocating numbers to a campaign when it starts. @matteosb

Alternatives

See the discussion here. The two other options described are:

• On-the-fly phone number provisioning: rather than having a separate screen to buy numbers ahead of time, campaigns specify a list of preferred area codes and the application attempts to buy numbers in those area codes once the campaign is launched.
  • PROS: (1) fewer screens/simpler user experience, (2) closer to the experience P2P vendors provide
  • CONS: (1) doesn't allow for efficient phone number reuse, (2) less predictability around what area codes will be allocated to a campaign, (3) can't buy numbers ahead of time.
  • Note: These objections can be overcome by combining on-the-fly provisioning with a phone number inventory, so I could definitely see this being added in the future.
• 'Direct' management of Messaging Services, aka Phone Number Pools (rather than area codes) through the UI.
  • PROS: (1) even more efficient phone number re-use, (2) "sticky sender" persists between campaigns
  • CONS: (1) "stick sender" persisting between campaigns is not desirable for everyone, (2) considerably more complicated user experience, (3) Messaging Services are a Twilio-specific construct, whereas managing numbers and area codes feels more universal and could eventually be extended to other SMS providers.

Project Plan Information

Sponsor Organization (will test alphas/betas)

MoveOn

Developer allocated

@matteosb

Planned timeline

2 weeks - month depending on when #1478 and #1495 are merged.

Files/File-sections touched

• src/server/api/lib/twilio.js
• src/server/api/lib/fakeservice.js
• src/server/api/lib/service.js
• new queries and mutations
• new background job
• new page in the admin interface and a new section in the sidebar
• new organization setting

Once #1495 is merged:

• changes to startCampaign to have it transfer numbers to the campaign messaging service. To stay under the AWS lambda 30s timeout, the Warren campaign made this a background job
• loading indicator on AdminCampaignEdit when a campaign is starting.

Schema changes and migrations

• Adds a new table called owned_phone_number
• Modifies the job_request table to allow organization-scoped jobs

[schuyler1d]

This project is very exciting!

To repeat back the above to verify my understanding -- there will be an org-level group of phone numbers. When this feature is enabled, a message service will be created for each campaign, and then on starting (or reserving) a campaign, the numbers will be moved into that campaign. When the campaign is closed, it will move the numbers out of the message service back into the pool of available numbers?

Some comments (unorganized):

• While this may be a twilio-specific feature at first, I'd like to keep it potentially generic -- thus the table should be called owned_phone_number or bought_phone_number instead of mentioning twilio specifically. Maybe it should have a service column, as well? along with a nullable campaign_id column (that may or may not obviate the Warren schema's states). -- I think that will both allow management per-campaign, and allow future extensions of number management across campaigns, etc.
• Is it possible in the Campaign section where numbers are allocated to have this done automatically (based on contact area code counts?) -- An integration api for something like 'phone-allocator' could be picked per-campaign and possibly get called on save, startCampaign, and a hook on a new first-message?
  • I acknowledge that this might be over-engineering a framework, but it might help to define the boundaries of how this work can plug in, and allow others to write different strategies.
  • This might also help separate the code from the current 'default' (one message service per org/instance)
• Spend reporting:
  • Is there a good way to see a cost summary in the org-level view?
  • Are there other places we can track this -- either via the API or maybe a cost column in the phone number or campaign table? When a number is fully released do we delete it from the table?
• Permissions: agree that it should not be anyone that can start a campaign -- so maybe number buying needs to be an OWNER? But whatever is in the campaign view, necessasry to start a campaign would clearly need to allow ADMIN roles to do the action -- thus hopefully it wouldn't be overly detailed (choosing areacodes, etc).

[matteosb]

Added #1525, a proposed implementation of the inventory backend for twilio with a bunch of comments. It's waiting for #1495 to integrate with campaigns.

In the Warren fork, unstarted campaigns reserve numbers before starting to avoid races with other unstarted campaigns. This worked nicely but it did rely on a cron job to periodically release reserved numbers. As an alternative, we could omit the RESERVED state on phone numbers and just fail if startCampaign can't find the required numbers on start. I don't really have a preference.

[schuyler1d]

I feel like the phone_number.campaign_id solves both of those -- it reserves them (yes, do it before it starts). And then after some period of time, you can find reserved campaigns that are too old and clear the numbers then (because you can see if the campaign ever started).

But I suppose we could have another column that marks something like 'used' -- which actually 'locks' it to a campaign after the first message out with it?

[matteosb]

Right i think i see what you mean, the status column could be eliminated and instead derived by joining to campaign and maybe reserved_at for the reservation functionality, but I think having a status would help if we we tried to use this to manage org and global messaging services .

Answer to other questions/comments in order:

• I was thinking that we would add more tables (e.g. nexmo_phone_number) if we were to extend this but I can make the table more generic if you prefer it that way, however, I'd want to do some research into other providers to get the design right.
• Figure out what area codes to use based on zip/contact area code: yes I could see this - in principle it and other selection strategies could be built and plugged in client side to the AdminCampaignEdit page without touching the core logic. The backend just accepts a list of area codes and count of phone numbers for each. Generalizing beyond selection of area codes (random example: providing a strategy that bought numbers on the fly during messaging) would require a pretty major refactor and seems out of scope to me.
• Spend reporting: yes it looks like there is an api that would allow us to do this for Twilio, I haven't used it though.
• When a number is released from twilio it would have to be deleted from the table, i think we should add that to the management screen
• Permission: OWNER to buy, ADMIN to use the numbers sounds good to me. That's what I put in the PR.

[jeffm2001]

This all makes sense to me. Just want to flag that phone number reuse (in the vein of #1434) is something my team likes, but I don't think it's critical.

Also want to mention #1464. Not that it necessarily needs to be in the first version of this, but worth thinking about where it fits in.

[matteosb]

Updated the description to match some of the changes discussed here

[ibrand]

This should also close #1259

[Frydafly]

Hey folks! Am I wrong in thinking this is ready to be closed? I'm seeing the related PRs were merged. Really cool work!

[joemcl]

Hi Fryda thanks yeah sure but for next time FYI I prefer to close issues if I open them :-)