Add support for an index option when querying, other bugfixes

This minor version release includes one new feature and a bug fix...

New feature:

The new feature is the ability to pass an :index option when using an Ecto query against a secondary index; the value of the option will be either the string or atom name of the secondary index on which the query should be executed. For example:

import Ecto.Query

from(u in User, where: u.first_name == "foo" and u.last_name in ["bar", "baz"])
|> Repo.all(index: :last_name_first_name)

Why is this important? For the most part, the adapter is very good at inferring the "best" index from the fields presented in the query, and you can usually trust its judgement. However, there are some cases where the particular fields in the query may result in some ambiguity that could result in an error or inconsistent behaviors.

Take the example above - if the User table has two composite secondary indexes, :first_name_last_name and :last_name_first_name, the adapter won't be able to discern which choice would be best; however, selecting the former would cause the query to raise an error, as DynamoDB does not support :in operations against a range key... although the adapter might choose the correct index on its own, that behavior is not guaranteed. By explicitly providing the name of the index to use, you can be sure that the adapter will query correctly.

Bug fix:

Previously, queries that used the :in operation against partial primary and secondary composite keys (hash-only) were failing; this has been resolved, those queries will now work.

Use Hex source for ex_aws_dynamo

The dep ex_aws_dynamo has been updated to include support for setting a table's billing mode, and the changes have been published to Hex. With this patch release, we revert back to using ex_aws_dynamo's Hex source, rather than the fork of that library that we were maintaining.

Fix bug where partial secondary index may override hash-only index

This patch fixes a bug related to the adapter's choice of optimal index when querying. In previous versions, a query on single indexed field could result in a query being executed against a partial secondary index, which had the potential for data to be left out of a query's return.

For example, if a user table has two indexes, first_name (a hash-only index) and first_name_last_name (a composite index), queries on the first_name field could end up using the composite index, where the hash-only index would have been sufficient. Furthermore, if the data model allowed last_name as an optional field, the incorrectly chosen query could result in missing data, as it would fail to return any records where the range key would have a nil value.

Queries should now reliably choose the optimal index.

Support range queries on composite primary keys

1.1.1

increment patch version

Improved local migration support, dynamodb_local config option

This release adds support for a new boolean configuration option, :dynamodb_local, which can be used to specify whether the app is running against live production DynamoDB or the local development version. Production and development DynamoDB are not identical replicas - in order to make the experience of working against both versions as similar as possible, we have introduced this variable, which defaults to false, implying that it is running against production. We recommend that users should set dynamodb_local: true in their config files for any environments that they will be running against local DDB.

Related to that, we have introduced handling for a special scenario that may occur when there are disagreements between production and local DDB. Here's a brief synopsis of what happened and how we opted to handle it:

Several months ago, we wrote a migration to create a new provisioned table. Time went by and more migrations were added to the migration history. Eventually, an administrator changed the original table's billing mode from provisioned to on-demand via the AWS DynamoDB dashboard. A short while later, a dev needed to write a migration to add a new index to this table. Since the production table was now on-demand, the migration couldn't specify provisioned throughput for the index; the migration ran successfully against the production table, but failed locally (the process will hang without raising an error message), since the local table was still set to provisioned, as per its original migration. There is no local analog of the DDB dashboard (yet...) and the dev did not want to change the migration history (and then rollback and re-run all of the migrations), so we introduced special handling for such a situation based around the dynamodb_local config setting to help with situations where the production and local databases may fall out of sync...

By setting dynamodb_local: true under the adapter's configuration in config/dev.exs (or wherever needed), the adapter will quietly add default provisioned throughput ([1,1]) to an index if the table is provisioned but the index's migration file does not specify it. This also mirrors what seems to be a related default behavior of local dev DDB - if you create an on-demand table and then try to add an index with specified throughput, local DDB will simply ignore the throughput (production will raise an error); however, if you create a provisioned table ad try to add an index without specified throughput, local DDB will hang (again, production will raise)... this config option will make those local behaviors more uniform.

This release also changes the link for the fork of the ex_aws_dynamo dependency that is currently in use to its HTTPS source, as opposed to SSH.

Add support for on-demand billing mode

This release adds support for DynamoDB's "pay-per-request" table billing mode. As part of this update, we have removed a feature where we provided default values of [1,1] for throughput for provisioned tables - see the upgrade guide for more information.

We also recommend updating to the latest version of the local development version of DynamoDB (1.11.477 at the time of this release), which supports the billing mode options.

Bugfixes for certain update_all cases and 'all/in/and in' queries for composite primary keys

This patch release resolves two bugs that were observed -

• attempts to update_all when querying an index for a list of variable values would fail when attempting to set a value as nil; updates to non-nil values would succeed, but were passing irrelevant data into the query process.

• 'all/in/and in' queries were failing on composite primary keys - for example:

query = from bp in BookPage, where: bp.id in ["page:test-3", "page:test-4"] and bp.page_num in [1, 2]
Repo.all(query)

Add support for 'all... in...' queries on global secondary indexes

This release adds support for 'all... in...' queries on hash keys of hash key and composite global secondary indexes. For example:

TestRepo.all(from p in Person, where: p.email in ^emails)

where email is a hash key global secondary index, or

TestRepo.all(from p in Person, where: p.first_name in ^first_names and p.age < 50)

where first_name and age are a composite key global secondary index.

Update README

This release fixes a few typos in the README file, and includes proper documentation for installing the most up-to-date version of the adapter.

Handle .all() queries for more than 100 records.

DynamoDB restricts the number of items that can be requested during a BatchGetItem operation to 100; in previous versions of this adapter, .all()/in queries for more than 100 records would fail. In this release, we provide basic handling to batch requests into 100-record chunks, allowing for an unlimited number of records to be queried for.

See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html for reference.