Stores Objects (csv, parquet, JSON, etc) with a max size of 5 TB in Buckets with a unique name per region. To access data, network endpoints Access points attached to buckets can be created.
Amazon S3 main purpose is to have storage capacity independently from computing.
-
Lifecycle Buckets can be moved through different S3 Storage Tiers where Standard is for general purposes, going to
Standard-Infrequent Access (IA),One Zone-Infrequent AccessorIntelligent Tiering, which automatically move data across storage classes dependent on its use. For long-term achieve,Amazon Glacieris normally used as data is not available in real time and it charges per retrieval. These different S3 Storage Classes give the option to balance availability and price. With a Lifecycle rule, buckets or selected objects with aprefixortagscan transition from Standard (frequent used) to less frequent followingtransaction actionsand can be deleted withexpiration actions. -
Data partitioning A bucket can have partitions (folders) which should be defined according to how the data is query, i.e year, month, date. When data is extracted using
Glue crawlers, it will create partitions depending on how the data is organized in S3. -
Security
- Encryption: data in transit is secured using Secure Sockets Layer (SSL) or client-side encryption. Data at rest can be encrypted in the client of in the server using:
- SSE-S3: Server Side managed by AWS
- SSE-KMS: Server Side with key managed by the Key Management Service which can be monitored by the user.
- Access: Security policies for Buckets and Access Points can be created at different levels.
- User Base: Identity and Access Management (IAM) policies
- Resource Base: Object Access Control List
- Networking: VPC (Virtual Private Cloud) Endpoint Gateway
- AWS Documentation
- WITTIG, Michael; WITTIG, Andreas; WHALEY, Ben. Amazon web services in action. Second Edition. Manning, 2018.