S3 object storage (AWS, Minio, Wasabi, Linode, etc.)

Lyftdata can read from and write to S3 compatible object storage (AWS, Minio, Wasabi, Linode, etc).

Create AWS S3 bucket

• Bucket and objects not public
• Block all public access: On
• ACLs are disabled. All objects in this bucket are owned by this account. Access to this bucket and its objects is specified using only policies.
• create IAM user and policy (see below)
• do not enable console access
• attach policy directly
• create policy in separate window
• attach policy to new user

AWS user policy for programmatic access

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::*BUCKETNAME",
                "arn:aws:s3:::*BUCKETNAME/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::*BUCKETNAME/*"
        }
    ]
}

Recommendations for files and folder structure

• each file should not be larger than 100MB-150MB (compressed gzip or parquet)
• Y/M/ or Y/M/D/ or Y/M/D/H/ folders