DIY Guide: Building a Cloud File Storage service on Amazon S3

So, you want to let your users upload files in your app? Awesome! With S3 it's simplier than you may expect.

Here's a DIY guide on how to get started building a file sharing app with S3:

1. Tree-Style Organization on S3

Think of your S3 buckets like a virtual tree. Start with the main bucket as your base (let's call it your service HQ), then create branches for different users or groups, and finally, store files as the leaves at the end of those branches.

my_s3_bucket

- user_1/
	- file_1.zip
	- file_3.zip
- user_2/
	- file_4.zip
	- file_5.txt			

This setup helps keep things tidy. Each user gets their own space, making it easy to find and manage their stuff.

2. Secure Uploads with Signed URLs

Forget the old way of uploading files to your server and then transferring them to S3. Nah, let's do it smarter. Use signed URLs, which are like secret passes granting temporary access to upload stuff directly to S3.

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { createPresignedPost } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({
  region: 'YOUR_REGION',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY_ID',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'
  }
});

// Generate a signed URL with specific permissions (e.g., upload)
const generateSignedUrl = async (bucketName: string, fileName: string, userId: string): Promise<string> => {
  const params = {
    Bucket: bucketName,
    Key: `${userId}/${fileName}`, // File path in the bucket
    ContentType: 'application/octet-stream', // File content type
    Metadata: {
      UserID: userId // Adding user ID as metadata
    }
  };

  const command = new PutObjectCommand(params);
  const signedURL = await createPresignedPost(s3, command, { expiresIn: 3600 });
  console.log(`Signed URL generated for ${fileName}: ${signedURL.url}`);
  return signedURL.url;
};

// Usage example
const signedURL = await generateSignedUrl('your-service-bucket', 'example.txt', 'user123');
console.log('Use this URL for secure file upload:', signedURL);

This trick keeps things secure and snappy. Plus, it lightens the load on your server – win-win!

3. Automation Magic with S3 Triggers

You know what's cool? S3 triggers. They're like little prompts that can automatically start actions whenever something happens in your storage. Use these triggers to make things happen right after a file gets uploaded or modified.

For instance, you can set up triggers to automatically run tasks like file processing or updating databases whenever something new lands in your S3 space.

Here’s an example of how to do this with Serverless and Node.js:

service: my-s3-trigger-service

provider:
  name: aws
  runtime: nodejs14.x
  region: YOUR_AWS_REGION
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:GetObject
        - s3:PutObject
      Resource: "arn:aws:s3:::your-service-bucket/*" # Replace with your S3 bucket ARN

functions:
  processFile:
    handler: handler.processFile
    events:
      - s3:
          bucket: your-service-bucket
          event: s3:ObjectCreated:*

// Lambda function triggered by S3 event
export const processFile = async (event: any): Promise<void> => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = event.Records[0].s3.object.key;

  console.log(`Processing file ${key} in bucket ${bucket}`);

  // Perform actions like file processing or database updates here
};

Pretty simple stuff.

S3 triggers are event-driven, meaning they respond to specific actions (like object creation, deletion, or modification) immediately. This real-time responsiveness ensures that associated logic or workflows are triggered instantly without the need for periodic checks or manual intervention.

4. Get Organized with Metadata in URLs

Wanna make searching and sorting files easier? Add some metadata to those signed URLs! Stick in things like user IDs or file categories when generating the URLs for uploads. This way, you can easily sort and find files later based on this added info. It's like adding tags to your files without the hassle.

import { S3Client } from '@aws-sdk/client-s3';
import { createPresignedPost } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({
  region: 'YOUR_REGION',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY_ID',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'
  }
});

const generatePresignedURLWithMetadata = async (
  bucketName: string,
  key: string,
  userId: string,
  contentType: string
): Promise<string> => {
  const params = {
    Bucket: bucketName,
    Key: key,
    ContentType: contentType,
    Metadata: {
      UserID: userId, // Adding user ID as metadata
      CustomProperty: 'CustomValue' // Additional custom metadata properties
      // Add more metadata properties as needed
    }
  };

  const command = createPresignedPost(s3, params);
  const presignedURL = await command;
  console.log(`Presigned URL generated for ${key}: ${presignedURL.url}`);
  return presignedURL.url;
};

// Usage example
const bucketName = 'your-service-bucket';
const objectKey = 'example.txt';
const userId = 'user123';
const contentType = 'text/plain';

const presignedURL = await generatePresignedURLWithMetadata(bucketName, objectKey, userId, contentType);
console.log('Use this URL for secure file upload:', presignedURL);

Building your storage space on Amazon S3 is all about having a solid structure, making things automatic, and keeping it secure without all the tech jargon. Follow these tips, and you'll have a robust storage setup that's efficient and easy to manage!