Store your MarketsFlare files on your own server with MaxIO
Every document you upload to MarketsFlare is encrypted and written to object storage. By default that's our managed storage, hosted in Switzerland on AWS in Zurich. If your team needs those files on infrastructure you run yourself, you can connect your own S3-compatible server instead, and your workspace's files will be stored there.
This guide uses MaxIO, an open-source S3 server written in Rust. It's a single binary with one data directory and no database, so it's simple to run and simple to back up. It's also a drop-in replacement for MinIO, whose open-source repository has been archived.
You'll start MaxIO with Docker, put HTTPS in front of it with Caddy, create an access key for MarketsFlare, and connect it in your workspace settings. Set aside about 20 minutes.
What you need
- A Linux server with Docker and enough disk for your documents.
- A domain name for it, such as
storage.example.com, with a DNS record pointing at the server. - Ports 80 and 443 open to the internet. Caddy uses them to get a certificate, and MarketsFlare connects on 443.
- The AWS CLI v2 on the server, for creating an access key in step 3.
- MaxIO 1.0.0 or later. This guide uses its access-key management, which earlier versions don't have.
- Access to Settings in your MarketsFlare workspace.
Start MaxIO
MaxIO won't start with its built-in default credentials, so begin by generating your own admin key on the server. Then start the container and check that it answers.
marketsflare bucket and answers its health check.export MAXIO_ACCESS_KEY="admin"
export MAXIO_SECRET_KEY="$(openssl rand -hex 32)"
docker run -d --name maxio \
--restart unless-stopped \
-p 127.0.0.1:9000:9000 \
-v maxio-data:/data \
-e MAXIO_ACCESS_KEY -e MAXIO_SECRET_KEY \
-e MAXIO_REGION=us-east-1 \
-e MAXIO_DEFAULT_BUCKETS=marketsflare \
ghcr.io/coollabsio/maxio:1.0.0
curl http://127.0.0.1:9000/maxio/health/liveA few of those flags matter later:
-p 127.0.0.1:9000:9000keeps MaxIO on localhost. Only Caddy will be reachable from the internet.MAXIO_DEFAULT_BUCKETScreates themarketsflarebucket on first start.MAXIO_REGIONis only a label, but every request MarketsFlare sends is signed with it, so the two have to match.us-east-1is MaxIO's default. We set it explicitly so you can see it.- The image tag is pinned. Upgrade on purpose, after reading MaxIO's upgrade notes.
Save the secret in your password manager (echo "$MAXIO_SECRET_KEY"). You'll need it again in step 3.
Put HTTPS in front
MaxIO only speaks HTTP, so a reverse proxy handles TLS. Caddy is the shortest path: it gets a certificate for your domain and renews it on its own. Install it, then replace /etc/caddy/Caddyfile with:
storage.example.com {
reverse_proxy 127.0.0.1:9000
}sudo systemctl reload caddy
curl https://storage.example.com/maxio/health/liveIf the last command prints {"status":"ok"}, MaxIO is reachable over HTTPS.
Using nginx or Traefik instead? Make sure the proxy passes the original Host header through and doesn't cap the request size. S3 signatures include the host, and large documents are uploaded in 5 MiB parts.
Create a key for MarketsFlare
Don't give MarketsFlare your admin key. Create a dedicated user that can read and write objects and hand over that user's key instead. You can revoke it at any time without touching anything else.
marketsflare user with read-write access, and its key. MaxIO prints the secret once.export AWS_ENDPOINT_URL=https://storage.example.com
export AWS_ACCESS_KEY_ID="$MAXIO_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$MAXIO_SECRET_KEY"
export AWS_DEFAULT_REGION=us-east-1
aws iam create-user --user-name marketsflare
aws iam attach-user-policy \
--user-name marketsflare \
--policy-arn arn:aws:iam::000000000000:policy/readwrite
aws iam create-access-key --user-name marketsflareCopy AccessKeyId and SecretAccessKey from the last command. MaxIO only shows the secret once.
The built-in readwrite policy covers every bucket on the server. If this MaxIO stores other data too, narrow the user to the marketsflare bucket with Grant scope to user in the MaxIO console.
Connect it to MarketsFlare
In MarketsFlare, open Settings → Data Infrastructure. In the Storage (S3) card, click Add Server and fill in the form.
| Field | What to enter |
|---|---|
| Name | Anything you'll recognise, like Our MaxIO |
| Endpoint URL | https://storage.example.com |
| Bucket | marketsflare |
| Region | us-east-1, the same value as MAXIO_REGION |
| Access Key | AccessKeyId from step 3 |
| Secret Key | SecretAccessKey from step 3 |
Click Test Connection. MarketsFlare signs a request with those credentials and checks that it can reach the bucket. When the test passes, click Create Server.
Make it the default
Click the star next to your new server. From then on, every file uploaded in this workspace is stored in MaxIO.
Files uploaded earlier stay where they are and keep working, because each file remembers which server it was written to. Search embeddings are stored separately, on the embedding server listed on the same settings page.
Look inside your bucket
OptionalUpload a document in MarketsFlare, then list the bucket with the admin credentials you exported in step 3:
datas/<workspace>/<file>. The metadata holds the wrapped key and IV, never the key itself.aws s3 ls s3://marketsflare/datas/ --recursive
aws s3api head-object --bucket marketsflare \
--key datas/<workspace-id>/<file-id> \
--query Metadata.encryption --output textEach document is stored at datas/<workspace-id>/<file-id>, and previews and other processing output sit next to it with an @ suffix. Before anything is sent, MarketsFlare encrypts the file with AES-256-GCM using a fresh key for that file. The key is wrapped by our key management service and stored with the object as metadata, so the bytes on your disk are ciphertext.
If something goes wrong
| Error | What it means |
|---|---|
Invalid region in credential scope | The Region in MarketsFlare doesn't match MAXIO_REGION. |
SignatureDoesNotMatch | A proxy changed the Host header on its way to MaxIO, or the secret key was pasted with extra characters. |
RequestTimeTooSkewed | The server's clock is more than 15 minutes off. Keep it in sync with chrony or systemd-timesyncd. |
AccessDenied | The user from step 3 has no policy. Run the attach-user-policy command again. |
NoSuchBucket | The bucket name is wrong, or MaxIO started without MAXIO_DEFAULT_BUCKETS. The names maxio and ui are reserved. |
| The test times out | MarketsFlare can't reach your server. Check DNS and that port 443 is open. If you only allow known IPs, ask us for our outbound addresses. |
Before you rely on it
- Back up the
maxio-datavolume. It holds your objects together with.maxio-keys.jsonandiam.json, and the access key from step 3 is sealed with that key file, so keep them together and store a copy off the server. Copy the volume while MaxIO is stopped, or use a filesystem snapshot. - Point your monitoring at
https://storage.example.com/maxio/health/ready. It returns 503 when the data directory can't be read. - Watch disk space. MaxIO doesn't enforce quotas.
- Keep the image tag pinned and read MaxIO's upgrade notes before you move to a new version.
That's it. Your workspace's documents now live on a server you run, encrypted before they leave MarketsFlare. If you get stuck, reach out to your MarketsFlare contact and we'll help you get it working.
Written by Markets Flare