Using cURL to perform a multipart upload
In this tutorial, you learn how to upload large files in multiple parts by using cURL commands on a Linux operating system. Steps include running split in a terminal window for separating the file into parts, initiating the upload, uploading the parts, and completing the upload. In addition, this tutorial uses a script to upload each file part and get its entity tag (ETag) and to generate the XML block that is needed to complete the multipart upload operation.
Scenario
Most tools, such as the CLIs or the IBM Cloud® console, as well as most compatible libraries and SDKs, automatically transfer objects in multipart uploads. Refer to Storing large objects for details about multipart upload operations. For some scenarios, you might prefer to use cURL commands for a multipart upload, for example, if you are working within a read-only operating system in which you cannot install the CLI.
Before you begin
Before you begin the upload, you need:
- An IBM Cloud account
- An instance of IBM Cloud Object Storage
- A bucket
- Writable free space on your local system that is greater than the total file size of the upload
Using curl
assumes a certain amount of familiarity with the command line and IBM Cloud®, and have the necessary information from a service credential,
the endpoints reference, or the console.
Split the file into parts
For multipart uploads, every file part (except for the last part) must be at least 5 MB. To split a file into separate parts, you can run split in a terminal window. When you decide how to split the file given its total size, keep in mind that there is a maximum of 10,000 (10k) parts for a multipart upload.
In the following example, the split command uses the -b option with value of 100M, a file that is named TESTFILE.iso, and a prefix of "part-".
split -b 100M TESTFILE.iso part-
This command generates file parts of the size that is specified, with the names part-aa, part-ab, part-ac, and so on.
Initiate upload and get UploadID
To initiate a new multipart upload, run cURL to issue a POST
request to an object with the query parameter uploads
, which creates a new UploadId
value. You reference the UploadID
with each
part of the object that is being uploaded.
Example request
curl -X "POST" "https://s3.private.us-south.cloud-object-storage.appdomain.cloud/mputest/TESTFILE.iso?uploads"
-H "Authorization: bearer $bearertoken"
Example response
_<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>mputest</Bucket>
<Key>TESTFILE.iso</Key>
<UploadId>01000194-8fab-0f08-3c54-d7df03b4a127</UploadId>
</InitiateMultipartUploadResult>_
From the response, make note of the UploadID
. In this example, the UploadID
value is 01000194-8fab-0f08-3c54-d7df03b4a127
.
Use a script to upload parts and build XML block
Next, you run cURL to issue a PUT
request for each file part you want to upload. The response header for each part upload includes an ETag value, which you must save as input for the final step in the multipart upload operation.
Instead of uploading each part manually, you can use a bash script like the one that follows. The script has two functions:
- upload the parts
- create the XML part manifest, which is stored locally in the /tmp directory
As each file part is uploaded, the script extracts its ETag from the response header and saves it and the part number into an XML block as input for the final step.
Example script
#!/usr/bin/env bash
set -euo pipefail
# ─── Configuration ───────────────────────────────────────────────────────────────
# Public endpoint for your bucket
ENDPOINT="$COS endpoint"
BUCKET="$bucket"
KEY="$Filename"
UPLOAD_ID="$uploadID gained from initiation of upload"
ACCESS_TOKEN="$BEARERTOKEN"
PARTS_PREFIX="part-"
MANIFEST="parts_manifest.xml"
# ─────────────────────────────────────────────────────────────────────────────────
# 1) Start a fresh manifest
cat > "$MANIFEST" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUpload>
EOF
part=1
for f in ${PARTS_PREFIX}*; do
if [[ ! -r "$f" ]]; then
echo "ERROR: no files matching ${PARTS_PREFIX}*" >&2
exit 1
fi
echo "Uploading $f as part $part…"
# 2) PUT the part, capture headers
hdrfile=$(mktemp)
http_code=$(
curl -sS \
-D "$hdrfile" \
-o /dev/null \
-X PUT "${ENDPOINT}/${BUCKET}/${KEY}?partNumber=${part}&uploadId=${UPLOAD_ID}" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
--data-binary @"${f}" \
-w '%{http_code}'
)
if [[ "$http_code" != "200" ]]; then
echo "ERROR: HTTP $http_code for part $part" >&2
sed 's/^/ /' "$hdrfile" >&2
rm -f "$hdrfile"
exit 1
fi
# 3) Extract clean ETag (strip quotes/carriage returns)
etag=$(grep -i '^ETag:' "$hdrfile" \
| cut -d' ' -f2 \
| tr -d '"\r')
rm -f "$hdrfile"
if [[ -z "$etag" ]]; then
echo "ERROR: no ETag for part $part" >&2
exit 1
fi
echo " → Part $part ETag: $etag"
# 4) Append a well-formed <Part> block
printf ' <Part>\n <PartNumber>%d</PartNumber>\n <ETag>"%s"</ETag>\n </Part>\n' \
"$part" "$etag" >> "$MANIFEST"
part=$((part + 1))
done
# 5) Close the manifest
echo '</CompleteMultipartUpload>' >> "$MANIFEST"
echo "✔ Manifest generated: $MANIFEST"
## Complete multipart upload {: #curl-multipart-complete step}
When all parts are finished uploading, you complete the upload by sending a request with the `UploadId` and an XML block that lists each part number and its respective Etag value.
Example request
curl -X POST "https://s3.private.us-south.cloud-object-storage.appdomain.cloud/mputest/TESTFILE.iso?uploadId=01000194-8fab-0f08-3c54-d7df03b4a127"
-H "Authorization: Bearer $bearertoken" -H "Content-Type: application/xml" --data-binary @parts_manifest.xml
Example response
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Location>http://s3.private.us-south.cloud-object-storage.appdomain.cloud/mputest/TESTFILE.iso</Location>
<Bucket>mputest</Bucket>
<Key>TESTFILE.iso</Key>
<ETag>"1a90419141160aa6713b8d196773345e-94"</ETag>
</CompleteMultipartUploadResult>
Next steps
If all goes well, you receive a confirmation that the file was uploaded successfully to the wanted bucket. If you encounter errors, be sure to clean up incomplete multipart uploads. If an incomplete multipart upload is not stopped, the partial upload continues to use resources. See Using cURL for the syntax to get or stop incomplete multipart uploads.