Uploading and Scanning API Calls
Nightfall's upload process is built to accommodate files of any size.
The upload process consists of 3 stages:
- Initializing
- Uploading
- Completing
Once the upload is complete, you may initiate the file scan.
The expected content type for all endpoints is application/json
.
Initializing Phase
POST /v3/upload
The first step in the process of scanning a binary file is to initiate an upload in order to get a fileId
through the Initiate a File Upload endpoint.
As part of the initialization you must provide the total byte size of the file being uploaded.
You may also provide the mime-type, otherwise the system will attempt to determine it once the upload is complete.
curl --location --request POST 'https://api.nightfall.ai/v3/upload' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '{
"fileSizeBytes": 73891,
"mimeType" : "image/png"
}'
The id
of the returned JSON object will be used as the fileId
in subsequent requests.
The chunkSize
is the maximum number of bytes to upload during the uploading phase.
{
"id": "f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d",
"fileSizeBytes": 73891,
"chunkSize": 10485760,
"mimeType": "image/png"
}
Uploading Phase
PATCH /v3/upload/<uploadUUID>
Use the Upload a Chunk of a File endpoint to upload the file contents in chunks. The size of these chunks are determined by the chunkSize
value returned by POST /upload
endpoint used in the previous step.
curl --location --request PATCH 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d' \
--header 'X-Upload-Offset: 0' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-binary '@/Users/myname/Documents/work/Nightfall/Nightfall Upload Sequence.png'
Each request should send a chunk of the file exactly chunkSize
bytes long.
The final uploaded chunk is allowed to contain fewer bytes, as its length may be shorter than the chunk size returned by the initialization step.
The request body should be the contents of the chunk being uploaded.
Additionally, this endpoint requires the header X-UPLOAD-OFFSET
to be passed. The value of the header should be the integer byte offset specifying where to insert the data into the file. This byte offset is zero-indexed.
Successful calls to this endpoint return an empty response with an HTTP status code of 204
Completion Phase
POST /v3/upload/<uploadUUID>/finish
Once all chunks are uploaded, mark the upload as completed using the Complete a File Upload endpoint.
curl --location --request POST 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/finish' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '""'
When an upload completes successfully, the returned payload will indicate the mimeType the system determined to file to be if it was not provided during upload initialization.
{
"id": "152848af-2ac9-4e0a-8563-2b82343d964a",
"fileSizeBytes": 2349,
"chunkSize": 10485760,
"mimeType": "application/zip"
}
Scanning Uploaded Files
After an upload is finalized, it can be scanned against a Detection Policy. A Detection Policy represents a pairing of:
- a webhook URL
- a set of detection rules to scan data against
The scanning process is asynchronous, with results being delivered to the webhook URL configured on the detection policy.
Exactly one policy
should be provided in the request body, which includes a webhookURL
to which the callback will be made once the file scan has been completed (this must be an HTTPS URL) as well as a Detection Rule as either an a list of UUIDs or as a rule that has been defined in-line.
You may also supply a value to the requestMetadata
field to help identify the input file upon receiving a response to your webhook. This field has a maximum length 10 KB.
curl --request POST \
--url https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/scan \
--header 'Accept: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--header 'Content-Type: application/json' \
--data '
{
"policy": {
"detectionRuleUUIDs": [
"950833c9-8608-4c66-8a3a-0734eac11157"
],
"webhookURL": "https://mycompany.org/webhookservice"
},
"requestMetadata": "your file metadata"
}
'
Webhook Verification
Nightfall will verify that the webhook URL is valid before launching its asynchronous scan by issuing a challenge.
Full Upload Process Example Script
Below is a simple sample Python script that handles the complete upload sequence using a specified file.
from os import getenv, path
import fire
import requests
BASE_UPLOAD_URL = getenv("FILE_UPLOAD_HOST", "http://localhost:8080/v3")
NF_API_KEY = getenv("NF_API_KEY")
def upload(filepath, mimetype, policy_uuid):
"""Upload the given file using the provided MIMEType and PolicyUUID.
Arguments:
file_path -- an absolute or relative path to the file that will be
uploaded to the API.
mimetype -- (optional) The mimetype of the file being uploaded.
policy_uuid -- The UUID corresponding to an existing policy. This
policy must be active and have a webhook URL associated with it.
"""
default_headers = {
"Authorization": F"Bearer {NF_API_KEY}",
}
# =*=*=*=*=* Initiate Upload =*=*=*=*=*=*
file_size = path.getsize(filepath)
upload_request_body = {"fileSizeBytes": file_size, "mimeType": mimetype}
r = requests.post(F"{BASE_UPLOAD_URL}/upload",
headers=default_headers,
json=upload_request_body)
upload = r.json()
if not r.ok:
raise Exception(F"Unexpected error initializing upload - {upload}")
# =*=*=*=*=*=* Upload Chunks =*=*=*=*=*=*
chunk_size = upload["chunkSize"]
i = 0
with open(filepath, "rb") as file:
while file.tell() < file_size:
upload_chunk_headers = {
**default_headers,
"X-UPLOAD-OFFSET": str(file.tell())
}
r = requests.patch(F"{BASE_UPLOAD_URL}/upload/{upload['id']}",
headers=upload_chunk_headers,
data=file.read(chunk_size))
if not r.ok:
raise Exception(F"Unexpected error uploading chunk - {r.text}")
i += 1
# =*=*=*=*=*=* Finish Upload =*=*=*=*=*=*
r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/finish",
headers=default_headers)
if not r.ok:
raise Exception(F"Unexpected error finalizing upload - {r.text}")
# =*=*=*=*=* Scan Uploaded File =*=*=*=*=*
r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/scan",
json={"policyUUID": policy_uuid},
headers=default_headers)
if not r.ok:
raise Exception(F"Unexpected error initiating scan - {r.text}")
print("Scan Initiated Successfully - await response on configured webhook")
quota_remaining = r.headers.get('X-Quota-Remaining')
if quota_remaining is not None and int(quota_remaining) <= 0:
print(F"Scan quota exhausted - Quota will reset on {r.headers['X-Quota-Period-End']}")
if __name__ == "__main__":
fire.Fire(upload)
Updated 7 months ago