Using Redaction
The Nightfall API is capable of returning a redacted version of your scanned text when a Detector is triggered.
This functionality allows you to hide potentially sensitive information while retaining the original context in which that information appeared.
Specifying a RedactionConfig
In order to redact content, when you call the scan endpoint you must provide a RedactionConfig as part of the definition of your Detection Rule.
You may specify one of the following different methods to redact content:
- apply masking (e.g. asterisks)
- substitute a custom phrase
- substitute the name of the Detector triggered (referred to as "InfoType substitution")
- use encryption
A RedactionConfig is defined per Detector in a Detection Rule, allowing you to specify a different redaction method for each type of Detector in the rule.
By default, the redaction feature will return both the sensitive finding and the redacted version of that finding. You may set the removeFinding field to true if you want only the redacted version of the finding returned in the response.
Masking Characters
Specifying a MaskConfig as part of your RedactionConfig substitutes a character for each character in the matched text. By default the masking character is an asterisk (*). You may specify an alternate character to use instead (maskingChar).
You may also choose to only mask a portion of the original text by specifying a number of characters to leave unmasked (numCharsToLeaveUnmasked). For instance, if you want to mask all but the last 4 digits of a credit card number, set this value to 4 so that the redacted finding would be rendered as ***************4242.
In the case where you want to leave characters unmasked at the front of the string you may use the maskLeftToRight flag. This flag determines if masking is applied left to right (*****/1984) instead of right to left (01/01*****). By default, this value is false.
Below is an example of how a RedactionConfig would be configured to redact the text that triggers a DATE_OF_BIRTH Detector such that the text 01/11/1995 becomes ??/??/??95
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"DATE_OF_BIRTH",
"redactionConfig":{
"maskConfig":{
"charsToIgnore":[
"/"
],
"maskingChar":"?",
"maskRightToLeft":true,
"numCharsToLeaveUnMasked":2
}
}
}
Phrase Substitution
The SubstitutionConfig substitutes a sensitive finding with the value assigned to the property substitutionPhrase.
If no value is assigned to substitutionPhrase, the finding will be replaced with an empty string.
InfoType Substitution
It is possible to replace a sensitive finding with the name of the NIGHTFALL_DETECTOR that triggered it by using an InfoTypeSubstitutionConfig.
If you use the built in credit card Detector, the string 4242-4242-4242-4242 will be redacted to [CREDIT_CARD_NUMBER]
This config is only valid for Detector's with a detectorType of NIGHTFALL_DETECTOR.
Encryption
A CryptoConfig will encrypt a sensitive finding with a public key (provided as the publicKey property of the config) using RSA encryption.
Note that you are responsible for passing public keys for encryption and handling any decryption of the response payload. Nightfall will not store your keys.
Below is an example of a CryptoConfig being used to redact an EMAIL_ADDRESS detector.
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"EMAIL_ADDRESS",
"displayName":"email",
"redactionConfig":{
"cryptoConfig":{
"publicKey":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAydYMwOYUGyBXDgHkzv19YR/dYQES4kYTMUps39qv/amNDywz4nsBDvCUqUvcN3nEpplHlYGH5ShSeA4G/FcmRqynSLVyFPZat/8E7n+EeHsgihFrr8oDWo5UBjCwRinTrC0m11q/5SeNzwVCWkf9x40u94QBz13dQoa9yPwaZBX5uBzyH86R7yeZHpad2cLq0ltpmJ3j5UfsFilkOb3JB60TNpNDdfabprot/y30CEnDDOgAXGtV1m0AhQpQjKRnkUs39DntqSbS+i0UgbyqzEGNUkeR1WsotXekW4KnbWA7k6S8SfkO27vnTSY5b9g/KKaOdysn5YaWJPfTVT/nywIDAQAB\n-----END PUBLIC KEY-----"
}
}
}
Redactions in the Scan Response
The results of applying redactions are returned in the response payload for requests made to the scan endpoint as both part of an array named redactedPayload as well as additional properties of the finding object.
The original input payload with redactions made inline are returned as a list of strings under the redactedPayload property. Each item in the list of redacted payloads corresponds to the list of strings in the original input payload and, if a Detector was triggered, it will contain a redacted version of that corresponding string.
If an item in the input payload did not have any findings, the entry for that index will be an empty string ("").
The redactedPayload property is omitted if no RedactionConfig was provided.
Additionally, the fields redactedFinding and redactedLocation are added to the finding object when the redaction feature is invoked.
The redactedFinding field contains the redacted version of only the text of the finding without its surrounding context. This is useful when you are masking a portion of the text that triggered a Detector.
The redactedLocation property will be returned as part of the finding that corresponds to an item in the payload. This may be distinct from the location property that is returned for a finding by default.
In the unlikely case where there are findings that overlap, Nightfall will default to replacing the text of the overlapping findings with [REDACTED BY NIGHTFALL].
Example Redaction Call
The following example shows how the redaction functionality may be invoked, with a variety of different redaction methods applied to the different Detectors being used.
curl --location --request POST 'https://api.nightfall.ai/v3/scan' \
--header 'x-api-key: NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--header 'Content-Type: text/plain' \
--data-raw '{
"payload":[
"my ssn is 123-45-5555 and date of birth is 01/11/1995 and my credit card number is 4242 4242 4242 4242 and my email is [email protected].",
"my date of birth is 03 23 4242 4242 4242 4242 amex"
],
"policy":{
"detectionRules":[
{
"detectors":[
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"CREDIT_CARD_NUMBER",
"displayName":"cc",
"redactionConfig":{
"infoTypeSubstitutionConfig":{
},
"removeFinding":true
}
},
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"US_SOCIAL_SECURITY_NUMBER",
"displayName":"ssn",
"redactionConfig":{
"substitutionConfig":{
"substitutionPhrase":"*REDACTED*"
}
}
},
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"EMAIL_ADDRESS",
"displayName":"email",
"redactionConfig":{
"cryptoConfig":{
"publicKey":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAydYMwOYUGyBXDgHkzv19YR/dYQES4kYTMUps39qv/amNDywz4nsBDvCUqUvcN3nEpplHlYGH5ShSeA4G/FcmRqynSLVyFPZat/8E7n+EeHsgihFrr8oDWo5UBjCwRinTrC0m11q/5SeNzwVCWkf9x40u94QBz13dQoa9yPwaZBX5uBzyH86R7yeZHpad2cLq0ltpmJ3j5UfsFilkOb3JB60TNpNDdfabprot/y30CEnDDOgAXGtV1m0AhQpQjKRnkUs39DntqSbS+i0UgbyqzEGNUkeR1WsotXekW4KnbWA7k6S8SfkO27vnTSY5b9g/KKaOdysn5YaWJPfTVT/nywIDAQAB\n-----END PUBLIC KEY-----"
}
}
},
{
"minNumFindings":1,
"minConfidence":"POSSIBLE",
"detectorType":"NIGHTFALL_DETECTOR",
"nightfallDetector":"DATE_OF_BIRTH",
"redactionConfig":{
"maskConfig":{
"charsToIgnore":[
"/"
],
"maskingChar":"?",
"maskRightToLeft":true,
"numCharsToLeaveUnMasked":2
}
}
}
],
"name":"cc",
"logicalOp":"ANY"
}
]
}
}'
You can see in the response how the RedactionConfig associated with the various Detectors affects the different findings.
Note that because the 2nd item the payload matches multiple detectors, the redacted text in the redactedPayload property becomes [REDACTED BY NIGHTFALL]
{
"findings":[
[
{
"finding":"[email protected]",
"redactedFinding":"X8QL0mZGHZ+N47nPEccjsLHf2F/5cFqjF16P6wgYJhy8IaxHipHWMBRAufKR4T8FFkvTuTEanu6ZAA+V8NTkNmTLxHarcWPSVClJ8kjXAPltLuR4I2H4eeT+sWEvUP3ik/BF1KcxRpsYWDQO1bNYk+WReXkWlW72Q7rbWuTGFj2uDFCPS+DUraDh9wNBsMPELFOnh1GSQIKCp9U5GMp/kkpo/0idh83RVHXyjZPT4ReKEST2oG2lQ9UuP5LJy/mHX1VYgd8DwlETn8nkhqJ1T0mGs6kHSh22G6N0ic0PjHnj73RiMnQdPwlLw3qyPmFf6RRLKtFuzmFan8ZGtZhcKA==",
"detector":{
"name":"email",
"uuid":"c0235299-0f26-4ad6-ad8c-71f83daf44e9"
},
"confidence":"VERY_LIKELY",
"location":{
"byteRange":{
"start":120,
"end":135
},
"codepointRange":{
"start":120,
"end":135
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":120,
"end":135
},
"codepointRange":{
"start":120,
"end":135
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
},
{
"finding":"01/11/1995",
"redactedFinding":"??/??/??95",
"detector":{
"name":"DATE_OF_BIRTH",
"uuid":"540856cb-99cb-42e7-b8aa-cd4f22f019d7"
},
"confidence":"LIKELY",
"location":{
"byteRange":{
"start":43,
"end":53
},
"codepointRange":{
"start":43,
"end":53
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":43,
"end":53
},
"codepointRange":{
"start":43,
"end":53
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
},
{
"finding":"",
"redactedFinding":"[CREDIT_CARD_NUMBER]",
"detector":{
"name":"cc",
"uuid":"74c1815e-c0c3-4df5-8b1e-6cf98864a454"
},
"confidence":"VERY_LIKELY",
"location":{
"byteRange":{
"start":84,
"end":103
},
"codepointRange":{
"start":84,
"end":103
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":84,
"end":103
},
"codepointRange":{
"start":84,
"end":103
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
},
{
"finding":"123-45-5555",
"redactedFinding":"*REDACTED*",
"detector":{
"name":"ssn",
"uuid":"e30d9a87-f6c7-46b9-a8f4-16547901e069"
},
"confidence":"VERY_LIKELY",
"location":{
"byteRange":{
"start":10,
"end":21
},
"codepointRange":{
"start":10,
"end":21
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":10,
"end":21
},
"codepointRange":{
"start":10,
"end":21
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
}
],
[
{
"finding":"",
"redactedFinding":"[CREDIT_CARD_NUMBER]",
"detector":{
"name":"cc",
"uuid":"74c1815e-c0c3-4df5-8b1e-6cf98864a454"
},
"confidence":"VERY_LIKELY",
"location":{
"byteRange":{
"start":26,
"end":45
},
"codepointRange":{
"start":26,
"end":45
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":26,
"end":45
},
"codepointRange":{
"start":26,
"end":45
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
},
{
"finding":"03 23 4242",
"redactedFinding":"????????42",
"detector":{
"name":"DATE_OF_BIRTH",
"uuid":"540856cb-99cb-42e7-b8aa-cd4f22f019d7"
},
"confidence":"LIKELY",
"location":{
"byteRange":{
"start":20,
"end":30
},
"codepointRange":{
"start":20,
"end":30
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"redactedLocation":{
"byteRange":{
"start":20,
"end":30
},
"codepointRange":{
"start":20,
"end":30
},
"rowRange":null,
"columnRange":null,
"commitHash":""
},
"matchedDetectionRuleUUIDs":[
],
"matchedDetectionRules":[
"cc"
]
}
]
],
"redactedPayload":[
"my ssn is *REDACTED* and date of birth is ??/??/??95 and my credit card number is [CREDIT_CARD_NUMBER] and my email is X8QL0mZGHZ+N47nPEccjsLHf2F/5cFqjF16P6wgYJhy8IaxHipHWMBRAufKR4T8FFkvTuTEanu6ZAA+V8NTkNmTLxHarcWPSVClJ8kjXAPltLuR4I2H4eeT+sWEvUP3ik/BF1KcxRpsYWDQO1bNYk+WReXkWlW72Q7rbWuTGFj2uDFCPS+DUraDh9wNBsMPELFOnh1GSQIKCp9U5GMp/kkpo/0idh83RVHXyjZPT4ReKEST2oG2lQ9UuP5LJy/mHX1VYgd8DwlETn8nkhqJ1T0mGs6kHSh22G6N0ic0PjHnj73RiMnQdPwlLw3qyPmFf6RRLKtFuzmFan8ZGtZhcKA==.",
"my date of birth is [REDACTED BY NIGHTFALL] amex"
]
}
Updated about 1 month ago