Data Warehouse: Exception Handed

Even with a well-designed system, unexpected issues can arise. This article explains how the Data Warehouse (DW) handles exceptions — situations where data from Filevine (FV) doesn’t arrive as expected due to missing webhooks, system errors, or strange characters (basic letters like A, B, C; symbols like € or ©; and characters or emojis like 😊) in the data.

Missed or Unexpected Webhooks

Sometimes, Filevine doesn’t send webhooks when it should. These are not errors on our side — they’re caused by unpredictable behavior from FV, such as:

• Random or runtime failures.
• UI actions that don’t trigger events.

Because the webhook never arrives, we don’t know the change happened.

We use a special tool called Audit Webhooks, which tracks each webhook from our Data Sync (DS) system all the way to its final step (upsert). If a webhook isn’t marked as received in our DynamoDB tracking table, we know it never arrived.

• If a future event happens on the same record, we’ll update the data then.
• Otherwise, we rely on scheduled rehydration to refresh the missing data.

These issues are not logged in real-time because the event was never received in the first place.

Network or System Errors

Sometimes an internal system process fails, this could be due to:

• Runtime exceptions.
• Timeouts.
• System crashes or throttling.

When this happens, the data doesn’t make it into the table until we detect the issue and retry.

We have alerts and logging built into our pipeline to track:

• Timeouts
• API throttling
• Unhandled errors

All these failed attempts are recorded in the DW under the table: fvdw_failed.

• A subsequent webhook for the same record will update the data.
• We also have nightly automated rehydration or manual rehydration to fix it.

These issues are logged, so we can track and retry them later.

Data Anomalies (Strange Characters)

Sometimes Filevine sends data with unexpected or special characters that don't match the expected data format. These characters can break the update process or prevent fields from saving correctly.

We’ve set up alerts in our functions to identify these unusual characters through error logs. These help us locate and diagnose anomalies in the data stream.

We compare the faulty data with the expected schema, then make any necessary updates to handle these edge cases systematically.

These issues are logged, and we continue improving the system to prevent them in the future.

Summary: Exception Handling Scenarios