System Integration

Why integration is the hard part

Every "AI" or "automation" project we have ever shipped failed at integration before it failed anywhere else. The model is fine. The user interface is fine. The flow is logical. Then:

• The CRM rate-limits you to 100 calls per minute and you have 500 leads to enrich.
• The ERP returns a schema you've never seen before because someone added a custom field three weeks ago.
• The SaaS tool's webhook fires at-least-once and you've just sent the same invoice twice.
• The OAuth token expires at the worst possible time.
• The system you depend on is down for an hour and your queue silently grows.

These problems are mundane and they are where careers go to die. They are also where we spend most of our engineering time, because they are the difference between a demo that works on Wednesday and a system that works in October.

Patterns that survive production

The patterns are not novel. They are just consistently applied.

Idempotency

Every action gets an idempotency key. Sending the same invoice twice with the same key is a no-op the second time. Re-running a failed workflow from a checkpoint produces the same outcome, never duplicates. This sounds obvious until you trace why your CFO got two of the same payment alerts last quarter.

Retry with backoff and jitter

Transient failures self-heal. Permanent failures land in a dead-letter queue with full context — original payload, error message, stack trace, retry count — so a human can replay or annotate. We default to exponential backoff with jitter to avoid stampedes.

Schema validation at the boundary

Zod (TypeScript) or Pydantic (Python) at every integration point. Inputs are validated before they enter our system; outputs are validated before they leave. When a schema changes upstream, the integration fails loudly at the boundary, not silently in production.

Observability you can query

Per-record traces. Cost attribution per integration. Latency distributions per endpoint. We use OpenTelemetry where possible, Langfuse for AI-touched flows, and structured logs everywhere else. The dashboard answers two questions in one glance: is anything broken right now, and is anything trending toward broken.

Replay from arbitrary point

Every workflow is restartable from a checkpoint. Re-process the last 24 hours of invoices? One command. Re-run integrations for tenant X only? One command. This is the difference between a system you can operate and a system you have to babysit.

Where Microsoft Power Platform fits

Power Platform is a strong default for Microsoft 365 organisations. It is also commonly misused. Here is our honest position.

Use Power Platform when:

• You are already a Microsoft 365 tenant. The license is paid for.
• Your IT will only sign off on Microsoft. (More common than vendors will admit.)
• The integration is mostly with Microsoft systems — SharePoint, Teams, Outlook, Dataverse, Power Apps.
• The team that will maintain the integration is more comfortable with low-code than with Node.js.
• Throughput is moderate (low thousands of operations per day, not millions).

Use custom code instead when:

• You need millisecond latency or extreme throughput.
• The integration involves non-Microsoft systems with deep edge cases.
• You need full version control, code review, CI/CD, and per-PR testing.
• Cost at scale matters more than build speed.
• You want a long-term zero-license-cost stack.

Most of our engagements end up hybrid: Power Automate for the Microsoft-native parts, custom TypeScript on Cloud Functions or Cloud Run for the parts where Power Automate hits its ceiling. We have built custom connectors for both directions of the bridge.

For the deep cut on Power Platform's limits, see our PowerApps to production article.

The integration stack we tend to reach for

Real-world integrations we have shipped

A short, anonymised list of patterns we have shipped that you might recognise.

• Microsoft 365 → custom dispatch app — Outlook calendar events sync into a custom React Native field-ops app; status changes flow back into Teams channels.
• NetSuite ↔ external invoicing portal — bidirectional sync with idempotency, schema validation, and a daily reconciliation report.
• HubSpot lead enrichment — webhook receives new lead → enrichment via Clearbit + LinkedIn + custom rules → score → routing → CRM update → Slack alert.
• Microsoft Dataverse ↔ Postgres — analytical Postgres mirror of operational Dataverse for reporting, with sub-minute lag and conflict resolution.
• Custom API → Zendesk — convert custom support events into Zendesk tickets with attachments, tags, and proper routing.
• Multi-tenant SaaS → multi-tenant CRM — fan-out integration where each tenant maps to a different CRM instance with different schemas.

The MeatLogix case study shows a multi-topology dispatch integration in detail.

Our process

1. Audit — we map current state (systems, data flows, gaps) and target state. Two weeks for complex environments.
2. Design — architecture diagram, sequence diagrams for the trickier integrations, technology selection with rationale, cost & timeline estimate.
3. Build — slice-by-slice, with each slice deployable. We never do big-bang integration cutovers. Manual workflow keeps running in parallel until the integration is proven.
4. Cutover — phased rollout with rollback plan and observability ready.
5. Operate — on retainer or with handover to your team. Always with runbooks.

Pricing — the honest version

What you should expect from us

We will:

• Insist on the audit being a paid engagement so the build estimate is grounded in reality.
• Tell you when an integration is the wrong answer (sometimes the answer is to switch tools).
• Use existing connectors where they exist, build custom where they don't, document the decision.
• Build idempotency and observability into every integration from day one.
• Ship in slices.
• Hand over code in your git, runbooks for your ops team, dashboards your CFO can read.

We will not:

• Promise a date before the audit is done.
• Build integration around a "we'll figure out the data flows during implementation" plan.
• Hide costs in custom-connector licensing or hosting that we secretly own.

If you have a stack-untangling problem in mind, send a note. One paragraph is enough.