Duplicate leads in your CRM from an integration: why it keeps happening
An integration creates duplicate leads when it cannot find the existing record, so it makes a new one. The usual reasons are an exact-email match that misses a typo or a second address, a sync login that cannot see the original, two connections feeding one CRM, or an import that skipped the duplicate warning. Trace the duplicate before you merge it.
What should I check first?
Merging duplicates fixes today's pile and nothing else. If the cause is still running, the pile grows back. Before you merge, answer three questions.
- Is it a new problem or an old one? Sort the duplicates by created date. A sudden start on one date points at a change (a new form, a new connection, an import). A steady trickle points at matching rules.
- Is it one source or all of them? Group the duplicates by lead source. One source, such as a webinar tool, points at that connection.
- Are they really the same person? Two records with the same name and different companies may be two people. Shared addresses such as a team inbox can also belong to several real people.
How do I trace where a duplicate came from?
Pick five duplicate pairs from the last two weeks and run each step on all five. The steps are in order, fastest first.
- Look at who or what created each record (5 minutes; you or your CRM admin). Open both records and read the created-by field and the creation date. If the second record was created by the integration's login and the first by a person or a form, the integration is the likely source. If a person created it, this is not an integration problem. In one public forum thread, a reply suggests checking the duplicate's activity log to see whether it came from a form, a list upload or a manual entry.
- Compare the two email addresses character by character (5 minutes; you). A typo, a different case, a second address or a plus-sign tag can defeat an exact match. Replies in the same thread note that email is not a unique key.
- Find the start date (15 minutes; you). Export leads created in the last 30 days and sort by created date. Count duplicates per day. A cliff on one day points at what changed that day.
- Ask whether the sync login can see the original (20 minutes; your CRM admin). A reply in that thread says a sync user who cannot see all CRM records can cause new records to be created. In many setups this happens when a record is reassigned to an owner or a group the integration's user has no access to. This is our reading of how visibility rules work, so check yours.
- List every connection that writes leads (30 minutes; your CRM admin or marketing ops). Look for two connections, or a connection plus a form tool, that both create leads for the same sign-up. Two writers with no shared matching rule will each create a record.
- Check imports, conversions and merges (30 minutes; CRM admin). A report on 264 billion CRM records noted a clear link between frequent record loading and the use of de-duplication features, and its authors say frequent loading brings higher levels of duplicates. It gives no duplicate percentage. Our reading is that automated writes and list imports often skip the duplicate warning a person typing in the CRM would see; confirm that in your own settings.
Stop at the first step that explains all five pairs. If different pairs have different causes, you have more than one problem, which is common.
What are the usual causes?
Matching by exact email. Matching only on the email field misses typos, second addresses and the same person using a work and a personal address. In many setups it also misses a person who is already a contact when the new record arrives as a lead.
A converted lead creates a second record in another tool. In one public forum thread, converting a lead straight out of a queue produced a second person record in the marketing tool; the suggested workaround was to change the owner first, then convert. A second commenter said that is not the expected result of a standard conversion, so treat it as one reported case, not a rule.
Two writers, one destination. Typical pairs are a webinar tool plus the marketing tool, or an old connection nobody switched off after a migration. This is our reading; step 5 tests it.
A merge that the other tool did not follow. After you merge duplicates in the CRM, the sales tool may still point at the record that no longer exists. One sales tool's help article says its "could not sync this record" error usually happens when records are merged in the CRM and not updated accordingly in the sales tool. Check enrolments after any big merge.
How do I measure how bad it is without false alarms?
Export leads created in the last 30 days. Sort by lowercase email, then by last name plus company. Count pairs. Repeat for first name plus phone. As a rule of thumb of our own, not a benchmark, more than 2 to 3 percent of new leads having a twin suggests matching is leaking.
Keep the count honest:
- Compare first-time addresses. Count only people who did not exist before the period. A returning contact who fills in a form again should update their record, and that is not a duplicate.
- Exclude internal tests. Remove your team's and your agency's addresses from the export. A test lead may be routed to a rep, enrolled in a sequence and counted in reports, so tell people before you submit one.
- Do not count leads and contacts as duplicates by default. A person who is both a lead and a contact is a conversion question, not always a duplicate.
- Check the record's age. A twin created years apart may be a real change of job, not a sync fault.
How do I stop it coming back?
- Name the source of truth. Decide which tool owns the record and make the others update it, not create it.
- Give the integration's login the access it needs. It must see every record it might match.
- Match on more than one key. Use email plus a second field such as an ID the tool stores, where your tools allow it.
- Keep the duplicate warning on for imports and tell the importer to read it.
- Re-run the 30-day count monthly. It takes about an hour by hand.
Questions people ask
Can I just merge the duplicates?
Yes, but find the cause first, and check the sales tool afterwards, because a merge can leave enrolments pointing at a record that no longer exists. Merge a small batch first and look at the result.
Why does my integration create a duplicate when the email looks identical?
Look at the characters: a trailing space, a different case or a hidden character can defeat an exact match. Also check whether the integration's login can see the original record.
Is a lead and a contact with the same email a duplicate?
Not always. It often means the person was converted and then arrived again from a form. Compare the owners: if they differ, two reps may be working the same person.
Watching this all the time
Navceed is being built to watch hand-offs like this continuously: follow each record through every step, notice when one breaks, show the cause in plain words and prepare the fix for the data owner to approve. Today it works on an example company; connecting your own tools is coming next. Take the workflow break check on your own setup.