Duplicate materials: how to identify them when descriptions differ

To identify duplicate materials when the descriptions differ, stop comparing text and start comparing characteristics. Group items by family, pull out the attributes that define each item (manufacturer and part number, dimensions, material, standard, unit of measure) and compare only within the same family. Software finds the candidates; deciding that two codes are the same item belongs to someone who knows the material, with the evidence recorded.

I have seen this scene in a lot of storerooms.

The same bearing registered under three codes. One written the way purchasing writes it, another the way maintenance writes it, the third with the manufacturer's part number buried in the middle of the description. One of them has been sitting on the shelf for months. Another is on an urgent purchase order, because a machine stopped and the system says there is none in stock.

Nobody did it on purpose. Each person created what they needed, the way they knew how, and the system accepted it. The result is idle stock on one side and an emergency purchase on the other, for the same item.

What is material deduplication?

The 4MDG glossary defines deduplication as the process of recognizing that two or more records describe the same entity. Once that is recognized, there are three possible outcomes: merge them, point one record to the other (a cross-reference) or mark them as distinct.

Official statistics deals with a similar problem under the name record linkage. In a U.S. Census Bureau paper, William E. Winkler defines record linkage as "the means of combining information from a variety of computerized files." He notes that the topic also goes by data cleaning and object identification, and that the basic methods compare pairs of records to find those that refer to the same entity. That is useful context: deduplicating a materials master is a version of the same challenge, with the specific needs of the people who buy, stock and maintain those items.

Why do free-text descriptions hide duplicates?

Because two people describe the same item in different ways, and a text search only finds what was written the same way. The most common variations:

• Abbreviations: "BRG", "BEAR.", "BEARING". Every department has its own shortcut.
• Word order: "BALL BEARING DEEP GROOVE" and "DEEP GROOVE BALL BEARING" ask for the same thing, but they are not the same string.
• Units and formats: millimeters in one record, inches in the other; a decimal comma in one, a decimal point in the other.
• Typos: one wrong letter is enough to make the search miss.
• Language: one code described in the local language, another copied from the manufacturer's catalog in English.
• Manufacturer part number inside the text: the strongest piece of data in the record is mixed into the description, with no field of its own, so nobody can compare it.

That is why I say deduplicating by comparing descriptions is like looking for a needle with your eyes closed. The way out is to turn the description into characteristics you can actually compare.

Which signals show that two materials are the same item?

Not every signal carries the same weight. Some nearly confirm the duplicate. Others only tell you the pair is worth a look. The table below sums up how I separate the two groups.

Signal	Strength	Watch out for
Same manufacturer and same manufacturer part number	Strong	Check that the part number was not entered with a different suffix that changes the item version.
Same key attributes for the family (for example, dimensions, material, standard)	Strong	Only valid if the family attributes are defined and filled in on both records.
Same unit of measure and same supply form	Medium	A different unit does not prove the items differ; it may just be a data entry error. If there is a conversion factor (for example, box versus piece), check the packaging before merging.
Same NCM code	Weak	NCM is a goods category. Very different items can share the same code.
Similar description	Weak	Good for generating candidates, never enough to decide on its own.
Same usual supplier	Weak	One supplier sells many similar items; this only helps prioritize the review.

A note on NCM for readers outside Brazil: it is the Mercosur Common Nomenclature, the goods classification used in Brazil. Brazil's Federal Revenue Service (Receita Federal) describes it as a regional nomenclature for categorizing goods. A category, not an identity. Two bearings of different sizes can share the same NCM and still be different items. If you work with another classification system, the same logic applies.

How do you identify duplicate materials, step by step?

1. Group by family

Comparing everything against everything creates noise. Bearings get compared with bearings, valves with valves. If your master data does not have reliable families yet, that is the first thing to fix.

2. Normalize the text

Standardize case, accents, abbreviations and units. Replace "BRG" and "BEAR." with a single term. Convert measurements to a common unit. Normalize the part number too: remove spaces, hyphens, periods and leading zeros, and put the manufacturer in its own field. It is manual, tedious work, but without it the next comparison does not work.

3. Extract the key attributes

Pull out of the description what defines the item: manufacturer, part number, dimensions, material, standard. Each family has its own key attributes, and the people who use the item are the ones who define them. This is where a materials PDM (a product data model, the attribute template for each family) makes a difference, because it already says which characteristics each family must have.

4. Compare within the family

With attributes in their own fields, the comparison becomes objective. Same manufacturer with same part number goes one way. Matching key attributes go another. Similar descriptions with no matching attributes come last.

5. Sort into certain, probable and review

I like three tiers. "Certain" when the strong signals match. "Probable" when most key attributes match and one piece of data is missing. "Review" when only weak signals match. Where to draw each line is a decision for whoever leads the work, family by family, and it should be revisited as results come in. There is no magic number that fits every database.

6. Human review with evidence

Every pair goes to someone who knows the material: maintenance, engineering, procurement. That person decides and records who decided, on what basis and when. The 4MDG glossary stresses this point: similarity is not identity, and the decision needs an audit trail.

This is not excessive caution. The U.S. Census Bureau's Statistical Quality Standard C4, "Linking Data Records," calls for "Evaluating the accuracy of automated linkages by a manual review" and states that "Documentation needed to replicate and evaluate the linking operations must be produced." It is a standard for U.S. government statistical data, not a rule for materials master data. But it makes the point: even in official statistics, automated linkage is checked by people and documented.

Which traps produce false positives?

A false positive is flagging as a duplicate something that is actually different. In a materials master, that mistake is expensive, because merging two distinct items means someone receives the wrong part. The most common traps:

• Same name, different dimension: two bearings with the same generic description and different diameters. The description matches; the item does not.
• Same part, different standard: one item meets one standard, the other meets another. To a buyer they look the same; to someone maintaining regulated equipment, they are not.
• Kit versus component: the repair kit contains the part, but it is not the part. They are different codes by nature.
• New versus repaired or refurbished item: same specification, different value and use. Many companies track them separately on purpose.
• Batch or shelf-life controlled material: two codes may exist because traceability requires it, not because of an error.

What can't be automated in deduplication?

Finding candidates can be automated. The decision cannot. These stay with a person:

• Technical equivalence: stating that one manufacturer's part serves the same purpose as another's.
• Interchangeability: knowing whether one item replaces the other in every piece of equipment or only in some.
• Maintenance-critical items: when a wrong swap stops production or creates risk, the decision belongs to whoever is accountable for the equipment.
• Impact on inventory and contracts: merging touches stock balances, open purchase orders and contracts priced by code. Someone has to check the effect first.

What should you do once you find a duplicate?

The temptation is to delete it. Don't. The duplicate code carries purchasing, consumption and movement history, and that history tells the item's story. Deleting it breaks reports, audits and traceability. What I recommend:

• Choose the survivor: the code that stays, usually the most complete and most used one. Its attributes can be completed with whatever is correct in the other record.
• Block the duplicate for purchasing, planning and reservations: nobody orders under the old code anymore, material requirements planning (MRP) stops generating demand for it and it accepts no new reservations or requisitions.
• Transfer the balance: stock from the duplicate moves to the survivor, with the movement recorded.
• Handle open purchase orders: let them finish under the old code or adjust them with the supplier, according to your company's process. What matters is that no new order is opened on the blocked code.
• Create the cross-reference: the old code points to the survivor, so anyone searching by the old number finds the new one.
• Keep the history: the old record stays, blocked and linked to the survivor.

How do you stop duplicates from coming back?

Cleaning the database and leaving the door open is mopping the floor with the tap running. The 4MDG glossary points out that fixing at entry costs less than fixing later.

The simplest rule is to search by characteristics before creating. Brazil's federal government asks for this in CATMAT, its federal catalog of materials used in public procurement. The cataloging request tutorial on the federal Procurement Portal ("Portal de Compras do Governo Federal") states that it is important to run a prior search to check whether the desired item is already registered in the catalog, and the system filters items by the characteristics selected. If a federal catalog asks for this before a new item is created, a company can ask for it too.

In the item creation request, this becomes a mandatory step: the requester picks the family, fills in the key attributes and sees the items that already exist with those characteristics before asking for a new one.

What does this look like in a hypothetical example?

Hypothetical example, created to illustrate the method. It does not describe a real company.

A company has three codes in its bearings family: code A, described as "DEEP GROOVE BALL BEARING" with the manufacturer's part number written at the end; code B, with a description copied from the catalog, same manufacturer and same part number; and code C, "BALL BRG", with no manufacturer and the dimensions in the description.

After normalizing and extracting attributes, A and B share manufacturer and part number: they go to "certain." C has the same dimensions but no manufacturer: it goes to "probable." Maintenance confirms that A and B are the same item and picks A as the survivor. For C, the technician checks the specification and finds that the seal is different: they are distinct items. C stays, gets the missing attributes and its own description. Without human review, C would have been merged by mistake.

Where do you start deduplicating materials?

1. Pick a family with many items and heavy purchasing, where duplicates hurt the most.
2. Define the family's key attributes with the people who use the items. If your company does not have that yet, start with a PDM.
3. Normalize the text and extract the attributes for that family.
4. Generate candidate pairs and sort them into certain, probable and review.
5. Take the pairs to people who know the material and record every decision.
6. Handle confirmed duplicates without deleting: survivor, block, balance transfer, cross-reference.
7. Put the characteristics search at the point of entry and track the duplicate rate, as I explain in the article on how to measure master data quality.

Frequently asked questions

Can materials be deduplicated with AI alone?

AI helps find identical items described in different ways and suggests pairs. Deciding that two codes are the same item is still a job for someone who knows the material, because it involves technical equivalence, criticality and impact on inventory. Use AI to shorten the list, not to make the final call.

Can I delete the duplicate code in the ERP?

It is not advisable. The duplicate code carries purchasing, consumption and movement history. The safer path is to block the code for purchasing, transfer the balance to the survivor, create the cross-reference and keep the old record linked to the new one.

What is the difference between a duplicate and a similar or equivalent item?

A duplicate is the same item registered under more than one code. A similar item looks alike but has a relevant difference, such as dimension, standard or seal. An equivalent item is a different item that serves the same purpose; recognizing equivalence is a technical decision and usually becomes a cross-reference, not a merge.

Do I need a PDM before deduplicating?

It does not have to be finished for the whole database, but the family you are deduplicating needs defined key attributes. Without them, the comparison falls back to text. 4MDG's data cleansing page states that deduplicating without standardizing only treats the symptom.

How often should I run the duplicate search?

It depends on how many new items come in and how well entry is already controlled. If the characteristics search at request time works, the periodic sweep becomes a check. If entry is open, duplicates come back and the sweep needs to run more often. The duplicate rate indicator tells you which case you are in.

What should I do with the stock balance and open orders of the duplicate item?

The balance moves to the survivor code, with the movement recorded. Open orders can finish under the old code or be adjusted with the supplier, according to your company's process. What must not happen is a new order opened on the blocked code.

How 4MDG can help

At 4MDG, deduplication is one stage of master data cleansing, done after standardization, with a similarity score and attribute-by-attribute survivorship (the rule that sets which value stays in each attribute of the surviving record). In the 4MDG materials PDM, suggested matches go through expert approval, and the final decision is always human. If you want to know how much duplication exists in your materials master, request the assessment, run on a real sample of your data with nothing installed in your environment. If you would rather talk first, get in touch.

Sources

• 4MDG. Glossary: Deduplication. https://4mdg.com.br/en/glossario-deduplicacao. Accessed on October 8, 2026.
• U.S. Census Bureau. William E. Winkler, "Overview of Record Linkage and Current Research Directions," RRS2006-02, 2006. https://www.census.gov/library/working-papers/2006/adrm/rrs2006-02.html. Accessed on October 8, 2026.
• U.S. Census Bureau. "Statistical Quality Standard C4: Linking Data Records," revised October 8, 2021. https://www.census.gov/about/policies/quality/standards/standardc4.html. Accessed on October 8, 2026.
• Brazil's Ministry of Management and Innovation in Public Services, Federal Procurement Portal. Cataloging request tutorial for the materials and services catalog, "Catálogo de Materiais e Serviços: Pedidos de Catalogação" (original title, in Portuguese). https://www.gov.br/compras/pt-br/acesso-a-informacao/manuais/copy_of_manuais/manual-fase-interna/manual-catmat-catser/temporario-tutorial_pedidos-de-catalogacao_web_final.pdf. Accessed on October 8, 2026.
• Brazil's Federal Revenue Service (Receita Federal). NCM page (in Portuguese). https://www.gov.br/receitafederal/pt-br/assuntos/aduana-e-comercio-exterior/classificacao-fiscal-de-mercadorias/ncm. Accessed on October 8, 2026.