← Back to Field Notes

March 26, 2026 · The Bloomfield Team

The Manufacturer's Guide to Evaluating AI Vendors

A search for "AI for manufacturing" on Google returns over 400 million results. LinkedIn shows hundreds of companies advertising AI solutions for the shop floor. Trade publications run AI feature stories weekly. The volume of vendors, products, and promises has reached the point where evaluating them is itself a meaningful time investment.

Most of these vendors will not be useful for your operation. Many are selling generic tools repackaged with manufacturing language on the landing page. Some are selling technology that requires data you do not have and infrastructure you have not built. A few are selling exactly what they claim: tools built specifically for manufacturing workflows, designed to work with the systems and data that exist in a real shop.

Telling them apart requires asking the right questions. Here is a framework for evaluating AI vendors when you are a manufacturer with 25 to 200 employees, running an ERP, and trying to figure out where AI can produce a measurable return.

Question One: Have You Worked With My ERP?

This is the filter that eliminates 70% of vendors in the first conversation.

Manufacturing AI tools need data. That data lives primarily in your ERP: job records, purchase orders, customer information, operation histories, material costs, and shipping records. Your ERP database is the foundation of anything an AI tool will do for your operation.

If the vendor has never connected to Epicor, JobBOSS, Global Shop Solutions, IQMS (now DELMIAworks), Infor, or whatever system you run, they are going to spend the first two to four months of your engagement figuring out the database schema. You are paying for their learning curve. If they have connected to your specific ERP before, they already know where the data lives, how it is structured, and which tables contain the information that matters for the application they are building.

Ask for specifics. Which version of the ERP? Which modules? How many clients running that system? Can they show you data pulled from that ERP in a live demo, not in a slide deck? A vendor who has worked with your ERP before will answer these questions immediately and with detail. A vendor who has not will pivot to talking about their "flexible architecture" and "universal connectors." Those phrases translate to: we have not done this before and we will figure it out during your project.

Question Two: What Happens to My Data?

This question has technical and legal dimensions that matter.

On the technical side: does the vendor's tool send your data to an external server, a cloud API, or a third-party AI model? If so, where is that server? Who has access? Is the data encrypted in transit and at rest? Is your data used to train the vendor's models, which means your job records, customer names, and pricing information could influence the outputs that the vendor's other clients see?

The correct answers vary depending on your operation's requirements. A defense contractor with ITAR-controlled data has different constraints than a commercial job shop. But every manufacturer should understand the data flow before signing. Your job records contain competitive information: what you charge, what your costs are, which customers order what, and where your margins sit. That data leaving your network without your full understanding of where it goes and who sees it is a business risk, not a theoretical concern.

On the legal side: who owns the outputs the AI tool produces? If the tool generates a quote recommendation or a scheduling suggestion based on your data, does the vendor claim any rights to that output or the underlying analysis? Read the terms of service. Most vendors' standard agreements include clauses about aggregated data usage that are worth scrutinizing.

The best arrangement for most manufacturers is a tool that runs on your data without sending it to a shared environment, produces outputs that you own completely, and does not use your operational information to train models that serve other customers.

Question Three: What Does the Demo Use?

Every AI vendor has a demo. The question is what data the demo runs on.

A demo running on generic sample data tells you that the vendor can build software. It does not tell you that their tool works with your data, your part numbers, your ERP's specific field structure, or your operation's particular workflow.

Ask for a proof of concept on your data. Provide a sample dataset: a few hundred job records from your ERP, anonymized if necessary. Ask the vendor to show you what their tool does with actual information from your operation. A vendor who is confident in their product will agree to this. A vendor who pushes back, citing timelines, additional costs, or scope concerns, is telling you something about how much work stands between their demo and a working tool in your shop.

The proof of concept does not need to be a full deployment. It might be a one-week exercise where the vendor connects to a copy of your ERP database and shows you the similar-job search results for a few recent RFQs. The point is to see how the tool handles real manufacturing data with all its inconsistencies: incomplete descriptions, non-standard part numbers, missing fields, and the general messiness that accumulates in an ERP over years of use by dozens of different people.

AI tools that work well on clean sample data and fail on real manufacturing data are common. The proof of concept reveals this before you commit.

Question Four: How Do You Measure Success?

A vendor who cannot articulate a specific, measurable outcome for their tool is selling technology, not results.

Good answers to this question include: "We measure quote turnaround time reduction, typically from an average of 3 to 5 days down to under 1 day." Or: "We measure the accuracy of cycle time estimates versus actual floor times, targeting a reduction in variance from 25% to under 10%." Or: "We measure the number of labor hours spent on manual data entry before and after, targeting a 60% reduction."

Bad answers include: "We improve operational efficiency." Or: "Our clients see significant ROI." Or: "We help you make better decisions with AI." These statements contain no measurable claim. They cannot be evaluated at the end of the engagement. They provide no basis for determining whether the tool worked.

Push for a specific metric, a baseline measurement methodology, and a timeline for evaluation. If the vendor expects to be measured on results, they will help you define those results clearly. If they resist specificity, they are not confident the tool will produce measurable improvement in your operation.

Question Five: What Does It Cost After Year One?

The initial deployment cost is the number vendors want to talk about. The ongoing cost is the number that determines long-term value.

AI tools in manufacturing typically carry three categories of ongoing cost. The license or subscription fee for the software itself. The infrastructure cost for any cloud services, data storage, or API usage the tool requires. And the maintenance cost for updates, bug fixes, and adjustments as your operation evolves.

Ask for a three-year total cost of ownership. Include implementation, training, the first year of operation, and projected costs for years two and three. Some vendors front-load the economics with a low implementation cost and recover it through high annual subscriptions. Others charge more upfront and have lower ongoing costs. Neither model is inherently better, but you need to see the full picture to compare.

Also ask what happens if you stop paying. Does the tool stop working entirely? Can you export the data and analysis it has generated? Are you locked into a platform that becomes more expensive to leave as you accumulate more data in it? Vendor lock-in is a real concern in manufacturing technology. The tools that serve you best are the ones you choose to keep using because they deliver value, not the ones you keep paying for because switching is too expensive.

Question Six: Who Built It?

The team building the tool matters as much as the tool itself.

Ask whether anyone on the vendor's team has worked in manufacturing. Have they been on a shop floor? Do they know what an ERP dispatch list looks like? Can they name the difference between a job shop and a process manufacturer? Do they understand why quoting in a 40-person precision machining shop is different from quoting in a 500-person automotive stamping plant?

Vendors staffed entirely by software engineers who have never been inside a manufacturing facility will build tools that are technically sound and operationally disconnected. They will miss the workflow details that make a tool useful versus annoying. They will design interfaces that make sense to a software developer and confuse an estimator. They will underestimate the data quality challenges that exist in every ERP that has been running for more than five years.

The vendors that produce the best results in manufacturing are the ones whose teams include people who have walked the floor, talked to operators, sat with estimators while they build quotes, and watched production planners assemble the Monday morning schedule from four different screens. That operational understanding shapes every design decision in the tool.

Question Seven: Can I Talk to a Reference?

Ask for references at manufacturers similar to your size and type of operation. A case study from a 5,000-person automotive OEM does not tell you anything about how the tool works in a 60-person job shop. The scale, the systems, the data volume, and the organizational complexity are too different.

When you talk to the reference, ask three things. First, how long did it take from signing the contract to having a working tool in daily use? The answer reveals the implementation reality versus the sales pitch. Second, what did not work as expected and how was it resolved? Every implementation has friction. The question is whether the vendor responded quickly and effectively. Third, would you hire them again? The answer to this question, and the speed and tone with which it is delivered, tells you most of what you need to know.

The Decision Framework

After asking these seven questions across the vendors you are evaluating, the field narrows quickly. The vendors who have worked with your ERP, can demo on your data, measure specific outcomes, provide transparent pricing, have manufacturing experience on their team, and supply referenceable customers in your size range are the ones worth a deeper conversation.

The best next step is a paid pilot with the top candidate, scoped to a specific problem with a defined success metric and a 60 to 90 day evaluation window. The pilot answers the only question that matters: does this tool produce measurable results in my operation, with my data, for my team?

Everything else is marketing.