Adventures in System Integration
May 1, 2009
The success of automation projects depends on a number of factors-not least of all the skills of the system integrators.
The system integrator who wouldn’t leave isn’t a legend but an unhappy reality of automation projects of the not too distant past. Open architectures and standardized hardware are squeezing much of the time and expense out of projects, allowing food and beverage companies to begin generating returns on their automation investments more quickly. But careful planning is a precursor to vertical startups.
Evidence of the benefits of open systems was on display recently in Cologne, Germany, host to the triennial Anuga FoodTec trade show. Encouraged by the reception to a 2006 demonstration, organizers of the Robotik-Pack-Line project strung together automation equipment from 30 vendors to assemble, package and case-pack hamburgers untouched by human hands. “The digital factory is the future trend in the food sector,” proclaimed Klemens van Betteray, vice president of CSB-Systems AG, which coordinated the integration work. Ulrich Mergemann and one other technician handled the potentially painful programming over the course of three weeks.
Planning the project, on the other hand, played out over a year and a half. Manufacturers speak with one voice when demanding open systems that slash integration time and cost, but there are no shortcuts to solid planning. There also is little consensus among professionals who draft those plans on how best to organize them.
Maverick Technologies’ Dean Ford advocates an agnostic approach to the selection of automation software, postponing the choice until the need defines the solution. But few projects begin with a clean slate, automation experts argue, and that may be a good thing. “Most projects start with assumptions about certain vendors and suppliers,” suggests John Gunst, packaging design manager at Boise, ID-based POWER Engineers Inc. “It’s human nature to pick people and solutions we think are going to make us successful.”
A clean-slate approach was adopted at Truitt Brothers Inc. The Salem, OR, firm got its start in 1973 when brothers David and Peter Truitt acquired a cannery that has operated since 1917. By the end of the ‘70s, the company had started pilot-scale production of ready-to-eat meals in flexible pouches, and a plant adjacent to the original cannery followed several years later, recalls David Truitt, vice president. Instead of continuous retorts, end-over-end rotary water-immersion units were installed for copack projects because of “the quicker cook, which leads to a better quality product, and the flexibility they give us to use any type of package,” he says.
The Truitt’s intuition, that prepared meals in heat-seal trays, pouches and other alternatives to the can would find a wide audience, proved correct. Six years ago, the firm acquired a third plant in nearby West Salem and outfitted it with a dozen rotary retorts. As with most gradual expansions, units from virtually every major retort manufacturer were represented on the three production floors. By the time six new units were added in 2005, the lack of controls standardization had become a glaring problem. “Some of the older controls no longer were being manufactured and were barely supported,” according to Scott Williams, vice president at Covington, LA-based Allpax Products Inc., which supplied the new battery of retorts. Truitt expanded the project scope and directed Allpax engineers to rip out all the retorts’ controls and replace them with Allpax’s system.
Truitt’s units produce “at least a couple of hundred different products,” David Truitt says, adding a high degree of recipe-management and record-keeping requirements. The Allpax controls package was built around the requirements of 21 CFR Part 11, the regulatory guidance for documentation and track-and-trace requirements. “It’s a more robust control system,” observes Rick Kimball, Truitt’s technical director. “It provides the type of detail we want.”
“The control system provides iron-clad security to prevent record tampering while using open architecture and off-the-shelf software,” says Williams. Several years ago, Allpax began building a team of software engineers, shifting responsibilities previously handled by electrical engineers. “The electrical engineers’ programming was functional, but it was not robust,” says Johnny Watkins, Allpax’s director-software engineering. “Now, if something times out of the norm and the machinery hiccups, the controls can prevent a fault because the scenario has been anticipated.”
“We were putting in new capacity,” Truitt continues, but demand from new and existing customers was outstripping the company’s ability to meet it. The brothers determined it was time to get closer to Eastern customers. A site search ended 2,500 miles from Salem in East Bernstadt, KY, where the company’s fourth production facility opened in December.
While the controls standard already had been set for the retorts, the ammonia-based cooling system, wastewater treatment plant and electrical design required integration work beyond Allpax’s scope. Site supervision also was an issue. Allpax tapped Maumee, OH-based Matrix Technologies Inc. to handle those tasks. “An existing structure always poses challenges,” Matrix Director David Blaida says of converting a former food distribution center in East Bernstadt. “Lots of piping was required, the roof had to be fortified with additional trusses, and new flooring and drainage had to be installed.” Competence in project management supplements Matrix’s original integration focus to satisfy manufacturers’ preference for turnkey solutions on these types of projects, Blaida says.
Take my integrator, pleaseMatrix’s involvement at Truitt underscores the expanding role of traditional system integrators in automation projects. E2M, the Atlanta sister company of Polytron Inc., handled piping design, controls programming and mechanical-construction management for some of the digital blending systems deployed by Rockwell Automation at 35 Frito-Lay (F-L) plants in North America. F-L engineers had pigeon-holed Rockwell as a controls vendor, and the delivery of a turnkey skidded solution was a revelation, commented Mike Walker, an F-L vice president who discussed the project at November’s Rockwell Automation Fair in Nashville, TN.
Digital blending is standard procedure in soft drink production, but as the number of variables increases, controls complexity expands. Brewing and fermentation “is a fairly complex process,” notes Jonathan Baer, a market development manager at Milwaukee-based Rockwell, and in recent years the automation firm has focused on developing a repeatable, standard fluid routing and blending solution on a process controller platform for the beer segment. The brain is a digital blending algorithm that uses PLC code blocks to standardize functions that can be scaled up to mix more than 20 different feed lines.
“Digital in-line blending is geared toward retrofit opportunities rather than a ground-up mechanical solution where ripping out existing systems is involved,” explains Baer. “Once installed, it’s fairly simple to configure to the user’s systems.”
While F-L’s plants are scattered coast to coast, oil processing plants are primarily in the Midwest. Instead of relying on oil processors to blend corn oil, soybean oil and sunflower oil, the salty-snack company wanted to bring blending in house. Three options existed: ratio blending, batch or digital. The highest accuracy and shortest cycle and lead times could be realized with digital, and the technology had the additional benefit of being the least costly option. Buying oils at favorable prices and blending them in house saved enough to cost-justify the project, Walker said, and fewer truck shipments slashed costs further and supported the corporate goal of greenhouse-gas reductions.
“We did not want to disturb legacy control loops if possible,” he added, and a standard solution with massflow meters that were self-monitoring and could be calibrated automatically had appeal. Planning for the unexpected is part of every project, and after eight weeks, the massflow meter on the first installation failed. It was replaced, but eight weeks later it failed again. Engineers discovered the frequency of the pump matched a critical frequency for the meter, causing it to vibrate to failure. Reprogramming to block out the frequency resolved the issue, and the rollout resumed, Walker said. By November, in-line blending of oil was in place at 27 facilities, with the remaining eight expected to be operational by the second quarter 2009 deadline.
Equipment failures that take months to manifest themselves are an argument for long rollout schedules. The longer the schedule, the more important the upfront planning and detailed discussions. Otherwise, priorities will shift and “scope creep” will take hold, cautions POWER Engineers’ Gunst.
Plan for successA detailed “design criteria matrix” is pro forma at the start of projects, says Gunst, but if managers without budget responsibility make amendments, seemingly minor requirements ripple through and drive costs beyond budgetary limits. An assertive executive is needed when establishing the matrix, and the integrator must enforce the plan.
Gunst recalls a greenfield project where the manufacturing executive strictly specified every element, determining which areas required redundancy and which it could do without. “Over the course of two years, as new people became involved and changes to the design criteria for the line were requested, I told them, ‘You tell the boss why this change is needed,’” he relates. The original plan held.
Lack of planning, poor communication and failure to follow through characterize projects that go south, believes Scott Richards, a project manager with E2M/Polytron Inc., a Duluth, GA, integrator that also delivers electrical and mechanical engineering services. The assumptions and caveats of plan details need to be explicit to head off disagreements later on, he adds.
To illustrate, Richards cites a project requiring large amounts of steel. By the time materials were bid, steel prices had spiked, forcing cutbacks in other areas. Commodity prices often are volatile, and that needs to be understood by all parties at the outset. “It might be 18-24 months before you buy steel,” he points out. “The risk of higher prices needs to be put up front.”
On the surface, integration projects seem straightforward, but competing agendas and unexpected snafus can derail even the most detailed plan. Experience teaches where many of the upsets may occur, though unanticipated developments always loom. Technical expertise and relationship skills must be brought to bear if automation projects are to end in vertical startups.
For more information:
T. Scott Williams, Allpax Products, 414-427-6322, firstname.lastname@example.org
Ron Margo, Hi-Tech Control Systems Inc., 920-965-1717, email@example.com
David J. Blaida, Matrix Technologies Inc., 419-897-7206, firstname.lastname@example.org
Dean Ford, Maverick Technologies LLC, 443-876-5217, email@example.com
Scott Richards, Polytron Inc., 678-328-2969
John Gunst, POWER Engineers Inc., 208-288-6100
Jonathan Baer, Rockwell Automation, 414-382-2000, firstname.lastname@example.org
Please click here for a list of Control System Integrators Association Food & Beverage Specialists.