Inside OCZ's Factory: How SSDs Are Made
by Kristian Vättö on May 20, 2015 8:30 AM ESTAt CES I had the opportunity to sit down with OCZ's CEO, Ralph Schmitt, to discuss the state of OCZ after Toshiba acquired the company in late 2013. We talked about how the company has changed and evolved under the new ownership and how Toshiba has brought in some much needed NAND supply and expertise. In the article we posted summarizing that discussion I also mentioned that I would be taking a closer look at OCZ's manufacturing and validation in the coming months, and today's article focuses on that.
OCZ flew me out to Taiwan to get an in-person look at the factory and I had Jason Ruppert, Senior Vice President of Operations, and Jim Van Patten, Vice President of World Wide Quality, as my hosts and guides throughout the trip, and a big thanks to both of them. A bit of background, to ensure these were the people to speak to: Mr. Ruppert has been with OCZ since March 2012 and before joining OCZ he was the Vice President of Manufacturing Operations & Engineering at Harmonic Inc, which focuses on video delivery infrastructures. Mr. Van Patten was actually Mr. Ruppert's first hire and joined the company in May 2012 from Logitech where he was the Vice President of World Wide Quality Assurance. Mr. Ruppert holds a Master's degree in Systems Engineering from North Carolina State University (for your interest, Anand did his BS in Computer Engineering in the same university), while Mr. Van Patten received his Ph.D. in Instructional Design & Evaluation from Syracuse University. It goes without saying that both Mr. Ruppert and Mr. Van Patten have extensive knowledge and experience within their operational areas, so the two were the best people to guide me through the manufacturing and validation.
The Development Process of an SSD
Before we move on to the actual factory tour and see how SSDs are made, let's outline the development process first. After all, a product must be designed and developed before it can be manufactured and there are some items that show up in both development and manufacturing processes.
As with any product, the development process starts from an idea, which can be practically anything (completely new model, refresh of existing model with new NAND, higher capacity model, different form factor, new software etc.). In phase zero the idea is shaped to become a concept and usually results in a short (1-2 pages) document that describes the opportunity presented by the product.
Once the concept is clear, the marketing and engineering teams will give their initial feedback. Both are very important because a product must be marketable, but at the same time it needs viable to execute from the engineering standpoint. Normally phase one takes about three weeks and results in a more in-depth description of the opportunity, and if it fails at this point it is either scrapped or moved back to a concept stage.
In phase two, OCZ starts to commit more significant resources to the project. The first two phases merely outline the concept and determine its opportunity, so phase three begins the actual planning of the product. The two key documents that are finalized in this phase are the marketing requirements and engineering response documents, but each function group (e.g. quality and supply chain) will also deliver their support plans. Basically, the purpose of phase two is to construct a comprehensive project plan that includes all aspects and teams involved in the product, including the budget.
Phase two is probably the most critical phase because the project plan and budget are used to decide whether OCZ puts hundreds of thousands (or even millions) of dollars behind the product, so all documents must be carefully made and evaluated in order to make the best decision for the company. The length of phase two depends on the complexity of the product and what teams need to be involved, but it typically takes from one to three months to build the final plan and budget.
If the project is funded, OCZ moves to phase three, which is where most of the engineering work is done. OCZ of course wants to keep the exact details of this phase close to its chest, but the ultimate goal is to build the first working prototypes, so the project can move to testing the prototypes. While the engineers are busy with their work, the remaining teams work on their own functions and prepare to manufacture the pilot samples (this includes tasks such as qualifying suppliers, securing long lead time parts, developing preliminary spec sheets and marketing materials). The length of the design and implementation phase depends greatly on the product, but even a drive that uses an existing controller can spend up to a year in this phase. The development of a totally new controller like the JetExpress is obviously a multi-year project given the sheer amount of engineering work required.
The Validation Phase
Phase four, which is validation and qualification, essentially consists of three main parts: the Engineer Verification Test (EVT), the Design Verification Test (DVT) and the Production Verification Test (PVT). EVT is run on the first engineering prototypes and it tests that the drive works in real life as it was designed to work on paper. The test suite is relatively straightforward and tests aspects such as power levels, signals and interface timings to ensure that the prototype works as it was planned to. There is also some preliminary performance testing in EVT phase, but because the firmware is usually far from final the results almost never illustrate the performance of a final product.
DVT is further broken down to two areas: normal DVT and quality/reliability. The normal DVT has a broader set of tests than EVT and more variables (e.g. power, temperature and host variations) are added to the mix to ensure that the drive operates and performs as it was supposed to in a variety of environments. Each test is also run on at least four samples, whereas the initial EVT testing is usually performed on just one or two samples. I'm not going to list and describe every individual test here because the DVT phase consists of dozens of different tests, but there are product compliance, data retention, power loss and die failure tests to mention a few, along with thorough performance testing to evaluate the firmware.
The reason why DVT is split into two parts is because EVT and normal DVT are both performed by the engineering team (who also designed the drive), which can create conflicts of interest during the validation process (in the end, human beings tend to be blind to their own faults and mistakes). Most of the DVT tests are rerun as reliability tests, but in this case the tests are performed by the independent quality team that is lead by Mr. Van Patten. The number of samples is also considerably higher and each test is run for a longer duration to verify the reliability of the design. Again, the full list of tests is several pages long, but aspects such as durability against vibration, shocks and low/high temperatures are tested in addition to the normal DVT tests. Basically every spec that is mentioned in the spec sheet is tested in this phase, including all standard JEDEC tests and certifications.
The first level of PVT tests are also run later during phase four and focus on the reliability and repeatability of the manufacturing process. Basically, the purpose of the PVT tests is to ensure that every drive coming out of the mass production line will be of the same quality and that is done by examining the drives from the production line using dye-and-pry and x-ray to inspect the PCB for any defects caused by the soldering process. The other PVT tests evaluate the readiness of the factory's quality system (incoming and in-process quality inspection, final quality control and out of box inspection) to make sure that all quality control phases are capable of separating good and bad, and that no defective products will get through to the customers. Ongoing Reliability Testing (ORT) is also set up to test a few drives from every production run to guarantee that nothing changes over time.
The total length of the validation phase varies greatly. It can be as short as two months if the design is relatively simple and similar to previous ones, but it can easily take over six months for more complex ones. Usually OCZ creates 2-5 sets of engineering samples during validation as issues are found and fixed, but there isn't really any preset duration for validation -- it always depends on what is found during the verification and how significant modifications are needed. Ultimately a drive cannot move to the next phase until it passes all quality and reliability tests, so setting a strict deadline would be a bad idea (for the company and for consumers) to begin with.
Entering Production
In phase five the drive moves from engineering and verification to operations (i.e. manufacturing), which usually takes 3-6 weeks to complete. Final PVT tests are conducted to ensure that the manufacturing quality meets the specifications and that necessary tests are in place to spot any changes/errors in the production. Other teams also finish up their actions to be ready for the launch and this is also the point when OCZ contacts us and other media about an upcoming product launch and sends out the review samples (i.e. the samples we get are typically manufacturing pilots as the mass production hasn't begun yet).
When the manufacturing side is ready to start putting out the new drive, a public announcement of the new product is made and the mass production as well as shipments to customers begin. As part of this visit, we had an inside look into the mass production side of the equation.
64 Comments
View All Comments
dreamslacker - Thursday, May 21, 2015 - link
Not quite. You can only do that if the entire process is automated and centrally controlled.Their SMT line uses tape reels for parts, if you wanted to pre-test NAND flash before mounting them, you will need to extract, test and reinsert into the tape reels (or trays) in a specific order.
The mapping information from testing will also need to be entered into a centralized database. The reels or trays will then need to be sent to the SMT line in the same specific order and the circuit boards, once soldered, will also have to be sent to the firmware flashing racks in order as well. Otherwise, there is no way of telling which board contains which flash chips.
Again, they have to extract and send for labeling in a specific order so that the label serials will match what is programmed into the drive.
All of these basically means you need to invest into a completely automated line with conveyor belts, robotic arms, automatic labeling machines, database servers and apps to integrate.
It's simply more economical, for a smaller scale operation like this, to solder first and ditch the defective units as long as your components do not come with a high defect rate.
caleblloyd - Thursday, May 21, 2015 - link
I think the article got posted Tuesday morning then taken down a few minutes later. But all good now!dreamslacker - Wednesday, May 20, 2015 - link
Odd that they intend to both shift towards a rack-based system and also MST at the same time.A decent rack type system would be running ASICs that offload the test regime from the host computer(s) and can be configured on-the-fly to run both drive tests as well as programming.
DanNeely - Wednesday, May 20, 2015 - link
Even if most of the testing can be done on board the drive itself using the MST; they still need to plug the drives into something else to verify the sata connection and to read out the results of the MST. MST will just let them swap new batches of drives into the tester more frequently.dreamslacker - Thursday, May 21, 2015 - link
MST is normally used when you have reduced capability test chambers / racks. For mechanical drives that need to be tested under high temperature conditions, it allows them to dump a load of drives into what is effectively a large oven with triggered power supplies for thermal tests.If they are utilizing 'smarter' racks, such as the Xyratex (now owned by Seagate) modules, then the ASICs will handle everything and loading/ unloading MST firmware manually isn't needed.
The primary difference is that a smarter ASIC based unit can automatically run the tests, dump the logs and prep the drive for sale (clear the test data, reset the counters etc) whereas MST requires the worker to extract the drives and send them to another workspace to have the MST logs dumped and the production firmware loaded in.
The ASIC based test racks still connect to a computer, you just have the ability to use a single computer for more drives (for Seagate's Xyratex modules, it's 1 computer to 192 drives). They are, however, headless units and are fully controlled by a central server via CGI & Python scripts.
Very little user intervention is required on their fully automated line except when they need to re-configure the racks to accept modules with different drive interfaces. This is done by a single terminal console where they telnet into the machine handling the modules and edit the configuration file by VI.
GTan36 - Wednesday, May 20, 2015 - link
Bought a OCZ Arc100 240GB SSD last week for my first PC build. It's really fast and you can tell the drive is high quality and manufactured well. Boots Windows 8.1 in less than 15 seconds and was pretty affordable.It's nice to see OCZ becoming a quality brand.
r3loaded - Wednesday, May 20, 2015 - link
Excellent article, it was like a written version of a How It's Made episode!Phynaz - Wednesday, May 20, 2015 - link
Presented by Anandtech, a division of OCZ Marketingxthetenth - Wednesday, May 20, 2015 - link
Yes, this tour of a company's factories and discussion of how they make things is different than all the others becauseMrSpadge - Wednesday, May 20, 2015 - link
Don't fault AT for OCZ being more open than other manufacturers.