Model 01 Day 743: PVT

Hello from Oakland!

TL;DR: PVT assembly is done; Mass Production should start next month.

We have quite a bit of news to report this month. Probably the biggest and most exciting news is that we’re (finally) no longer three weeks away from starting the pilot run (also known as PVT). Roughly 85 PVT units were produced last week and are finishing up 72 hours of ‘burn-in’ testing now. On Wednesday, our third-party QC agency will double-check them as they go into their boxes. On Thursday or Friday, an express shipper should pick them up and put them on a plane destined for our fulfillment partner in California.

From there, they’ll be dispatched to some early Kickstarter backers.

(We know we haven’t sent your surveys yet. We’re working on the survey design with BackerKit now. Look for email about the survey in your inbox in the next week or so.)

Long time, no backer update

It’s been about six weeks since we last wrote. The first three weeks were pretty quiet, but the last three weeks have been mind-bendingly busy. The past week was really, really busy. We live-tweeted the PVT run.

The first few Model 01s start their 72 hour burn-in tests

Algernon collected our tweets from last week into a series of moments:

Jesse just returned from two weeks in Shenzhen. The plan was for him to spend a little over a week there finalizing the keycap laser engraving, checking out last minute details and supervising PVT on Monday, June 19. Things didn’t go exactly to plan, but as you’ll read, things went!

Keycap factory

Jesse’s first day in Shenzhen, he visited our keycap factory to check in on updates to our injection molds and to talk through some issues with the trays they made for the laser engraving factory to use during the painting and engraving process.

Visiting with our keycaps’ injection molds

They promised to get things fixed up right away and to produce sample keycaps within a day or two. When the caps still hadn’t shown up by Friday morning, we got a little antsy and asked what the problem was. Our factory’s team said they weren’t quite sure, since the keycap factory claimed to have already completed the modifications.

At this point, we were prepared for any number of possible explanations or excuses. We’ve had a lot go wrong during the manufacturing process and thought that we were beyond being surprised.

The factory called up the keycap supplier, who told them that the modifications weren’t done… and that getting them done that day might be difficult.

Because the factory had caught fire.

They reassured us that nobody had been hurt (and that our tooling was fine) and seemed to be up and running by the next day.

They did manage to get 120 sets of keycaps and painting jigs done in time for PVT, though the painting jigs appeared to still be of the older design that was prone to warping. Our factory doesn’t know why the keycap factory hasn’t delivered the newer painting jigs, but has promised to get to the bottom of it.

Keycap laser engraving factory

On his second day in Shenzhen, Jesse and the team from the factory braved typhoon Merbok to pay a visit to the keycap laser engraving factory in Dongguan. The primary issue on the agenda was figuring out why the keycap labels kept ending up significantly offset from where we’d put them.

 

We’d been sending files to the factory that showed the key labels with a variety of annotations to help the laser operator place them correctly. We’d gone so far as to build a layout that matched the painting jig to the millimeter.

Once we got to the keycap factory, the laser company took the sample keycap sets we’d brought with us and painted them black. They put them in the drying oven and suggested we go for lunch.

An engineer at the laser-engraving supplier aligns key legend

When we got back, we watched as the laser engraving technician printed out a copy of our alignment template, deleted all of the guide lines from the file, imported the legends into the laser engraving software, and proceeded to try to eyeball the correct label placement.

Going in, we were annoyed at how far off the legends were. In retrospect, it’s astonishing how good his manual placement was.

After a bit of back and forth, we found a workflow that seemed to work better. We convinced him to keep the keycap top edge outlines in the drawing and to use those to make sure he was placing the labels correctly. That seemed to work better.

PCB Fabrication

The factory’s regular PCB fabrication company wasn’t able to make the boards in time for PVT, so they ended up using another supplier we recommended. These are folks that friends of ours have used on numerous occasions. Instead of the 20 days the factory’s regular supplier quoted, these folks quote 6-7 day lead time. Well, they usually quote 6-7 day lead time. This time around, they quoted a 10-12 day lead time. Twelve days would have had the boards done on the Thursday before PVT began. But on Thursday, they told our factory that the earliest they could promise delivery was Monday. When the factory pushed back, they said that if we didn’t like the lead time, we could try another supplier.

The factory escalated to a customer service manager who said that they could do Sunday, but that was the best we’d get. Sunday delivery meant that PCBA wouldn’t start until Monday, which would delay PVT. This was not ideal, but we seemed stuck. Asking around a bit, some friends mentioned that the supplier had been “closed by the government for some reason.”

We reached out to Ken Li, our amazing project management consultant. Ken said “What’s our order number? Hang on a minute.”

Ken came back about 10 minutes later saying that he’d reached out to the CEO of the PCB factory, whom he happens to know, and that the CEO had promised our PCBs would be delivered on Saturday. In the end, he was as good as his word and the boards arrived late on Saturday.

PCBA

The factory’s regular PCBA (PCB Assembly) shop is located in the same building as the factory. Unfortunately, they don’t work on Sundays and the factory couldn’t get them to budge, so, in an effort to catch up to the promised PVT schedule, they reached out to another supplier who do work on Sundays. This was the first time they’d worked with this supplier. And, as of last Wednesday, it is the last time they will work with this supplier.

This diode should never have made it through the PCBA shop’s QC process. Consequently, the factory won’t be using this PCBA shop again

Every single PCB had problems. Most of the issues were mis-soldered diodes. The boards looked like they’d been assembled by hand, rather than by machine. They also looked like nobody had done a simple visual inspection after assembly.

Visual inspection is a standard part of commercial PCB assembly. Any halfway decent vendor has automated equipment to do visual checks of boards after assembly.

The factory ended up having to “rework” every board to fix surface mount assembly issues. This ended up being a two day setback. In hindsight, we would have been better off if they’d used their regular SMT shop, like they’ll be doing for mass production.

Type C Cables

We thought we had a workable Type C cable for PVT, but when the factory placed the order with the supplier, they admitted that they didn’t actually have the cables or the required raw materials to make more in stock and that we’d be looking at a lead time of 3 weeks or more.

Plan B was to wake up early on Saturday morning and go to Huaqiang Bei, the world’s largest electronics market and buy some cables retail for PVT. Ken offered to come along and to bring a friend who was a professional cable maker who wanted a chance to pitch for our business.

Over coffee, he convinced us that we really, really didn’t want to buy retail cables in the market and said he could build cables to our spec, on our deadline.

He ran off to find some samples of a few options. When he got back, we talked through our requirements and he promised to send samples.

Four days later, we had three sample cables that looked like we expected. Shortly after Jesse headed home from Shenzhen, the vendor delivered all the cables for PVT.

PVT production

Jesse didn’t tell the factory it was his birthday, but they still delivered quite the birthday present. The factory fired up the assembly line and produced all of the PVT units on Wednesday, June 21.

We’ll talk more about the assembly line in a future update.

QC

The next day, Jesse sat with the factory’s QC team and our third-party QC agency to review the assembled units and to start to identify what quality issues they need to look out for and what we generally consider acceptable. For the sorts of parts they’re used to dealing with, this is pretty easy. For some things, like the wooden enclosures, we needed to be a bit more detailed.

This wooden enclosure failed QC because the edge had been sanded too much, rendering it flat, rather than rounded.

Wood is, of course, a natural material without a uniform color. We showed the factory examples of unacceptable problems, which included tooling marks on visible surfaces, wood filler on visible surfaces, and ugly discolorations. At the same time, they pointed out issues they thought we’d reject. For the most part, we agreed with them.

Model 01 keyboards that have passed our QC checks

It was a pretty long day, but we managed to go through about 70 keyboards. The way it worked was that we’d choose a category of defect to review (like “keycaps” or “wood”) and then queue up each keyboard. The factory’s QC team would check the keyboard and either pass it or reject it. If they rejected it, they’d add a little arrow sticker pointing at the problem. If they passed it, they’d hand it to our QC guy. He’d review it, either adding a sticker and rejecting it or pass it and hand it to Jesse. He’d review the keyboard. If Jesse didn’t pass it, we’d stop the line, discuss the defect and the QC team would modify their acceptance criteria.

At the end of a given pass through the keyboards, we’d all review the rejected keyboards together to make sure they weren’t being overly critical.

This “birdseye” Model 01 is the one Jesse gave Kaia for our 10 year wedding anniversary. (If you look closely, you can see a keycap that failed QC and had to be swapped out.)

The one surprise for everybody was “birdseye figure” Maple. Our jaws dropped when we found a few examples of birdseye maple in with the PVT keyboard enclosures. They were gorgeous. The factory’s QC team thought they looked funny and didn’t look like they were supposed to. It took a bit, but they now understand that if any birdseye Maple makes it into the supply chain, they shouldn’t reject it. (Right now, we’re not able to promise anyone a birdseye figure enclosure. It’s very much luck of the draw.)

At the end of the night, the factory packed us up a carton of six keyboards, which Jesse hand-carried back to California, where we’ll be doing a little bit of additional QC.

In-house QC

Issues we found during PVT

By far, the biggest issues we ran into during PVT were with the quality of laser-engraving on the keycaps. At some point, we’ll do a writeup of the root cause of the keycap issues, but that’ll be a few thousand words on its own.

Checking every keycap by hand

We ended up with an eye-watering defect rate of around 25% on the laser engraving for the keycap sets. The two biggest failures were poorly aligned legends and keycaps that had been laser-engraved backwards. The proximate cause of these problems was that the keycaps need to be moved (manually!) from one painting jig to another in order to make sure the bottom edges of the keycaps get good paint coverage. Because this process requires a human to touch every single keycap, it’s a point where errors get introduced.

These are a subset of the keycaps that failed QC. The factory lined them up and counted them to help drive-home the magnitude of the problem to their suppliers

Needless to say, this isn’t going to work for mass production.

We’ve spent a bunch of time with the factory going over process improvements for mass production. They want to do a bit of research and testing before deciding on a final solution, but we’ve got a few promising options.

What’s next?

Now that PVT production is “done”, there’s still a lot to do before you get your keyboards. Thankfully, almost all of it is straightforward for the factory.

Once we get the PVT units to the states, we’ll ship them out to willing early backers. (There will be a question on the backer survey about it.) After that, we’ll be eagerly awaiting feedback from folks who get PVT keyboards to make sure that we’re not missing something obvious.

The factory will be subjecting a few Model 01s to torture tests. They’ll be exposing them to a few days of ultra-bright UV, spraying them down with a saline fog, and dropping them on a concrete floor. While we sort of wish we could be there to watch these tests, we’re probably better off not having to watch the torture taking place. That should take about two weeks.

The factory will also be sending one keyboard to the FCC test lab to send onward to their partner lab in Taiwan for final testing and issuance of the “Declaration of Conformity”. That should also take about two weeks.

At the same time, the factory is working to refine their internal test and assembly process based on the PVT run. That’ll let them build Model 01s faster and more reliably when they have to make a few thousand of them at a go.

Right now, we’re expecting the first couple thousand units to roll off the line in July.

<3 j+k

Back to blog