State of the Kossel Pro

Hello all,

It is time for us to give an official update on the status of the Kossel Pro project and Kickstarter campaign.  The first half of this update will be administrative in nature, followed by an in-depth technical outlook.

Project Administrative:

Between our Shopstarter and Kickstarter campaigns, we took on a liability of approximately 150 printers, of which 10 are to be pre-assembled.  To date, we have delivered approximately 100 of the 140 kits, and we expect to deliver the remaining 40 kits and assemble the 10 pre-assembled machines in the first 2 weeks of Feb.  We found that we ran short on a few critical components such as linear rails and motors – so we built all the printer kits we can build prior to Christmas break, shipped them to everyone who we got through to confirm address (with a priority given to Kickstarter backers, as they have been waiting the longest) and used the lull to work on documentation.

This past weekend, we received our last air shipment of linear rails and glass, stepper motors and carbon fiber tubing.  We’ve learned a valuable lesson in Chinese export laws and we’ve since moved our freight consolidation to Hong Kong.  Hong Kong is a free port – no import or export taxes, which drastically reduces the logistics challenge of getting packages out the door.  Once we figured out the proper paperwork to file, it took about 3 days and a mere $850 to get our latest shipment of linear rails and stepper motors shipped, via commercial air cargo (meaning, the cargo flew on a commercial airline flight, in the cargo hold, with people’s luggage.  In our case, it was on an Eva Airline flight out of HKG through Taoyuan airport in Taipei; other times it gets booked onto Korean Airline through Incheon.  Expeditors International handles the booking of the flights (on a space available basis) and clears it through US customs for us.  To save some money, I drove down to the air cargo terminal at Seatac Airport to pick our stuff up from a warehouse right next to the runway).  As a comparison, it took close to $5000.00 to get our last air cargo shipment out of China, via DHL as small parcel, to circumvent Chinese export taxation on the goods.   (It wasn’t that we didn’t want to pay export taxes.  We would have gladly paid them to get our stuff out of China.  The problem is we had no way of paying them.  Because our consolidator didn’t buy the material, he wasn’t able to pay export tax on it and have the books match up to support paying export tax – there would have been no POs from OpenBeam or a money trail to support making the export declaration and paying the export tax)

We have already started the process of kitting and preparing for final push to close out Kickstarter / Shopstarter liabilities.  Specifically, we have hired on additional help and will be using said help for crimping thermistors, heater cartridges, etc, and we’ve already got MOST of the subassemblies kitted.  Very likely, we’ll delegate Mike and Nick, our new tech to handle the 100% QC and packaging of the remaining kits, while I focus on completing the instructions.

Another batch of hot ends in the works.

We’ve lost a lot of “kitting time” due to our air cargo being held up in Hong Kong (since it was going on a commercial airliner, we had to file a declaration of conformance to IATA non hazmat standards, and since OpenBeam technically do not have a business presence in Hong Kong, there was extra paperwork to set up the export as a “known consignor”.  This resulted in our air cargo leaving HKG a lot later than we expected.  On top of that, end of January is when Washington State Business & Occupation taxes are due, and as part of doing our taxes (both state and federal) we needed to shut down and conduct an inventory.  For most businesses, such administrative tasks aren’t very visible externally, but when the entire company’s staff is also the kitting department, the inventory taker, the delivery truck driver and engineer, well, unfortunately these things takes time.

A batch of hot ends took a bit too long of a bath in the simple green... :-(

A batch of hot ends took a bit too long of a bath in the simple green... :-(

Finally, there are backers who had backed us to get injection-molded parts, stamped parts, etc.  We’ll fill these rewards towards the end of Feb.  As for those who have not confirmed their addresses (especially from Kickstarter), we’ll hold onto their printers in our storage locker and ping them every so often.

Life as a post-Kickstarter company

Neither Mike nor I are in a position to quit our day jobs; my wife and I are also expecting our firstborn son in June. There is no way we can continue to develop new products and take care of existing customers, given other pieces of work-life balance.  To make the company sustainable, we’ve planned from the beginning to resell Kossel Pro kits through distributors only and as part of the distributors’ responsibility, have them provide user training and after sales support.  We have since moved future pre-orders to MatterHackers, and we have already started, over a month ago, the procurement process for printers for post kickstarter / shopstarter fulfillment.  We are taking steps to move as much of the kitting process to our overseas contract manufacturer, and we are getting blanket terms from our contract manufacturer as well.  This will hopefully help ease supply chain constraints in the future – we are working towards a future where you can order a Kossel Pro and have it shipped to you immediately, but we are NOT there yet.  

Packing box in "Master Box" configuration to prepare for Amazon FBA shipment

On the home front, we are moving towards Amazon fulfillment of spare parts and individual kits as upgrades for the Reprap community and expect to be online in the next few weeks.  (Remember, we went through a similarly painful transition to move OpenBeam onto Amazon and to smooth out supply chain issues.  Except, back then it took MONTHS and it was our only source of income).  Our current projection for being able to resume Kossel Pro shipment is in April.  (The big schedule time bomb called Chinese New Year in the middle of all this does NOT help things).   

Kossel Pro Documentation

Like most (stereotypical lazy) engineers, we’ve pushed documentation to last.  (There’s 2 reasons for this; one is we really needed to build the printer a few times before coming up with the optimum recommendations for how to build the printer.  To do so requires production parts, which weren’t available early on in the project).  Unfortunately with the shipment out to our backers and subsequent time spent on support, we have not been able to spend time on documentation.

We are diverting as much of our resources as we can towards documentation, and we are using this time period where our subcontractors are working on kitting and prep work to get as much work on the documentation done.  We have now dropped the password protection on KosselPro.com and we will host all documentation there, marking clearly what is still work in progress.

Technical Details:

My boss at my day job always told us:  “If you really want to learn about your product – get a bunch of dudes to build you a hundred of ‘em.  Then line them up in a room, and bang on it”.

Obviously, both with my day job (designing and building scientific instruments, not exactly high volume manufacturing) and with the Kossel Pro, we really can’t afford to do that.  Our December shipment of 100 printers was the largest Kossel Pro deployment into the wild, and we’ve since learned a few things about the printer.  As much as we try to test for and mitigate real life scenarios, these tests are still devised and conducted by engineers who had spent the last two years obsessing over the project.  The real world, as we found out, is highly unpredictable.  We’ve been doing preliminary analysis and debugging on our machines in the field and these are some of the common issues we’d like to address:

  • Heater cartridge inconsistency and tuning
  • Control board failure due to cleaning of hot ends while energized
  • Probe deployment issues
  • USB port strain relief
  • Auto Levelling accuracy

Heater cartridge inconsistency

We purchased the heater cartridges from another manufacturer as “surplus stock” – the supplier had screwed the pooch and shipped them 24V heater cartridges and they use a 12V drive system.   Our specifications calls for a 24V, 40W heater, and working backwards, we needed to deliver approximately 1.667A at 24V (24V x 1.667A = 40W).  To draw 1.667A at 24V, the resistance of the heater cartridge needs to be approximately 14 ohms (Ohm’s law of V=IR; V=24V, I = 1.667, solving for R gives us about 14 ohms).

However, we are seeing, especially in failed units, as low of a resistance at 8 ohms.  At 8 ohms, the heater circuit will dump a whopping 3A of current into the hot end, for a power dissipation of 72W - nearly double the design limit.

Unfortunately, when we did the firmware tuning, we did not think that there was such a wide range of heater core resistance, and even knowing this problem replicating it is still difficult.  We are going to build a fixture to do 100% inspection on our heater cartridges’ resistance and we are going to have to start batching heater cartridges.

We are in the process of talking to Chinese heater cartridges manufacturers for a custom crimped and assembled heater cartridge assembly for the Kossel (we crimped the current ones ourselves).  Some of the things we will be controlling are the tolerances on the outer diameter (they were ALL over the place with our current batch of heater cartridges – we did 100% test fits with our J-Heads before shipping, and on a few of the units, we had to ream out the J-Heads prior to shipping).  We’ll also be specifying a UL compliant, but thinner insulation on the heater cartridge wires.  This will prevent the breaking of the crimped connector issue that we’ve seen with some of the heater cartridges, as we’d be able to properly crimp the connection wires (which involves one set of crimps compressing around the insulation for proper strain relief.

Control board failure due to cleaning of hot end while energized

We've had a few failures of the Brainwave Pro boards in the field.  Most of these boards failed the following way:

  • Print had peeled off the bed, or some sort of catastrophic failure on a print resulted in PLA smeared all over the board.
  • User heats the hot end up and while the board is energized, attempted cleaning with a metal wire brush.
  • Board dies.

In one case, the board’s failure also caused the MOSFET controlling the heater cartridge to fail short.  This is an especially dangerous situation as it results in the hot end overheating and will destroy the hot end.  It is also a HUGE fire risk.

We believe that cleaning the hot end with a metal brush caused a shorting of the 24V power for the heater cartridge to the thermistor contacts.  We followed the standard Reprap electronics design and this line is not protected on the Brainwave Pro (or any other Reprap electronics out there, for that matter).  However, most printer builders run at 12V.  Applying 12V to a microprocessor might make it unhappy, but it appears to be a fairly survivable scenario for the microprocessor.  Exposure to 24Vs though seems to be universally fatal to the microprocessor.

We are still investigating and we will be putting additional protection onto the next run of the Brainwave Pros (in the form of clamping zener diodes to protect the microprocessor lines).  In the meantime, the recommended procedure for cleaning out a clogged end effector is as follows:

  1. Invert the printer, if necessary for ease of access to the hot end.  To be absolutely sure, you can unclip the glass and leave the HBP in place.  Nothing should fall.
     
  2. Bring the hot end up to 180 deg C
     
  3. Using a NYLON (ie, NON conductive) brush, carefully brush away stuck plastic, staying clear of the thermistor wires!

If there's any doubt / questions, just imagine you are cleaning the insides of your computer.  You wouldn't go at it while the computer is energized with a metal brush, right?

We consider the permanent nature of the thermistor wires a feature, not a bug.  We want the thermistor to be impossible to dislodge from the hot end – as mentioned before, disconnecting the thermistor from the hot end during operation is a fatal error that results in equipment destruction and a huge risk of fire.  The unfortunate part about this arrangement is that if the thermistor wires do break the tip will have to be replaced.  It comes back down to not having the space to do a removable / servicable solution on the J-Head’s heater block.  Our next hot end (whose R&D is being accelerated, to deal with some of the issues that had come up) will feature user servicable temperature sensors.

Probe Deployment Issues:

There are a few issues and comments about the probe’s design and its deployment:

  1. The probe standoff loosens, and eventually flexes enough that it doesn’t push down on the switch.
     
  2. Concerns about the general slop and positional repeatability on the probe when deployed.
     
  3. Related to 1 – the probe impacts the plastic housing (and switch) too hard, drastically shortening the life of the switch.

Here’s what we are doing to improve things:

  1. We are revising the assembly instructions on the probe to call out the addition of Loctite onto the threads of the probe to stop it from loosening.  We are going to try to source the M2 FSC with Loctite preapplied for the next manufacturing run
     
  2. We are going to be talking to our vendor about changing the probe arm from round to a square profile.  This should resolve a lot of the positional repeatability issues on the probe.
     
  3. Unfortunately, this is not something that we can address.  The spring force is required to overcome the 75 gram force actuation limit on the switch (the lowest we could spec, by the way).  The nature of hooks law and how compression springs work results in the slam.

For those who are really concerned, the only work around we can offer is that you can deploy the probe by hand.  The switch in question is freely available from Mouser for less than $2.00, and the next generation auto levelling system does away with the auto probe all together.  In the grand scheme of things, this is way too far down the priority list for us to address.

USB strain relief

When we designed the Kossel Pro, we chose a full size USB B port for robustness.  During my time working with the original Brainwave, I’ve sheared my share of MicroUSB ports off that board.

However, what we didn’t factor in is the weight of the USB connector (on the cable end).  We are learning that a vertical USB port really requires additional plastic around the connector to provide strain relief.  One of the warning signs is that we see the sheet metal peeling from the shield of the connector.  This forms a sharp barb that catches the edge of the USB B connector, making it impossible to remove the cable (this happened on one of our engineering test printers as well as one of our user’s printers).

We've also found, on the engineering test printer, that due to the cable not being able to be removed, the additional stress on the board will cause intermittent connection on the USB port, to the point where the board is no longer usable.

To fix this, and to head off a major recall, we are planning, in the next week or so, release a 3D printable part that can be added to frame to provide additional strain relief to the connector.  We are certainly happy to catch this train wreck before more issues appear.

Auto Probe Accuracy

The G29 Auto Levelling routine that the Kossel Pro uses is directly from the Reprap Marlin firmware.  It was written and improved upon by Johann Rocholl, and it’s used by delta printer builders all over the world.

We now have reasons to believe that G29 doesn’t actually work very well. One of the biggest issues that’s been pointed out, by the software team at MatterHackers, is that the least-square-sum method that is used to compute the bed tilt really requires double floating point precision to work, and Arduino can only handle single floating point precision.  As a result, a lot of the precision is discarded and what we are seeing is actually rounding error.

It should be pointed out that this is NOT an OpenBeam Kossel Pro issue, this is a general delta printer issue.  Dudes a lot smarter than me, such as the software team at MatterHackers, are looking into different solutions and options.  Ultimately, the changes will be committed back into Marlin, and everyone in the delta printer community will benefit.

In Closing

We would like to thank all our backers and preorder folks for their patience – this project wouldn’t have been financially possible without our backers – and frankly, also without the support of all the people that have been buying and using OpenBeam for the last two years.  (OpenBeam essentially bankrolled a lot of the development of the Kossel Pro, to bring the project across the finish line).  We are aware of the shortcoming on the documentation and we are fixing them.  We are aware of some of the minor design tweaks required to polish the printer and we are fixing it for the upcoming production run (and making the parts available to our existing backers – we’ll figure something out).

At the just concluded 3D Printer World Expo in Burbank, more than one person had come up to us and complimented us on the engineering on the Kossel Pro – that it is one of the more polished open source hardware printers out there, and that the craftsmanship put into it by all our vendors – our machinist, our injection molder, our OpenBeam extrusion supplier, is very apparent.  We are certainly proud of the engineering that we have done, but over the course of the project, we’ve also identified shortcomings with the Reprap project, and we look forward to fixing and addressing some of these issues we have identified.  We look forward to continuing to carry the torch and innovate, and designing and building better tools to enable makers out there.

Thank you for your support of Open Source Hardware,

-=- Terence and Mike

 

 

 

 

Closing of Pre-orders on ShopStarter

Terence here.  I will do our “year in review” and go over some of the good things that are happening with our most recent Asian trip at a later time.  For now, an administrative update on the Kossel project.

When we first started on our journey, we knew that designing a good product would not be our biggest challenge; as engineers with day jobs, our biggest challenges would be customer service and after sales support.  It had always been our plan to resell Kossel Pro kits through a distribution network and have our distributors provide after sales support.

Today, I am happy to announce that we are taking the next step toward making this a more stable and sustainable company by shutting down our Shopstarter page, and handling all future printer kit sales through our distributors.  We would like to welcome Solarbotics and MatterHackers as our launch distribution partners.

Solarbotics had been involved in the educational and robotics market for a very long time – in fact, I still remember visiting the Solarbotics webpage, in its infancy, as one of the first websites I visited when I first got access to the internet, back in the 90s..  MatterHackers, although a much younger company, is an established filament provider, and an active contributor back to the 3D Printing community.   You will find MatterHacker’s staff giving lectures at every 3D Printing show, such as the upcoming 3D Printer World Expo in Burbank, CA.

Both companies received some of the first Kossel printer kits to roll off the assembly line, successfully assembled the printers (sans instructions ;-) and are printing great parts on their printers.  Both companies are also staffed with makers who are passionate about advancing the state of the art of low cost 3D Printing.  We can’t ask for better launch partners in our endeavor. Mike and I will still be actively involved in the project, and we are committed to providing support to our existing customers, but our role in the future will be supporting our distribution partners, and letting our distribution partners be the first line of response when questions arise.  This way, we can get back to engineering as soon as possible.

So, with that said, we are shutting down the pre-order site on ShopStarter and directing future customers to MatterHackers (US and International) and Solarbotics (Canada, International and Educational Institutions).  We are hoping to have kits to each company to ship to customer by the end of March.  This is a pessimistic projection, and one that we are hoping to beat.  We have resolved all manufacturing issues, and we have resolved our shipping, kitting and consolidation logistics as well. 

For orders already on the books, the goal is to ship all printer kits the week of Jan 26th. We are bringing in additional linear rails, glass sheets and motors to complete the last few printer kits – the merchandise is already cleared for export and will be going via air freight in the next day or two.  We have also already started building all the rest of the kits to part exhaustion– and surplus kits, after fulfilling our Kickstarter and Shopstarter obligations, will go onto our Amazon web store.  We have already started loading some Mini Kossel and Kossel Common Core specific parts onto our web store.  Additionally, we are finishing up the instructions, and we are testing these instructions with a builder who will be building up all the pre-assembled machines for our backers who have ordered pre-assembled printers.

Finally, one of the cool things we brought back from Asia, is a new photo studio setup.  This will no doubt be useful in the next few days as we finish up the documentation, so that we can finally drop the password protection on KosselPro.com.

It's been a long road to get here, and we are looking forward to our partnership with MatterHackers and Solarbotics to bring on additional after sales support for users of our machines.

Thanks,

-=- Terence

Lifting ship hold

Two weeks after returning home from Thanksgiving, Mike and I have performed a 100% Dis-assembly, Re-inspection and Re-assembly of all kits in inventory.  We've also shuffled a lot of work around on our kitters and we took back over a lot of the menial chores with a "Git-r-done" mentality to  just knock stuff out.

The bad news is that for the past two weeks, I have not been able to touch the assembly documentation as much as I like.  KosselPro.com is still under password protection, as we are still working on the site, and I've made little progress on the build instructions.  There is, however, a good discussion going on the OpenBeam forum, and some of the early bird backers have been very helpful in helping with documentation.  Right now, my focus is trying to get kits out the door before I leave for Christmas.  I will be travelling to Malaysia for my cousins' wedding in Kuala Lumpur on Christmas Eve, returning via Singapore and Hong Kong on the week of Jan 5th.  My goal is to get all the missing pictures taken, and if necessary, update the final set of build documentation on the plane and on the ground in Kuala Lumpur.  Instructions and updates to the blog I can do remotely.  Wrapping motors in Geami wrap and messing up Mike's house requires my physical presence.  Mike too will be heading back to the Olympic Peninsula to spend Christmas with his family, so the drop dead date for us getting things out to UPS is Monday, Dec 22nd.

The good news though, is that as of tonight, we have 33 kits packed up.  Tomorrow at lunch we'll run back to Mike's house real quick, load up our 2 cars with printer, and haul them to the UPS store.

The 33 printers going out tomorrow, along with the 15 that we've already shipped, corresponds to roughly 1/3 of our Kickstarter and Preorder liabilities.  We would have built and shipped more, but at this time we are still depend on our kitters returning some of the parts back to us for QC.  We are still doing 100% IQC (Incoming Quality Control), but we are hoping to do a major push on getting parts built and out the door this coming week, during the week.  We've already calculated how many trips it'll take the two of us to run the remaining 90 printers (for both Kickstarter and ShopStarter Preorders) down to the UPS store...

UPS Labeled applied, all ready to go.  This is not including the kits that my wife and I took, for local deliveries and for international test shipment to make sure that all the customs declaration forms are filled out correctly.

UPS Labeled applied, all ready to go.  This is not including the kits that my wife and I took, for local deliveries and for international test shipment to make sure that all the customs declaration forms are filled out correctly.

Next week, we are going to try to get the rest of the kits out.  It will be a tough week, but we have a good shot at getting the majority of the outstanding printers out the door if our kitters can deliver.  We can't guarantee Christmas delivery, but it'd be a relief to get things out the door to our backers.

If you have not done so already, please look for the email from kickstarter@OpenBeamUSA.com and reply with an address confirmation.  We had to skip over quite a few of our Kickstarter backers today and we do not have the storage room to hold onto printers - if you have not confirmed your address, we go down the list and ship the printer to the next person that confirmed their shipping address.  The nice thing is, if you were in the US and  you confirmed your shipping address, you will be receiving your printer this coming week. 

And here's a few more pictures and videos from our kitting adventure this past week:

Mike and I took back over the cutting, stripping and tinning of the end effector fans.  In total, we were about to cut, strip and tin  over 300 fans in less than 3 man-hours of work. These fans go into our end effector kits, which is currently one of the hard gating items at the moment to get more printers out the door.  

Mike and I took back over the cutting, stripping and tinning of the end effector fans.  In total, we were about to cut, strip and tin  over 300 fans in less than 3 man-hours of work.

These fans go into our end effector kits, which is currently one of the hard gating items at the moment to get more printers out the door.  

This fixture is used for cutting the wires to length.  There is only one way to mount the fan onto this fixture block, and a pair of diagonal cutters (or a sharp razor blade) is used to cut the wires. Want to get better at designing fixtures?  Eat your own dog food.  After 90 minutes of using this fixture, trimming about 300 fans, I can think of some improvements for Version 2.  But that'd be another blog entry for another day.

This fixture is used for cutting the wires to length.  There is only one way to mount the fan onto this fixture block, and a pair of diagonal cutters (or a sharp razor blade) is used to cut the wires.

Want to get better at designing fixtures?  Eat your own dog food.  After 90 minutes of using this fixture, trimming about 300 fans, I can think of some improvements for Version 2.  But that'd be another blog entry for another day.

Paladin #1113 Strapax Pro 6 wire stripping tool.  It's one of the most consistent strippers for finer gauge wire. and the orange strip depth control is very repeatable.  Here, I'm stripping 2 pieces of 24 or 26AWG wires at the same time without nicking any of the conductors.  Not cheap, at around $90.00 a pop and about $45 for a replacement set of blades, but very worth it if you do a lot of control wires.

Paladin #1113 Strapax Pro 6 wire stripping tool.  It's one of the most consistent strippers for finer gauge wire. and the orange strip depth control is very repeatable.  Here, I'm stripping 2 pieces of 24 or 26AWG wires at the same time without nicking any of the conductors.  Not cheap, at around $90.00 a pop and about $45 for a replacement set of blades, but very worth it if you do a lot of control wires.

To comply with EU RoHS and WEEE directives, we only use RoHS solders and materials.  However, not all RoHS solders are created equal.  Because some of our kitters have young children, I've chosen an antimony free solder.  This is a 3% silver bearing RoHS solder - more expensive to start for sure, but a bar goes a long way.

To comply with EU RoHS and WEEE directives, we only use RoHS solders and materials.  However, not all RoHS solders are created equal.  Because some of our kitters have young children, I've chosen an antimony free solder.  This is a 3% silver bearing RoHS solder - more expensive to start for sure, but a bar goes a long way.

Mike applying flux and tinning the fan cables.  It's much easier and faster with a solder pot, and it's much easier for the builder to deal with a tinned cable end.

An older video found on my phone - Laser cutting cardboard spacers and air duct for the Kossel Pro.

Mrs. OpenBeam doing the final QC inspection.  She's the one signing off on every kit's packing list.

Mrs. OpenBeam doing the final QC inspection.  She's the one signing off on every kit's packing list.

We look forward to having a shop one day.  Until then, Mike's kitchen floor is the final assembly line for building these printer kits.

We look forward to having a shop one day.  Until then, Mike's kitchen floor is the final assembly line for building these printer kits.

Finally, last but not least - here's the Brainwave Pro test fixture in action.  This is how we are able to speed up the testing and evaluation of our controller boards.  So far, we've tested about 90 of them, and in another evening Mike can easily test and program the rest of the boards to close out our Kickstarter campaign.

That's it for this update!  We'll be going radio silent for a little bit, and we've also won't be updating the public shipping status page for the next week or so, as we are going to be focused on getting kits packed, etc.

-=- Terence

 

Battlefield demotions

Mike and I have received our battle field demotions to grunt line workers this past week, and they will likely stay in effect all the way up to the week leading up to Christmas in our attempt to get all kits, Kickstarter & Preorders, out the door, to our backers.  

This is a list of the inventory of built and Q/Ced kits on storage shelves, and every kit that goes into a Kossel Pro HBP.  Brainwave Pros show zero qty at the moment because they have to be individually programmed and tested, and Oopsie Bags are being revved to ensure that even if we miscounted screws, there will be enough fasteners to complete the printer.  

ZT-KIT-00132 is our most complicated kit; it's the one box that contains all the parts required to build a Kossel Reprap.  Here we are tracking the subcomponents that makes up -132; we're waiting for Kemcor to deliver the soldered auto probe switch to build more auto probe kits and I'll probably have to take over the lubrication of the rails soon.

As of this moment, all mechanical only kits have been prepped, and we've shipped all the ones that have address confirmation.  (If you have not confirmed your address, please do so.  The email should have came from kickstarter@OpenBeamUSA.com).  We will likely push out a batch of printer kits next Monday or Tuesday, and we should be able to get the balance of the kits out the door shortly after that.  

Temporarily taking over Mike's living room.  Everything on the shelves have been through 100% Q/C inspections.  Ball joints, motors, power supply kits, bed clamps and build tools on the shelves.  250mm Glass Plate kits in the big stack by the fireplace.

To be honest, we were disappointed by the progress and accuracy by some of our kitters.  To fix this, we are throwing all the resources we can muster at the problem.  We have brought on additional kitters.  We now check every sub assembly that's coming in, either by weight against a golden sample on a 50mg resolution weight scale, or visual inspection if the weight deviates more than a certain percentage.  We are now personally kitting every single box that goes out the door to prevent kitting mistakes; we work in batches and we weight each kitted box again on the same precision scale after kitting to ensure that one of the packaging elfs didn't take a mental vacation halfway through the process.  We've paid a local contract manufacturer to get some of the cables crimped, and we personally crimped enough cable sets to hold us over until our contract manufacturer can deliver these cables.  

Stack of HBPs on Mike's staircase.  We've assigned this work over a month ago; Mike ended up taking it back and soldering each and every one of these boards in an afternoon.

Truth be told, we got into this mess by being too personal.  We thought we were helping out our friends financially by giving them the (well paying) work, but as the project leader I should have taken a more active role in managing this.  We got burned as they failed to deliver on a timely manner, and we are now doing everything we humanly can to get this fixed and get kits out the door.  

Anyway, I want to share some more pictures of the progress from last week.

My lovely wife Rachel helping with folding boxes.  3 people are about the max # of people that can work comfortably without stepping over each other.

100% QC, 95% reject.  Of all the kits, the ball joints had the worst issues.  We noted almost 95% reject rate due to the wrong screws being used, and on the kits that were about to go out, 2 out of 5 kits had mislabeled ball joint kits.  We already have to ship a replacement kit to New Zealand...

Mike prepping insulation for the thermistors for the HBP.  When you receive your HBP, look closely at the thermistor.  The legs are individually protected with PTFE insulation, before being soldered to the HBP board.

Working as a mini assembly line, it's quite possible to burn through parts quickly and efficiently.  Crimping 35 sets of cables took Mike and I about an hour or so, with us alternating on crimping and stripping.  And this is with relatively primitive, Chinese made tools and no fixtures.  Of course a shop like Kemcor can do this even faster, with automated stripping machines, but 2 man hour for 105 cables is still no slouch.

AC cables being crimped.  Next to it is our automated tape dispenser.  That machine was part of a very expensive "lesson learned" from our original OpenBeam Kickstarter campaign and paid for itself many, many times over during our current predicament.

By default, I am the Geami-elf, responsible for wrapping parts in its protective paper wrap.  

When you pay for bubble wrap, you're paying to transport air.  I wanted a more environmentally friendly way to package our products, and it also worked out well that the Geami wrap is stored and shipped in an un-expanded state, which greatly simplify material handling and storage requirements.  Storage space is always a premium for a startup.  The machine costs $7.00 per month to rent.

Here, Mike is taping and labeling a batch of HBPs.  I have already finished Geami and boxing the HBP with the cable set, and am about to start performing a 100% reinspection of the boxes by weighing them.  At this point we are weighing to see if we forgot a subassembly (such as the cable pack).  If the weight deviates by more than a certain amount, we unseal the box for a visual inspection.

Here, Mike is taping and labeling a batch of HBPs.  I have already finished Geami and boxing the HBP with the cable set, and am about to start performing a 100% reinspection of the boxes by weighing them.  At this point we are weighing to see if we forgot a subassembly (such as the cable pack).  If the weight deviates by more than a certain amount, we unseal the box for a visual inspection.

I've always built my fixtures as if I'm the one using them.  Funny, because I ended up using them quite a bit. :-P.  This is the fixture for cutting the bowden feed tube to length. 

From my days as a FIRST robotics mentor, an engineering team, just like an army, operates on its stomach.  Having good food helps with morale.  On a nostalgic note, I used to make a pot of curry like the one shown here when Mike and I first met in college, over a decade ago.  The only difference is I can afford better cuts of meat now in the curry pot. :-)

From my days as a FIRST robotics mentor, an engineering team, just like an army, operates on its stomach.  Having good food helps with morale.  On a nostalgic note, I used to make a pot of curry like the one shown here when Mike and I first met in college, over a decade ago.  The only difference is I can afford better cuts of meat now in the curry pot. :-)

Having a sense of humor helps too. :-)

Having a sense of humor helps too. :-)

That's it for this update!  Thanks for being patient with us, and we look forward to getting the kits into your hands!

-=- Terence, Rachel and Mike

PS:  Here's a reference to ZT-FUD-00101, for those interested.  All our part numbers are in the form of ZT-XXX-NNNNN, where XXX is a 3 letter prefix denoting the part family, followed by a 5 digit number.

PSS:  We have 5 machines left on this batch that we are building for our push out before Christmas, on our Shopstarter page.

Paying dues as a small business owner, ship holds, longshoremen labor disputes, and the importance of documentation and delegation.

I've always told people that until a child comes along, being a small business owner trying to make it doing boutique manufacturing here in the US has got to be one of the most educational and humbling experiences  I will have.  So let's take a look at some of the reasons why I have not quit my day job, and some of the challenges we've been overcoming in the last few weeks.

Ship Hold.

After multiple delays, we've started shipping Kossel Pros.  Unfortunately, we are not out of the woods.  While we've asked for, and received, and qualified assemblies with incredibly detailed documentation and a rigid first article inspection (FAI) process, we found an unacceptable amount of error in our final kitted printers.  

Now the printers do work well, and the feedback so far from everyone who had been able to get their hands on one, had been very positive.  (Here's a photo stream from one of our early bird backers, and here's another forum member who received printer 001 showing off his prints).  The bad news is that virtually every kit that we've shipped in the past week seemed to have errors, and we've spent many times the dollar amount shipping replacement parts to backers than what these parts are actually worth.  So we made the decision last week to stop shipping kits and perform a 100% inspection on all our inventory, as well as to disassemble all our kitted kits for 100% inspection.

What we found was that some kits were ok.  Some kits, about 10% error.  (For comparison, we expect from our overseas contract manufacturers a 98% yield, on a mature product, and we expect 100% of the failures to be caught by end-of-line testing, so that no bad product makes it into the hands of the consumer.  Higher volume manufacturers should have even higher yield percentages).  On our ball joints though, the failure rate was over 90% mistake.  (Wrong screws packaged with the kits).  Compounding the issue is that the Kossel Pro is a kit of kits; about 13 kits make up the Kossel Pro HBP, and a single error in one of the 13 kits is the difference between a happy customer and a frustrated one.  

Earlier today, myself, my lovely wife Rachel, and Mike spent the entire day recounting screws, plastic parts and bearings:

Frankly, the time for "all hands on deck" to fix the problem is long overdue (it didn't help that it was Thanksgiving week last week and most of us were scattered into the winds due to family obligations) and between Rachel, Mike and I, we managed to form and self-organize a miniature assembly line pretty well.  It was a long day of work, but we managed to get EVERY kit on that shelf inspected.  We also were able to put a significant dent in the kitting process to catch back up.  In the next few days I will be able to rework the ball joint parts and we should be able to lift the ship hold hopefully Wednesday or so.  

For those who are curious:  We did a 100% inspection on an average of 3-5 samples per assembly, then we  used a weight scale to get an average weight across a few assemblies.  We primarily used our 1.5 kg, 50mg resolution counting scale, but we also have at our disposal a 30kg, 1g resolution counting scale, as well as a 50g, 1mg resolution scale.  If the weight deviated by more than 1g, we would set it aside for a more detailed look (often a 100% visual inspection).

We primarily scan for missing plastic parts, missing specialty hardware, and missing hardware such as screws.  We weren't so concerned about fasteners covered by the fastener packs, or M3 nuts, for that matter.  What we'll do is we'll increase the fastener variety in the "Ooopsie Bag" and we'll use that to compensate for a miscount.  (For most of the assemblies, we've already included extra nuts in case the builder loses a fastener or two in building the kit, this would be an extra precaution).  

Finally, in for the next batch of printers that we'll build, we've already found a Chinese supplier that will offer counting and kitting of the fasteners.  Not surprisingly, they are able to offer all the fasteners, counted and kitted, at less than half the cost of us buying raw fasteners here in bulk (20,000+) in the US.  

We will resume shipping of kits this week and we'll give a progress update later this week on shipping progress.

Longshoremen Labor Dispute:

There is an ongoing, ugly labor dispute between the International Longshore and Warehouse Union (ILWU), representing the longshoremen of the west coast ports, and the port operators.  In an effort to "negotiate", the union workers have been organizing labor "slowdowns", sometimes walking off halfway through a shift, etc.  

Now, I am a strong proponent of workers’ rights - paying a living wage, building a strong middle class, etc.  But it should be worth pointing out that Longshoremen are amongst the highest paid blue collar work force in the nation – a lot of these guys make as much, if not more money than the average Silicon Valley software developer.  The labor dispute is unlikely about pay – the last few labor disputes have been the union taking a hard line against deployment of automations and technology (which would make the port more efficient, but potentially taking jobs away from the union – and as a technologist, I have zero respect for anyone opposing the deployment of technology and the efficiency gains that comes with it for the greater good of a community in order to profit personally.  And, incidentally, this is the same union that refused to unload Bertha, the tunnel boring machine for digging the Alaskan viaduct replacement, unless members of their union were given high paying jobs unloading the dirt from said TBM.  )

In the case of our OpenBeam extrusions, our overseas vendor slipped the delivery by a week due to backlog from their summer vacation, and due to congestion, our merchandise sat on the outgoing cargo terminal for an extra week to wait for room on a cargo ship.  (These ships only sail once a week, so if you miss one, the next sailing is a full week away).  Although our merchandise was supposed to dock on Nov 9th, the severe port congestion caused the boat to sail away to offload at Vancouver BC, before returning a week later to unload, and it took an extra week to get our merchandise out of the docks.  (Fortunately, with Expeditors, they can perform customs clearance while the merchandise is out on the water).  So, a shipment with an original projected dock date of "End of October" didn't actually get delivered until two days after I left for Thanksgiving with my in-laws).  

(Above:  The suspension on Mike's Izusu Rodeo, before and after loading the OpenBeam shipment.   It took 3 trips to haul - it takes a lot of material to stay in stock consistently.  Thanks Mikey for moving all the goods while we were out of town! :-) ).

One of the things that we've done right, fortunately, was to build a strategic reserves of our biggest sellers.  We were successful in keeping on hand a backup supply of our signature 1m long extrusions, as well as our Mini Kossel kits.  In fact, we were able to sell a portion of our strategic reserves to the MakerLibre Kossel kick starter campaign ahead of the quoted end-of-October delivery schedule to help them get kits into their backer's hands.  Now that we've received our merchandise, we've trans-shipped these critical kits to Amazon's fulfillment centers already to guard against disruption to our ability to supply the market (ie, running out of stock).  We've also shipped the balance of MakerLibre's order.

Precut kits are a little harder for us, due to the fact that dealers carry this kit and it's much harder for us to make a projection on what our dealers will order.  We've ran out of Precut Kits (we've been out for a few weeks) and with kitting resources focused on the Kossel campaign, we'll try to get the precut kits back onto Amazon as soon as we can.  

Documentation (internal)

We've done a pretty good job with our kitting instruction and documentation, and as such we've already spun up additional kitting resources to help package these kits.  Due to the issues we've been having, Mike and I will perform 100% QC on all assemblies received from our kitters and we will perform the final kitting into the printer kits.  This is the only way we can be sure that the QC issues are behind us (at least for this batch of machines).  

Documentation (Customer Facing)

Before flying out to Chicago, I pulled an all nighter disassembling one of our test printers and photographing most of the steps in detail.  These pictures were uploaded to my OneDrive account, and brave early bird backers have been able to use some of these pictures to make good progress on their machines.  Between family obligations (and on the flights to and from Chicago) I was able to make some updates to the assembly instructions to the Kossel Pro.  

These documents are work in progress (hence the WR in the revision number, for "Working Revision".  We don't like publishing customer facing documentation without proof reading, but we also hate to imagine people staring at boxes without instructions, and we certainly don't want to replace any parts because someone built something wrong.  Over the next few days I'll continue to make stabs at these instructions to get them fleshed out.  (Today's work yielded lots of reject assemblies, which would also help with work instructions and illustrating some of the steps that were skipped.  We will eventually build these parts into the pre-built machines that we have promised for the kickstarter campaign, as well as for engineering testing).

We've opted for PowerPoint as the medium for our assembly instructions.  The thought here is that just about everyone owns a tablet or laptop, and PowerPoint formatted slides display much better on a landscape-oriented screen.  

Delegation

Remember those printers that went out to engineering partner companies?  They've now had some time with these printers and are doing really awesome things with them.   Early last week, MatterHackers returned some suggestions for firmware tuning, and we'll be testing them on our printer and loading this firmware onto outgoing Brainwave Pro boards.  We are also finding that with the HBP, we really do need an external fan aimed at the printer for print cooling.  I will share out our slicer profiles later this week, but in the meantime, enjoy the following pictures. all printed on the Kossel Pro with HBP:

That's it for this update.  We'll give as shorter update later in the week with an update on the shipping list.