Monday, December 28, 2009
Learning Lessons
We designed our own cast metal case, sheet metal top, screw mount to secure the OSAs, connector, circuit board inside with a very unique flex circuit, and surface mount connector…all done from scratch. The RJ parts also feature our own transceiver controller, called Leroy, which does all the magical things to control transceivers over temperature that the big ASICS inside of Finisar and Avago also do. Over temperature, the optical output values move less than 2 dB output power, is almost immovable on Rx Sense, and varies little on Extinction Ratio and OMA. Leroy also gets us Digital Diagnostics per SFF-8472.
We have a competitor that makes a part similar to our RJ but lacks digital diagnostics and a controller like Leroy. With the controller, we can select optical performance metrics based upon speed or link budget. The changes to the transceiver for these operating parameters are in the laser where there two variables:
• Modulation current, which in general, controls the height of the optical eye (OMA) as the light power goes on and off. The modulation is at the same rate (speed) as the data.
• Bias current, in general, controls the average power of the laser or its output power.
All our lasers are packaged with a monitor diode, a small receiver in the laser package. The monitor diode has a ‘slow’ response time – it can’t see the ‘fast’ modulation rate. It can only see the average power. Therefore, it can only help to control the Bias current. To control the modulation current, we use a curve fit or a table depending upon the part. The curve is simple and straightforward but requires a lot of work up front to qualify lasers, boards, assembled parts, and test parameters. We ship what we think will work, but don’t have any feedback to know its working. Any part without Digital Diagnostics operates without feedback, but at least our parts have a curve fit to match shipped and tested parts. For some applications, especially as speed increases or temperature range widens, we employ a table with feedback. This keeps the optical performance in specification, i.e. Ethernet, Fibre Channel, etc, over time and temperature. If a potential failure or failure occurs, the transceiver can let the board it’s on know. The best means of controlling the transceiver and laser in particular, is to use the extra pins we offer to connect via an i2c bus and set the speed or parameters for the specific application.
COTSWORKS transceivers are designed to last a long time and we back that up with support to help meet military, aerospace, or industrial specifications. In the last four years we’ve shipped nearly 50,000 transceivers and seen what happens to Commercial Off The Shelf transceivers in harsh environments. We learned lessons by trial and error…and trial again. While we’ve been in development on the RJs for only one year, there are really four years of experience at work.
Something to remember about learning lessons: the day you make the mistake is rarely the day you learn the lesson from it. With the RJs, we’re trying to learn from mistakes and invention over our whole time in business. Another learned lesson? The shortest path between two points in innovation and invention is not a straight line. And traveling the crooked road takes more time than what’s put in the specific project schedule.
Tuesday, August 25, 2009
In Turn
The gap between older and younger generations on social issues is narrower now than the 1960s or 70s, but the technological gap is big and growing. The older generation needs their old TVs, the younger knows only that “rabbit ears” are just that…cute and fluffy, not bow tied metal. The average kid coming out of high school knows that most of the tech tricks shown on TV is impossible and blows the storyline. Only older adults believe computers can be hotwired with chewing gum and paperclips; or that UNIX is still high tech. I think high school and even college students can be more productive than many experienced professionals today simply because they are comfortable with technology. For example, when COTSWORKS moved to Quickbooks for our accounting software, I had two high school kids handling the data processing. They flew through the conversion while chewing gum, patting their heads, and posting to their social net. Our accountants came and cleaned up things, but the kids can copy, paste, sort, and file in a seemingly blurred one touch motion.
From the start of COTSWORKS in January 2006, we brought in high school students to help manage our network, website, and process orders. Taylor kicked ass. He came in smart and determined and has grown in capability every year, as he still works here. Having students around made things feel like we were always learning and we are a group of people that care less for degrees and laurels and more for what can be done after everyone else has said it couldn’t. After just three years and still a small company, we’ve had nearly a dozen interns work here. Along with the outstanding design win on the Boeing 787; along with developing our own deposition based conformal coating; along with developing our own transceivers that operate in environments beyond what even the biggest companies in the industry can offer; along with the completion of our ISO 9000 certification this fall; the education and participation in learning with our student interns is one of our best accomplishments. At the end of each day, we ship parts; but at the end of an internship, we share in the embodiment of gained knowledge: wisdom and experience.
This year may be our worst financial performance yet as we hit the perfect storm of commercial aviation’s collapse and the loss of a major Industrial products design-win. Nevertheless, we hired four college interns for the summer on top of two high school kids from the spring. While other companies chose to avoid interns, we embraced them. While our production levels dropped, our innovation levels shot up. And if it weren’t for the interns…I’d have never made it through the summer.
I started ranting about TV because I really don’t think that giving coupons to move to digital TVs was a good use of financial resources. But what really got me about TV upgrade and tying to our interns this summer was the interns’ response to the project I tasked them with upon arrival here:
THEM: “What color should they be?”
ME: “What? Orange. Orange doors. They need to be orange.”
THEM: “Clear would be better. Why orange?”
What comes to mind when you say “sliding orange doors?” I hope you think of Star Trek. How ironic that the generational gap in technology exists with the Star Trek TV shows! Throwing my hands in the air after the lack of Bridge decor, I had them all go and watch an old episode on Hulu one of their first afternoons at work. What they must have thought about their summer job: they were paid to watch TV on the Internet. From that request and viewing came something wonderful. From an idea, came a million ideas. From their first try, came many attempts to solve the problem of how to open the doors like the TV show. Never once did they whine about how we could have just bought a kit to do this. And even when they were forced to learn what an RMA was, they kept at it.
http://www.youtube.com/watch?v=Qczo4unvDbU
The technology we ended up using borrows from our own network and optical test equipment including A to D boards, firmware creation, custom PCs, optical sensors, databases, and custom software to control the doors with near realtime feedback. They aren’t perfect, but they work. Yes Solomon…they work. You gave us a motor that could move my entire office around, but the doors open and close. David, the sensors work, the wires are connected, and nothing has sparked or caught fire yet (as of this writing). Who would have thought that your summer, Salil, would force you to “use what you learned in school?” We laughed together as you put into impractical use the calculations learned as a mechanical engineer. And of course we would have a video, Mike. Our ship is made of plywood, pulleys, motors, brakes, along with Warp engines, Phasers but none of it, none of it Mike…runs without software.
We had no huge celebration as the summer wound down and no big lunch fiesta for each intern as might be tradition. Truth is, I couldn’t bring myself to go to lunch as I was too sad the internship journey ended. From my heart, I mean that the best part of celebrating a journey are the memories of every step taken on the way. Hold onto what was, when none of us knew how to make a door open at the touch of a thumb, or read Korean, or knew that a black wire could be a positive, or that electrical tape really isn’t a valid means of insulating wires in air, or that a ventilation hole pattern in a computer case is not supposed to be an artistic expression. COTSWORKS got a lot more from the summer and our interns than just doors to seal our conformal coating room with the touch of a button: we were a part of a group of great people who built something wonderful that will blink, whirr, wind, and buzz for years to remind us ever often of the value of facing the stiff wind and running into it to see how high our kite of enthusiasm can fly. Of course that height is…Space, The Final Frontier.
Mike, Salil, Solomon, David, and Jared and Bobby, and everyone at 749 Miner Road…thank you for making COTSWORKS more than a place to go to work each day. It’s a place to play, invent, create, build, compete, try, fail, win, and have fun.
Even more so next summer.
Monday, August 17, 2009
Tolerance
In the three and half years that we've been in business, we have created a tremendous amount of stuff and value. Making fiber optic links work over an extended period of time and temperature involves research, trial and error, effort, and innovation. Not the innovation of the Telecomm world where cash and paranoia drive miniaturization and speed increases. Rather, the new ways of thinking about things at all levels. I give Luxtera and Infinera and Glimmerglass a lot of credit. Not the same for anyone who makes CWDM or DWDM gear or even transceivers …the business is so much of getting in line, not drawing a line. I'm very happy being Harold with a Purple Crayon and if the only folks who want to be in this book are companies like Protokraft and D-lightsys, so be it.
All of us who make parts for military or aerospace care about drawing that line, the new line to cross. We aren't chasing a predefined concept like PON or 40G, we're chasing more nebulous tasks like making aircraft lighter or creating multimedia shims to 30 year old warplanes. It's a hobby as much as a business sometimes. But that doesn’t keep us from paying attention to details. Human labor is always fraught with the inaccuracies and errors that we make. Building a transceiver with automated assembly lines (once setup) will yield a better part but when the volumes needed each year are only a few dozen to a few hundred, there is no automation. I'd like to think we'll find our way into some larger programs but for now, we are artisan transceiver manufacturers.
In highschool, I did a lot of painting and drawing. This moved to silkscreening and along with some friends, made our own press setup. I quickly learned about offset printing and the need for tight tolerances. To create an image in color requires at least four plates and getting them aligned was difficult. The shirt stretching was one thing, but the machine and frames another. At COTSWORKS, this translates into the tolerances of our transceivers. since i pay the bills I'm keenly aware of those tolerances and the costs to achieve them. 0.002" is stated specification and we achieve it on production parts. One transceiver from us might cost $100 while an equivalent telecomm part we recently bought only cost $18. How is that possible? The parts are designed very differently: we have separate power planes for the Tx and Rx and plenty of copper wherever we can get it in the PCB. We only use high quality parts and make sure everything is sealed. And, we're a little tighter on tolerances. I was struggling to put the low cost parts into a datacomm switch and only got them to work after two other technicians had tried and failed and spent several minutes at it myself. The problem? Tolerances...the parts were only rated to 0.008" but seemed further out than that. They didn’t line up right in the low-cost switch (Dell is a low cost switch damnit). And there it was in front of me, why our parts cost so much more. 30-45 minutes in a lab is nothing compared to several seconds in on a battlefield. One of our customers has a fixed time that they have to be able to bring up a redundant set of switches. Could they save $80 a part? Sure. And they would fail in the field.
I would like to have more time to work on this blog. it's not that I don't think about it. Priscilla Diem and I have talked about those tolerances and the ones that she encountered workign at NASA on the shuttle. Not only did I have to make sure we delivered parts to those tolerances which has kept me very busy, but I had to find a way to shut off the side of my brain that is capable of dealing with this kinds of things and switch to the side that lets me sit and type our a post. I had no measure for success or accuracy with this writing, just to keep going until i felt I'd made a point.
0.008, 0.002, or 3714 letters. the latter works tonight.
Ken
Monday, June 15, 2009
The Good Humor Man
We have been visited by the Good Humor man of optics and the last two months have been pretty hard for COTSWORKS. I relate this story because “distribution” and “manufacturing” are not what they used to be. The modernization of China and Taiwan have created a new threshold for commodity manufacturing. Ask the American worker what they think of China and you’ll get a string of complaints. Ask the same worker about how the material things they have in their house compared to their parents, and you’ll get a list of plastic and drywalled items that we take for granted but were the pervue (in finer form) of the rich and gentile of yesterday. Ask many small business owners, and you will find that many are basing their businesses on going to China to get parts made then marking them up substantially and calling it a business with value.
Of course, access to the customer is what matters, but where is the value here? And, how defensible is this as a business? How long will it be before the Chinese manufacturers call our customers, ourselves. We get besieged now by vendors trying to sell us parts and they are ruthless salespeople. In Europe it seems, they have already made many inroads.
What caused us problems, is that you cant put much faith in a brand anymore, say, Good Humor. We bought lasers from three of the most respectable sources only to find that the quality varied across the lot of parts we bought. And when we went back to them and showed them the broken welds, extra solder, non-hermetic seals, we didn’t get a full corrective action report. We got, “we will talk to manufacturing.” Which means that an email was sent and maybe a Skype call at night.
For Military, Aerospace, and Industrial optical transceivers, this just doesn’t work. There is no way that one person’s Industrial Temp. part is not another's…it either works at -40 to 85C or it doesn’t. Summer or winter, it’s always snowing at COTSWORKS,

I stopped wondering about why military parts cost so much when I read a story about Hyman Rickover and the US nuclear submarine fleet and how someone substituted lower cost portable toilets on the subs. Seems that the lower cost ones cracked when used up in the Arctic and sailors could sit on them and fall over…onto the ice or water. The higher quality and testing doesn’t manifest itself in every use but rather once in 100 or 1000 or 10,000. And that once can mean drastic consequences. But more importantly, it’s that there is no other toilet. With Datacomm…you pop the transceiver out and put a new one in.
Sadly, I’ve also learned why things cost so much in the military and aerospace world…and that’s time to market. We have two big programs in the works that will take five years or more to see production. The last few months have been really tough with the airline market in general collapsing. But that which does not kill you…in this case, makes you poorer. I hope things turn around soon; I hope the Dreamliner finally becomes the Realityliner.
Monday, March 16, 2009
Two Feet
Building transceivers for harsh environments is hard stuff. I’ve worked at Apple Computer, an embedded networking company who sold to Cisco, and created a nationwide optical network VAR. But building a transceiver is the hardest thing yet. Why? Because if something fails you don’t know who to blame: optical, electrical, or mechanical. You might guess the answer to the “how many engineers does it take to build a lightbulb” is three but you would be wrong…the answer is none. It takes a technician to debug it. We’re not designing SFPs for commercial temperature data center operation. we build stuff that operates up to 95C and stores up to 100 or more (sorry to switch from F to C in one article, but it’s not like this is a satellite program manual). And our link budgets are bigger and our parts have to survive being banged about. This is where the rubber separates from the road in our case.
Saturday morning, March 14, 2009. oops. 14th of March, 2009. I am just getting out of my hotel by London Heathrow Airport and going for a walk. Across the road is an office part which runs along one of the runways at Heathrow’s International terminal 4 sans British Airways which has its own terminal as I accidentally found out. I cross the road, cut thru the office park, and am at Flight Global/360 center elevation. There are dozens of people watching airplanes take off. No joke. It’s cold too. On my arrival to Heathrow, I saw the monster on the tarmac, but in what we call the “western wind” of COTSWORKS, it is my fortune to have just made it out of bed and crossed to the viewing area exactly as Singapore Airlines’ A380 is taking off for the day. It lands early morning, takes off late morning. 747s are parked here like Chevy’s at a dealership last month. My god there are lot of them. Sandwiched in between, however, is the great white whale (with a blue stripe and Singapore written on it). The A380 isn’t actually that big with regards to its length. The 747s seem longer, but that upper deck is deceptive. The tail is what really gets you. It’s just huge. Really, you have to see it to believe it. I’ve been next to the tail of the 747 at the Boeing museum, but this tail is enormous. What’s so nuts about it, of course, is that all it does is hold course (it has no storage in it). Just when I was settling in on the size, the plane took off. It doesn’t move fast and it takes a long time to get airborne. The view from, as the mechanical guys tell me, isometric, is shocking. There is so much metal and surface area where the wing joins the body you have to laugh a bit. That’s the critical point on a plane and this one is critical mass. It finally takes off and unlike the 737s, A330s, and 767s that flew first, it hangs in the air. It’s so big that your eyes see it all the way up into the clouds. You can actually see it go into the clouds even though its far away. Amazing.
There are four engines on that monster, like the 747, A340, and the old 707s. Four engines means more lift and in turn, more gas. So you have to carry a lot of people to make four engines worthwhile. That’s why the 777 remains the best plane in the skies today in my book. Two giant engines, a perfect tube, and delivered on time. I hope the arc of the 787 is beautiful enough that it makes up for the two year delay. Waiting has been painful. But neither Rome, nor the Internet was built in a day. And watching the A380 take off reminds me of why I do what I do. Why I get out of bed to work so hard; why I travel so much; why I have risked just about everything to get COTSWORKS going. I was in telecom on the end user side and the equipment side for ten years. I called on companies like Digital Domain who pioneered special effects and had renderfarms the size of a Venice, CA block. I sold optical networking motherboards to Cisco and tested and debugged fiber optic networks on Virginia Class subs. I drawled with the best of ‘em when I said the word “gigabit” and counted RJs by the ten, then hundred, then by stacked racks. But I know a bit about the network in the A380. and a lot more about the one in the 787. and I cried as it took off in front of me on that beautiful spring morning. Because the data centers I help build now come with two to three hundred thousand pounds of thrust attached to them and no IT manager who reads Network World can say that. That data center might not be the fastest network pipe, but it is for sure the fastest moving network pipe.
I miss my kids, my wife, my friends, and the people I work with at COTSWORKS when I’m away. The travel, especially an entire week, is hard but I do make new friends who also, commonly in this business, happen to be customers and salespeople for our products as well. I get to see amazing cities like Amsterdam, Munich, and London. Sometimes I’m lucky enough to go to Cedar Rapids, IA or Grand Rapids, MI too. But in the end, the work and effort are all proving to be worth it. I tell my kids that my job is to build lasers for airplanes and how cool is that? I dream about being a part of a worldwide team of professional engineers that make products which help make our world safer, smaller, and available for everyone. Getting stuck in London for one extra day gave me the time to watch a movie, Catch Me If You Can. It’s a great movie and happens to include scenes on how the main character impersonates a Pan Am pilot. Travel back in the ‘60s was so glamorous and exciting. It isn’t at all anymore. Singapore tries hard to make it so with their huge seats, HDTV, and the insane installation of a shower in one other airline’s planes. I don’t think that’s dreaming, I think it’s nuts.
Get it there safely. Make it cost effective. Do things like improving the air and services on board. And please, lighten the load so it uses less gas and costs less. One shower eliminates a mile of weight savings from copper to fiber wiring. These are the thoughts I have about that A380, 787, and others. Two feet. One in optics, one in aerospace. Head in the clouds…but both feet firmly planted on the ground.
Congratulations Airbus.
Last One Off The Ship
Holy Crap. It’s a good phrase to describe what’s happened to optical company stocks. What was once a first round funding is now the market cap of some of the biggest names in the business. Why are they trading so low?
At the OFC conference last year I accidentally had lunch with a Hedge Fund analyst. Yep, I missed my chance to poison her drink then, but now I understand the words she said to me. Wall Street couldn’t figure out how optical companies were going to make money. They were right about this and they bailed. I can’t figure out how they are going to make money and I’m in the business. If I were running one of them, I’d be looking for work or a new business model.
It’s the business model that is the problem. The idea back when we loved Finisar, E20, Stratos, OCP, etc was that with enough cash they could outlast their competitors and play in God’s greatest market since
The transceiver business is busted. As one expert transceiver salesperson put it: something has to change more than just me losing my job. The network equipment market isn’t too much better. The whole game in Internet connectivity was based on selling Ports. How many ports was the mantra of Ethernet strategic analysis experts. So everyone just gave stuff away to get as many ports out there as they could. Too bad they forgot that with no follow on business past giving away the razor, they couldn’t make money on razor blades that were all based on 802.3 and MSA standards. The only one laughing is Cisco, who buys parts for $20 and sells them for several hundred $s. and worse as they get to longer distance or higher speed parts. As long as the MSA exists, nobody is going to make money making transceivers.
This is direct contrast to the insanity of the
The
I guess that in the end of this downturn we finally see the companies that need to act, act. The OFC show returns next month for anyone who wants to go to what will likely be a wake. Their tagline is “where innovation comes alive” which is hilarious considering it really should be “where desperation keeps a few of us alive.” But I’d bet that the tag line was left over from some years ago and OFC cut staff including marketing and support. At the rate we’re all going we’re going to have to decide to go left or right, but not continue stumbling straight ahead.
Monday, January 19, 2009
The UN and Transceivers
Change is coming to this country with the Election of Barak Obama. Today I saw a headline calling it the end of the Baby Boomer Era. Minority President, Democrat, Centrist, and Diplomat with a new tack on
While our economy is stalling and worth his primary focus, he will have to look not only inward but outward. Our world is mostly connected (Madagascar still lacks an undersea fiber optic link so he can not call on a Monroe Doctrine as hemispheres matter much less these days. Central to a foreign policy is how he uses the United Nations. The UN is a problem because it is a broken system. There are no checks and balances to protect minority interests. There are no popular elections of judges in the
The UN reminds me (in a non-human rights way) of the Multi-Source Agreement standards committees that govern how transceivers are built. Representatives from big companies who make parts get together and decide what an industry standard should be as long as they can benefit from it. Then they run back to their respective companies to try and get parts made the fastest and cheapest way possible and dominate a market segment. No one enforces the MSA or provides an industry neutral testing certificate. Just as the UN has no way to enforce its decrees, neither does that MSA committee.
The major transceiver vendors in the MSA, Finisar, JDSU, Avago, Source Photonics (MRV), and Emcore (the G8 or Security Council) among others keep plowing money into projects that don't yield any positive return. It’s mind boggling that billions of dollars can go into building what engineers ten years ago would never believed possible and is now sold at below 30% gross profit often leaving a negative or minimal net profit. Even with complete vertical integration of manufacturing, these companies probably make more on the sheet metal than they do once they put a laser inside. How did this mess happen? Why can't these companies reach a level of integration where they are profitable?
I got in a discussion with one of the researchers at Lightcounting about Innovation. I don't think that innovation is simply faster and smaller. When no one makes money and the only barrier to entry is when Chinese OEMs upgrade their design sets, you have a broken system that stifles innovation. Is it innovation to go from 100Base to 40G in 10 years? To the network equipment vendors, yes. To the optical and electrical scientists who shrink the components and solve dispersion, impedance, or BER issues, yes. To anyone making transceiver? I'm not sure.
10Gb/s transceiver shipments are growing at a whopping 92% CAGR and will probably continue for the next few years. Having created my own YouTube channel...I hope to contribute to even faster growth. The MSA, however, has imprisoned companies in rat race that leads to fewer companies in the short term and no clear path to profitability. The Chinese transceiver companies are now making 10G parts. The so called Tier 1 transceiver manufacturers wear the Blue Helmets and get shot at without ever seeing a return. Why shouldn’t Finisar have 55+% Gross Profit like its customers? Looking at this picture from a distance, it may turn out that public and private investors are spending several trillion dollars in Research and Development (operational losses) over some twenty years to enable optical networks where the end users benefit but the manufacturers never see a return. They take a 1/2 step to the wall each time. Ultimately, the MSA and the transceiver companies have to see a new direction soon or they will hit a bottom where 2-3 silver haired engineers at a committee meeting decide what the Chinese OEMs will make next year at the request of network OEMs who have booked profitable service contracts with end users.
When will this end? Somewhere down the road a transceiver will be built into part of a network chip or PHY just as it is on copper. The best analogy I can see is to look at the copper network drivers. And there, the market shrank to just a few, Broadcom, Marvell, and Intel for the bulk of the market with a few vendors doing high speed or alternate protocol (10, 40G and Fibre Channel). Wouldn't it be cool to look inside a PC and see a chip with duplex LC connectors sticking out of it? But whose name will be silkscreened on the chip is the trillion dollar question.The bottom, then, is when circuit boards have optical interconnects in them. USB 3.0 will help. VITA 42 will help. But somehow, the O to E has to be an O to O in order for transceivers to become chips on boards. I can’t tell you when that will be, but I don’t see it too soon. Less a technological problem, this barrier is one of the biggest problems in aerospace: the connector. You can not solder fiber.
The MSA has democratized transceivers and the associated standards which is good, but since they only seem to care about the current standard, say 40G, the old 1G standards get old and stale. The image below shows a readout of a Finisar SFP using our Capstone TM software and OptoTrek test board .
Just about every SFP made today surpasses the MSA spec’d numbers. The readout to the left is highly formatted so it hides the old ratings for output power or Rx sens but they are included in default text in the Serial IDs of the modules. The Vendor OUI listed isn’t dynamic or...regulated. Why couldn't transceivers connect to an online specification server to get updates and check their accuracy? The Vendor Programmable section is used by companies like Cisco to lock out non-Cisco parts in their switches without providing any real value. Any engineer with i2C knowledge can reprogram them and put their own company in (a Trademark violation) and go to market as an OEM. Supposedly, the FDA is watching how lasers are shipped in this country but where would they start to enforce regulations? Getting a part on the open market through a Google Search yields hundreds of companies that sell SFPs. Many of them are Chinese OEMs where they have an equal number of spelling errors as products…how can you trace it? If Fiber To The Desk ever arrives with 10G LAN servers, we may see local shops and nationwide chains of transceiver exchange or repair.
In any system, the strength and reliability of that system is its weakest link. With the price pressures that exist today in optical transceivers, the weakest link is not the single SFP out of 100 deployed that might fail, but the whole business of profitably manufacturing pluggables.