One Starship a week

Three barrels welded together are lowered onto a pressure dome for SN2 at the South Texas Launch Site this week.
Source: Ars Technica

I have selected some excerpts from a wide-ranging and very informative article by Eric Berger from Ars Technica about Starship. 

How badly does Elon Musk want to get to Mars? Let me tell you a story. On Sunday, February 23, Musk called an all-hands meeting at the South Texas site where SpaceX is building his Starship spacecraft.

It was 1am.

He wanted to know why the Starship factory wasn’t humming at all hours. Why steel sheets weren’t getting welded into domes and fuel tanks, why tanks were not being stacked into rockets, why things weren’t going as fast as he wanted.

Musk always wants to go fast. He will not live forever, and the money may eventually run dry. He knows this. One day, the window to spread humanity to Mars may close, but Musk doesn’t know when. So he needs to squeeze through before the window shuts.

To really accelerate, his bleary-eyed engineers and technicians responded, they needed enough employees to assign workers to particular stations within the burgeoning factory, allowing each person to specialize. This would require a lot more hands that could build things.

“I said, ‘OK no problem,’” Musk recalls. “I said, ‘You can hire people—just know your reputation is on the line. Don’t bring your brother-in-law who can’t ever get a job. Not that person, OK? You’re going to be responsible for them. Everyone’s got their relatives that they know at the family gathering who, man, I sure as hell wouldn’t want to work with that person. Don’t bring that person. Bring the person who you’d put your reputation on the line for.’”

SpaceX had held a much publicized “career day” in early February, and the company hired several dozen new employees. By contrast, this Starship factory’s scale-up would be all word of mouth. And it would happen immediately.

Musk told his team members they would have a recruitment gathering just 12 hours later, at 1pm that Sunday. They would have another one on Monday at 1pm and then again at 8pm. Long lines of people showed up, family members and friends, mostly local. Cars and trucks jammed the roadside up and down Boca Chica Highway. At 11pm on Monday night, SpaceX was still hiring.

All told, the company added 252 people to its South Texas Launch Site on that Sunday and Monday. It doubled the workforce, just like that, to more than 500 workers. Most of the new hires, even those who had inked contracts at midnight, were told to report for work the next morning. A year ago, perhaps a dozen or so people worked on site. Soon, the Texas factory will probably be SpaceX’s largest location outside of its headquarters in Hawthorne, California.

[....]

SN1 was never destined to fly, anyway. The plan for this vehicle, had it survived the pressurization test, was to install a Raptor engine and perform a static test firing. If everything looked just right on this test, Musk might have greenlighted a test with three Raptor engines. But probably not. The attitude of engineers working on the program boiled down to this: it sucks to lose SN1, but the next vehicle in line is already outpacing it. SN2 will soon be ready for tank testing.

Not that Musk felt particularly happy about losing a Starship. On Saturday and Sunday, he huddled with his engineers inside the University of Texas’ Stargate building, at the periphery of the company’s South Texas footprint. A year ago, the few rooms SpaceX leased on the building’s second floor were the only facilities on site aside from a few construction trailers. Now it is merely the front door. SpaceX has taken over the entire building, turning it into a mix of offices and storage.

[.....]

“Well, I just had a lot of talks with the team about that today,” he said of the SN1 failure. “It’s what you might call the thrust puck—there’s an inverted cone where we mount the three sea-level engines. In fact, it’s drawn on that whiteboard over there.”

He walked up to the whiteboard and pointed to a frowning face. “This is my drawing,” he said with a smirk. Then, with a dry-erase marker in hand, Musk proceeded to lecture about rockets.

“There’s a sad face because we have an inverted cone,” he said. “It’s such a dumb design. It’s one of the dumbest things on the whole rocket because it’s heavy, expensive, and unreliable.”

Basically, the SN1 failure boiled down to bad welds in a weak section of Starship near the engine. When exposed to pressure, the welds burst.

Musk was not happy because he had not heard about this specific issue, in this section of Starship, before the test failure. Do you think Musk addressed that with his team? Yeah, he addressed that.

“We sent out a note to the team that this was badly designed, badly built, and badly checked,” he said. “That’s just a statement of fact. I met with the whole quality team, and I said, ‘Did you think that that thing was good?’ They said, ‘No.’ I told them that, in the future, you treat that rocket like it’s your baby, and you do not send it to the test site unless you think your baby’s going to be OK. They said that they did raise the concern to one of the engineers. But that engineer didn’t do anything. ‘OK,’ I said, ‘then you need to email me directly.’ Now they understand. If you email me directly, and if I buy off on the risk, then it’s OK. What’s not OK is they think that the weld is not good, they don’t tell me, they take it to the pad and blow it up. Now I have been clear. There’s plenty of forgiveness if you pass me the buck. There is no forgiveness if you don’t.”

What you need to understand about Musk is that he is the chief engineer of SpaceX—and that’s not a courtesy title. Musk previously told me that at the very beginning of SpaceX, no great engineers would take the job, and what’s the point of hiring someone to be chief engineer who isn’t great? So he became the chief engineer of SpaceX. Almost every technical rocket decision made at SpaceX comes to him eventually. Especially the hard ones. He has spent many, if not the majority, of his days since December in South Texas. During Christmas, employees there say, he worked all-nighters alongside them to get the dome structure and the welds right for SN1.

Yet Musk has not been spending so much of his time in South Texas just to build a Starship. Rather, he’s trying to build a production line for Starships. He wants to build a lot of them. And fast, always fast.

“Production is at least 1,000 percent harder than making one of something,” he said. “At least 1,000 percent harder.”

Musk should know. He lived through “production hell” at Tesla in 2017 and 2018, building up factories, changing processes, spending many sleepless nights and going through all manner of mental agony. Now, Tesla is making as many as 10,000 cars a week.

He wants to implement a similar system in South Texas. Musk, in fact, aims to reach a point where the company builds a Starship a week by the end of this year. And after that? Maybe they’ll go faster. SpaceX is designing its factory here to build a Starship every 72 hours [i.e, 2.3 Starships a week].

[....]

Musk wants a linear flow through the tents, whereby rocket parts come into one end of the factory and move from station to station until large chunks of Starship end up in a high bay for stacking into a vehicle.

The process of building a Starship on site begins with large “barrels.” Each of these is about two meters tall and nine meters in diameter. To make a barrel, a length of 301 full hard stainless steel is unspooled, cut once, and welded along this seam into a cylinder. In an unpressurized state the barrels are cumbersome, as each weighs nearly 1,600kg. To construct the outer skin of Starship, 17 barrels are stacked and welded together, with a nose cone on top. As of Saturday, since manufacturing operations began in Boca Chica about 11 months ago, the company has built 50 barrels. But the process is accelerating. The company can now make two barrels a day, and it aims to reach a production cadence of four barrels a day [To get to 2 Starships a week, it will need five barrels a day].

The first of the big new tents, erected in January, is filled with tank domes for the SN2 and SN3 vehicles. Pressurized domes are difficult because they cap off the ends of fuel tanks and must hold chilled rocket propellant—in Starship’s case, methane—and liquid oxygen at high pressures. So they must be made with care. The current process for building a pressure dome takes about a week; 1 or 2 days to tack up and fit steel sheets, 4 days to weld the sheets together, and 1 to 2 days for X-ray inspections and repairs. But in Muskville, taking a week to make a dome is way too slow.

So Musk has challenged his team to find ways to go faster, to cut production time, and to improve weld qualities. He has let his engineers engineer. And they think they’re close to a solution for dome welding with a tool called a “knuckle seamer.” They built a prototype in the second tent.

The knuckle seamer looks something like a giant zipper that articulates over the front and back of a dome, like a taco shell around its filling. On the front, the seam to be welded is designed to line up along the zipper, which clamps down for a precise fit. Then, in about 10 minutes, an automated torch will trace down the length of the curve, providing a precise weld. Following this, the dome is rotated to bring the next seam into view.

Other engineers have built an in-house, shielded machine to X-ray the welds. Presently, an X-ray team shows up, quarantines a work area, the premises are vacated, and then radiation tests look for imperfections. With this new X-ray machine, SpaceX hopes to compress a process that can take a day down to a few hours.

This is all happening so fast. These machines existed only in the minds of engineers four weeks ago. The tent they’ve installed them in for testing was built less than three weeks ago. Musk has always had a knack for hiring brilliant young engineers, and those in the Boca Chica tents were mostly in their 20s, busting their tails for the boss. Willingly. Why? Because Musk empowers them to go fast, do cool things, and, very soon, to see their machines fly.

Why the hell does Elon Musk need to build so many Starships, anyway?

Because he’s actually serious about settling Mars. It’s not a joke. It’s not a con for more government money (although Musk won’t turn that down). No, Mars is the raison d’être for SpaceX. And now, in South Texas, Musk is getting close enough to Mars that he can almost taste its red dirt.

Let’s just step back for a moment to acknowledge how nuts this is. Starship is only the upper stage for SpaceX’s Super Heavy rocket, but it is arguably the most novel spacecraft ever built. No one has ever built a fully reusable rocket[on Falcon 9, only the first stage is re-usable], and the second stage that goes into space is the hardest part. SpaceX remains a long way from making the interior of Starship habitable for humans on a journey to Mars. But even building a fully reusable vehicle that can lift 150 tons into low Earth orbit would be a marvel. That’s more throw capacity than the Apollo Program’s Saturn V rocket had.

And Musk wants to build one of these each week.

Compare that to NASA and its Space Launch System, the big rocket that the space agency has been developing for a decade and for which Boeing only recently completed a single core stage. This core stage is about 15 meters taller than Starship but lacks its complexity. NASA will, in fact, toss each SLS core stage into the ocean after a single use. And Boeing doesn’t have to make the engines, as the rocket uses 40-year-old space shuttle main engines. Despite this, and with nearly $2 billion in annual funding from NASA, Boeing’s stretch goal for building core stages is one to two per year... some time in the mid-2020s.

SpaceX’s stretch goal is to build one to two Starships a week, this year, and to pare back construction costs to as low as $5 million each.

[.....]

SpaceX engineers and technicians in Boca Chica needed eight months, from last April through November, to build the first Starship prototype, MK1. (It blew its top, too.) But the workers in South Texas only needed a month, from late January to late February, to assemble SN1. And SN2 is following only about two weeks behind SN1.

“The problem with the MK1 stuff was that I didn’t have my eye fully on the ball, because I was still taking care of a lot of Tesla stuff,” Musk said. “Now Tesla, I think, is in a good situation here, so that’s why I’m pretty much camped out in Boca. The MK1 was a failure not because the rocket failed at low pressure, but because we failed to build a production line.”

Now he has built the production system. It’s not finished. It’s not perfect. But it’s starting to pay dividends. The factory is beginning to flow linearly, from one station to another. And after last week’s flurry of hiring, the company has the workers it needs to produce rockets one after another.

“If you’re just trying to make one of something, it can all basically just be made by the engineering team,” he said. “But if you want to actually make something at reasonable volume, you have to build the machine that makes the machine, which mathematically is going to be vastly more complicated than the machine itself. The thing that makes the machine is not going to be simpler than the machine. It’s going to be much more complicated by a lot. Things need to be translated into instructions that the average person can understand. You can’t have somebody with an engineering master’s degree from MIT hand-making every single part. It’s not possible. There just aren’t enough. MIT’s not graduating enough people.”

[.....]

I’ve spoken with plenty of the earliest engineers who worked at SpaceX, and almost all of them have noted that Musk tackles the hardest engineering problems first. For Mars, there will be so many logistical things to make it all work, from power on the surface to scratching out a living to adapting to its extreme climate. But Musk believes that the initial, hardest step is building a reusable, orbital Starship to get people and tons of stuff to Mars. So he is focused on that.

He knows he won’t get Starship right at first. He employs some of the smartest engineers on this planet, and they’re still, in many ways, fumbling toward solutions for the extremely hard problem of getting a super-large vehicle out of Earth’s gravity well into orbit—then to land it and fly it again. Musk has come to believe the only way to realistically achieve this is through trial and error, by iterating closer and closer to the right design.

The hardest part of any rocket is the engine. And SpaceX is almost there with the Raptor engine iterations. “We’ve done this with Raptor,” he said. “Like, we’re on Raptor engine 23 or something, Maybe 24. It’s lighter, cheaper, better in almost every way than Raptor version one, which sucked and blew up, basically. One of about six or seven Raptors that blew up, I’ve lost count.”

The point is, Raptor is now in a good place. The next step involves building a spaceship with the tanks and plumbing to harness the power of six Raptor engines. The first two Starships have blown up during pressurization tests, but with each new iteration of Starship, Musk and his band of engineers are learning.

“A high production rate solves many ills,” he said. “If you have a high production rate, you have a high iteration rate. For pretty much any technology whatsoever, the progress is a function of how many iterations do you have, and how much progress do you make between each iteration. If you have a high production rate then you have many iterations. You can make progress from one to the next.”

[.....]

Even so, maybe you think Elon Musk is going to fail in his Mars ambitions. Any reasonable person might. This kind of thing makes the Apollo Program look like child’s play, and the Moon Landing is regarded as perhaps the most significant technical achievement of the 20th century. But should we really be working on a repeat of Apollo half a century after we already did it? Maybe we should reach higher and further.

Walking through those tents in South Texas, amid the bustle of those workstations, surrounded by rolls of stainless steel, it becomes easier to believe that we should and that we can. The place feels the way a US Navy shipyard must have felt in the weeks after Pearl Harbor—insanely busy but also purposeful.

These kids and swarms of recently hired technicians are fighting against impossible odds every day, and they’re determined to win. Don’t tell them it can’t be done. They’re not having any of that in Muskville.

[Read the full article here]