Status OpenPhoneux / GTA04

[Dr. H. Nikolaus Schaller]

Hi,
what has happened since the last report? (see below)

Am 03.07.2012 um 11:15 schrieb Dr. H. Nikolaus Schaller:

> Hi all,
> I did a long telephone call this morning to discuss the production situation of
> the GTA04A4 boards.
> 
> Am 25.06.2012 um 20:56 schrieb Dr. H. Nikolaus Schaller:
> 
>> Hi,
>> 
>> Am 25.06.2012 um 16:16 schrieb Gilles Filippini:
>> 
>>> Hi Nikolaus,
>>> 
>>> Dr. H. Nikolaus Schaller a écrit , Le 31/05/2012 17:36:
>>>> So please expect that it will become End of June for delivery...
>>> 
>>> Since we are near to End of June, time for asking how is it going? ;)
>> 
>> Neither good nor bad...
>> 
>> This means:
>> * we have already shipped some of the GTA04 Group Tour boards
>> * some friends have upgraded to a complete OpenPhoenux (we have a handful reworked GTA02)
>> * but we still have a list of open orders for roughly 90 GTA04 Group Tour boards
>> 
>> The production company had stopped again to do some more
>> analyses and the last status I received today is that they had
>> again made some X-Ray. Now they think they can again optimize
>> the soldering process a little so that we don't get so much broken
>> boards that need manual rework.
> 
> So here is a copy of the latest X-Ray image of the DM3730CBP with short circuits:
> 
> 	http://download.goldelico.com/gta04/images/20120622-GTA04A4-schlecht-01.jpg
> 
> If you want to compare to the GTA04A3 board X-Ray (with one short):
> 
> 	http://download.goldelico.com/gta04/images/20110707-GTA04A3-schlecht-xr002.jpg
> 
> or the GTA04A2 board (where only 1 unit exists):
> 
> 	http://download.goldelico.com/gta04/images/20101109-GTA04A2-overview.jpg
> 	
> 
>> Every rework ruins some components (unfortunately 1 DM3730
>> + 1 Memory chip > 50 EUR) and costs intensive working time.
>> And we simply can't afford that if it happens too often...
>> 
>> So the highest priority is to get rid of the root cause for these
>> soldering failures.
> 
> After analyzing and discussing the X-Ray images we now think we
> understand what happens (but you never know if you are right...).
> 
> The reason appears that the DM3730CBP gets some warp during
> the soldering process squeezing the balls in the middle of the BGA
> grid so that they may touch a neighbor and make a short circuit.
> And on the corners the balls lift off and this may result in interruptions.
> 
> This theory correlates exactly with the error patterns we see:
> 
> * some 50% of the boards come out of the soldering process with
> shorts on VDD1, VDD2 and/or VIO which are mostly available
> on the center balls
> 
> * if the chip has no short on the power supply or is reworked some
> 50% of the boards have interruptions which makes a single function
> fail. Sometimes the SD interface has just 3 data lines, sometimes the
> WLAN chip. Sometimes some sensor is not working. Sometimes
> the USB is not working, or one display data link distorting the
> colors etc.

In the meantime we got serveral very good hints that prove this is the
right theory:

1. OpenPandora had no problems with the OMAP3530 but also has
similar issues (shorts on the power lines) with the DM3730CBP

2. thanks to one of the guys here on this list we could get our
own X-Ray session at the demo center of a company that produces
such X-Ray devices for electronics production.

Using this photo session (many pictures [1]...), we could see:
a) all variants of the PCB (GTA04A2, GTA04A3, GTA04A4) have the
effect that the center balls are thicker than those on the edges which
risks they melt together forming a short circuit under the chip

b) the effect is stronger with the last production run

c) generally everything is ok except the DM3730CBP, even other
chips with the same 0.4mm BGA pitch

d) the GTA04A2 was populated with a OMAP3530CBB - which is thicker
and has a larger silicon die. This must have less warpage than the
DM3730. This may also be the reason why the OpenPandora was
easily produced while the OpenPandora 1GHz has comparable
problems.

3. we were notified of this Texas Instruments document which addresses
chip warpage (last slide):

> http://www.ti.com/general/docs/lit/getliterature.tsp?baseLiteratureNumber=swpa182 

Although it is for the OMAP4 and has its focus on the PoP memory
(where we don't have problems with) it carries a lot of hints for our
problem:

a) there is warpage risk
b) TI knows about it
c) warpage may squeeze the center balls so that they have shorts (bingo!)
d) there are tricks to reduce the risk (making the stencil opening
smaller for the center pads)

> 
> Now comes the crucial question: how can this be improved?
> 
> The specialists of the production company have spent whole weekends
> to do more experiments but did not yet find the right trick. It is not that
> they do not want to solve the problem (they see it as a challenge) they
> simply have not yet found the key.
> 
> One factor is that our GTA04 board has exceptionally squeezed
> components (since we must be small) so the soldering of a OpenPandora
> or the BeagleBoard may be a little easier from heat distribution.
> 
> I.e. we are pushing an experienced EMS to their limits...
> An EMS who has produced more than 2000 different products in small
> quantities (which is their strength) including some 0.4mm BGAs.
> 
> A new experiment is scheduled for week 29 to use a different
> reflow soldering machine and process which uses a little lower
> temperature.

Now we have week 29 and today we had a 3 hours workshop to go
through all those new findings to decide what to do.

The main finding was that production quality had decreased in
the last run significantly, compared to the A3 boards and the
first batch from January/February 2012. The main reason for
this effect turned out to be different. They had tried to optimize
the reflow profile - but the effect (visible on the new X-Ray
images) did make it worse.

This can be "solved" by going back to the old reflow profile
(which is partially out of the specs of the components) but appears
to be necessary to avoid that many open connections we see.

The other issue is the apparent warping of the DM3730CBP.

This can be addressed by changing the stencil and/or the flux.
E.g. making the center holes smaller (as recommended by the
OMAP4 experiments). And the chips can be dipped automatically
into the flux (currently this is done manually). This may require
that their machinery needs an extension.

> Unfortunately, we can't contribute to this production improvement
> to speed things up.
> 
> Unless one of the readers on these list knows some real specialist
> for such PoP soldering of BGAs who is willing and capable to
> consult (even if it costs money). So if you know someone, please
> let me know.

Thanks again for all those helpful hints we received.

>> 
>> The good answer is that as soon as we decide to continue,
> 
> So we had to decide to wait for the different reflow soldering process
> first.

We decided today to discuss the smaller stencil opening with the
stencil maker (if it is at all possible) and to check if the chips can
be dipped into flux automatically.

This again needs some time... But I think it is better to spend the
time rather than risking to have again piles of defective boards (there
are already >20 here). If you ever wondered why projects cost
more than anticipated or planned, this is another example...

But since we know that others have managed to solder the DM3730CBP
(there are industrial modules to buy and even some phones...), there
is no reason to resign.

> 
>> it takes less than 4 weeks to produce, test and ship all missing
>> boards. They have planned that it works even if holiday season
>> is coming. And our shipment plan is by sequence of order. So
>> if you did order early, it is even a little closer.
> 
> So it is a stony path towards a community driven independently
> developed and produced open smartphone platform. But Rome
> wasn't built in one day...
> 
> And we have to solve this issue before we can start to think
> about a future quad core LTE device with multitouch... Please
> keep the faith.

So the ball is starting to roll again. But still with a unclear outcome
and very unclear timing.

I never had expected that after all the challenges we had that
production yield (of small quantities) becomes the biggest one.
Even after doing test runs like the GTA04A3. This is very different from
writing software... And the most astonishing thing is that this TI
document mentioned above also mainly talks about the problems
but does not give clear solutions. I.e. it is a mixture of luck, machinery
and long-term experience. In German we call this "Lehrgeld zahlen"
(pay apprentice's dues).

Sorry and thanks again for your patience with this project,
Nikolaus

[1]: http://download.goldelico.com/gta04/images/20120711%20Xray/