Hacker News new | past | comments | ask | show | jobs | submit login
Using AI and AR to improve ArcellorMittal's production process at a Hackathon (onebonsai.com)
72 points by netb on Sept 21, 2017 | hide | past | favorite | 41 comments



This is a really cool piece of engineering, but I'm bothered by the idea that the going rate for a huge multinational corporation to get a prototype of a potentially extremely valuable technology is less than 10k of prize money [1], plus a "mini-drone race" and "snacks and food".

Maybe it's just me, but I find this type of corporate-sponsored hackathon exploitative and distastefull. We're not talking about an event that is producing a social good in an area that is otherwise underinvested. This is purely profit-motivated, and ArcelorMittal can afford to pay a lot more for a prototype like this.

1. https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack...


I agree. While I understand the OP's point that it's a valuable experience for the team, a glimpse into solving real problems, it's not a finished product, etc., I still think ArcelorMittal captures the bulk of the value here.

They had 70 teams competing. The team in the article had 5 members working for 36 hours, with 3 hours of sleep. Let's say the team members would fetch $150k/year. $75/hr. 5 people, 33 hours. $12,375. And there were 70 teams competing for $10k + snacks.

On the flip side, how much does ArcelorMittal lose on a single failed weld? I would venture a guess it's well north of $10k.


Just to clarify, there were only ~15 teams, 70 is the number of contestants


For you, and I'd say the whole HN audience, it is quite obvious that spending money in an AI approach would be extremely useful. For a steel producer, this is not obvious at all, and consider it a long shot.

I would even bet that the idea of whoever organized the hackaton was not to get free labor, but to prove to someone up there that AI can bring new interesting solutions. In view of the results, I'm quite sure they will be spending more in the future, maybe even starting a new department or a serious research program or something like that in the long term.


I'm with you on this. I can't imagine any other complex discipline where experts offer their time & creativity to the lowest bidder so quickly.

Programming seems uniquely generous in this regard. Or stupid.


Thanks for your view. However, it is important to see this event also for what it is. It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. Secondly, it is a way for students to get a glimps of what the corporate projects and challenges are.

Additionally, for us, as professionals, it is a way to show, although in a very rough fashion, what is possible, and where we can help. We saw this event more as a team building than work.

The result we got at the hackathon is a very rough prototype, not a finished product. In no way, this can directly be implemented on the floor. As such, input from experts is still required, either us, or other experts in the field.

Therefore, these hackathons are useful, in my opinion, not only to the company, but also the participants :)


> It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business.

And there's nothing wrong with that, so much so, that we created a title for these people: Consultant.

They're highly paid, and available to bring years of domain experience to a business.

How much value did ArcellorMittal get in exchange for 8.5k and some pizza? How much would <leading AR firm> charge, in comparison?

(Man, programmers desperately need someone to tell them that no, not everyone just "gets" programming. You have a skill, charge for it!)


Really agree with this. Not everything is about money, if it was we wouldn't have squid, postfix, apache, nodejs, python and the millions of other open source software that I and everyone else here relies on daily. Ask the maintainer of GnuPG if he was ever given $10K by a company (prior to the Linux Foundation's Core Infrastructure Initiative grant).


This is the distinction I was getting at - something like GnuPG provides tremendous value to society as a whole. Saving ArcelorMittal some money in its manufacturing process doesn't benefit anyone except ArcelorMittal's executives and stockholders.

So I stand by my point. There's no doubt the participants got a lot out of this Hackathon, but I'd wager that ArcelorMittal got significantly more.


I see your point but I assume the producers of the hack own the rights and IP to their work i.e. there's nothing stopping them releasing what they did as open source and building a community around it (at least that's been the case with every hackathon I've ever participated in).

If the above assumption is true then I don't see the problem. ArcelorMittal get some buzz and maybe some ideas to pursue, the hackers get a fully paid fun weekend and a little cash for their efforts, society potentially benefits from the seeds that were planted.


Oh but this was about money. Saving money for a multinational corporation that crushed its rivals in Europe. Nice job.


Why are you bothered by it? The 70 teams that joined the hackathon did it out of their own free will, and probably enjoyed it?


Hey!

Thanks guys for your comments! I'm the writer of the article (Evarest).

Just a bit of an update in the mean while :) Our AI team is now busy writing a bit more detail on the actual AI part of the project. The project used:

Random Forested Height Data Deep Learning Models

We trained on the entire dataset of 60.000+ elements, but did pre-processing on the data. I am not placed to answer detailed questions on the AI part, but we will update the article soon with more details.

Note that this is not a perfect result, nor that we claim to be all-knowing or have all the answers. This is a result from a hackathon, which involved working in suboptimal conditions under heavy time constraints.

As for the AR part, we did not have access to AR helmets. We are aware that not all the data that we showed would work on an AR helmet (currently). However, this tech is totally feasible in the short to medium term. Also, it is completely feasible on tablet or similar.

The concept of the hackathon was to show the potential of these technologies, and to integrate the AR and AI, which I find we succeeded in.


Great hack. Really enjoyed reading this! Noob question: the weld as shown in the AR video looks really detailed (fissures, deformations etc). Was that rendered by you using the heightmap data alone or did you have access to top and bottom images of the welds as part of the training data?


I think it was data from the laser scanner as it was monochromatic (w/ shading)


Thanks!


Thanks all for your great comments! For some reason, I cannot answer all your questions as the system blocks me from posting, probably because I did not post the original link here... Therefore, I am going to answer as much as possible in one post :)

I totally understand the feelings of some about "corporate hackathons". While it is valid that companies get probably more from these hackathons than us, I want to stress the following:

- We had the opportunity to work on a challenge with a huge amount of data that I would not otherwise be able to get

- We were able to work with great guys I would not have met otherwise

- We stretched ourselves to see if we could combine AI and AR, which would probably not have happened otherwise

- The implementation of anything built during the weekend would takes months for anyone, even with the code we produced

- This is a learning experience for all parties

The data we used was based on input from a laser scanner. The scanner provides a height map, which would be shown as a colored PNG. We also had point clouds, but only for a few months, so not much use in AI... We used the PNG to generate a bump and normal map to show in AR.

Thanks again and hope that you had fun reading it :)


UPDATE: Thank you for your comments and remarks !

Based on the inputs of the community, we rewrote the AI part of the article (see section 'Building the AI Assistant'). More details are available now.

--> https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton...


I suggest changing "conjest" to "condense".


Damn, how many years of experience do you need to be able to come up with workable solutions as fast as this?


6 months, here's why: They random forested height data, everyone random forests these days. It's quick and dirty and yields good accuracy. Not only that, everyone seems to support a built in Random Forest training model, programs from pandas/numpy/scikit to R has it built in. Height data seems the easiest to go by, to short and it has a kink that may cause cracking in the future as the metal oxidizes. What were their 3 models? Who knows? But their idea of combining 3 models, that 3 different teams have made is standard in all DataScience competitions. This is called ensembling. ArToolkit for Unreal Engine, someone else made that and they probably just connected a bunch of blocks together with Unreal's script engine. Honestly, 6 months max and you can do this in a day yourself.


Multiple years, here's why: you need to decide what will work and what will not, as you don't have time to go the wrong route. And that takes experience which it would be difficult to gain in 6 months.


Yeah right:

We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction. Everything was trained on the raw images resized to 20x150 pixels.


I am no expert into AI. The AR part we learned because we are busy in VR. It is a natural evolution in that case, as we can use the workflow we are used to in VR.

The AI part however is more complex, as it requires understanding of the concepts and methodologies behind it. We used Deep Learning methodologies for this project. This is an upcoming field in AI, which evolves very quickly. If you are interested in it, it's a great time to start learning the basics of it!


I just got into Unreal Engine with the VR template project, which includes grabbing/teleportation functionality by default. It's been really easy to get into and if you have a headset, I'd really recommend trying it out if you've had a VR development itch.


Indeed, thanks for the comment! Unreal Engine is a great engine to quickly build prototypes in. We are used to work with the included VR Template of Unreal, and it most of the times works great.

Also for more serious projects, Unreal did not let us down.


How much money will Arcelor earn from this piece of Software? You just implemented it for free!


Hey ! I am one of the AI guy of the team. I agree that if the solution gets implemented on the lines, they will probably make more net money out of it than what was spent for the challenge (which is more than 8.5k€ and some pizzas as said in other comments BTW, the challenge was probably more expensive to organize). However, let's be realistic, they will have to put a lot more money into this before they get a viable product to insert in their production lines. The solution we have built is way too simple regarding actual operational constraints (e.g.: operator safety).


This. A lot of the other commenters have no idea what it takes to get software productionized in an industrial company.

This is more of a proof-of-concept (and a very impressive one, congratulations), but it will take much more development and resources to get it working on the production line.


I think nobody here thinks that this piece of code from weekend will be used on Monday on the band. The hardest part is coming up with a solution and that company got it almost for free. I am pretty sure that this company will even aplly for a patent for this solution! The team should at least get a remuneration if this "proof-of-concept" implemented and will actually be used.


Can you elaborate on: "We received feedback from the AI subteam, and incorporated methods to show parts of the weld that were prone to breaking"? What techniques did they use to extract this information from the network?


Hi! I'm from the AI subteam. We actually didn't have time to implement this part of the solution. Our idea was to use this library: https://github.com/raghakot/keras-vis


Awesome work! I understand its a proof of concept for a hackathon but what's the value of AR here? If AI can indicate a fault with a high success rate how does AR help this situation?


Thanks! Good question, the AI part is clear, it will indicate if there is a big chance of break. The AR will help the operator decide without going to the actual break whether the AI is correct.

The AI only suggests, does not decide on its own. As such, an operator will be able to see the break in full detail (the top and bottom and up close) to better be able to decide whether to continue production, or do a reweld.

Also, sometimes the AI will not be able to provide a good input, as new materials might be welded together and no data is available for them. In that case, again it's up to the operator to actually give the go or no-go.

The AI will inform the operator after about 2 seconds, the operator then still has about 8 seconds to finalize his decision.

The current prototype is not really usable, as it has some big usability issues. We might work further on improving this prototype with the organizer, depending on their interest.


I didn't understand where the AR was overlaid. Was it on the glasses on which the operator was wearing or on a LED monitor ?


They did not have VR glasses, so they simulated them using a helmet-mounted webcam and a monitor. The AR weld risk profile (my words) is overlaid on the captured video images. See the article's video with caption 'Screen-capture of the AR experience...'


Nowhere, it wasn't actually AR.

It was a video feed that had the 3D model overlayed on the desktop/laptop the video was feeding from.

This isn't just semantics - moving from their setup to an actual AR HMD - even if it was just the current state of the art (Hololens, ODG, Epson etc...) wouldn't be trivial and would likely be a different workflow.


FTA: "Furthermore, we decided that we would require a setup that actually worked. Given that we did not have any AR hardware available (eg AR glasses such as a Microsoft Hololens), we quickly built one ourselves using a production helmet and a webcam."

Medium images sometimes take a while to load, but if you wait you can see their "webcam on a hard-hat helmet" setup for the demonstration to the judges. The overlaid AR + video feed then shows up on a monitor.


Can you please elaborate on the deep learning part. What kind of network dis you use?


We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction.

Everything was trained on the raw images resized to 20x150 pixels.


extremely inspiring!!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: