Help with Bricklink

[Nick2253]

I'm not even sure where to ask this, but I figured FBTB is as good a place as any:

Having used Bricklink for a number of years, I can honestly say that I hate the interface. Unfortunately, there are no serious competitors in town, so we get what we get. However, Bricklink does have all the data that I would need in order to determine everything I want.

So, the question: does anyone know if BrickLink provides an API in order for third-party programs to access and download their database of parts and seller information, including price and quantity of parts for sale? I know you can manually download the catalog, which would be fine if that's all I needed, but it's not. I would also like to be able to download/upload wishlists to the site.

My final goal is to code (or find--could I get so lucky as to have someone who already did this?) a program that allows me to specify a set of parts, and then, based on the properties of those parts and the different BrickLink stores, the program automatically determines the optimal store or stores to buy from in order to minimize the total cost (including at least a rough shipping estimate). Algorithmically, I know exactly how to do all of this; I just need the data.

Right now, I'm stuck trawling through the stores manually, and make a greedy guess as to which stores will give me the lowest cost.

If you have any advice, pointers, or recommendations, I'm all ears!

Thanks,
Nick

[AbstractedIdeal]

I don't know if there's an official API, but you might want to look at how BrickStore pulls info. I'm not sure if it pulls Seller info, but you might be able to co-opt or adapt whatever methods it's using.

Nevermind -- looks like BrickStore pulls its data from the author's own site, so you're likely stuck with screen scraping the data.

[GIR3691]

Have you tried making a wish list of the parts you want? It's not ideal, but you can select all categories, and then "show all." It will give you a list of stores that stock each item on your wish lsit. Refine your parameters and you can look for store names that pop up for multiple things.

In my experience, you'll never find everything you want in one store. I usually end up making two or three orders whenever working on a specific project. Try to go by the rarest/ highest priority items first, then see what stores stock them. From there you can browse their inventory to see if they have other parts you want, and you can usually narrow it down to a couple of stores to get the most out of each order.

Hope this helps somewhat. I too have thought about how nice it would be to make a wishlist and have it generate recommended stores automatically.

[bigospedros]

Wouldn't it be a good idea to approach the owners of Bricklink and talk to them about your proposed idea. They may work with you to incorporate it into the site ?

[Nick2253]

I currently use their wishlist in order to find what I need, and there are actually more efficient ways to determine what I want on BrickLink than your method, but it's still incredibly tedious. I always seem to be designing models that use some of the rarer parts, so I know what you mean about placing multiple orders. However, I have to do lots of legwork in order to determine the best store for my orders, and when all is said and done, I'm putting in a lot of work for minimal savings. Yes, it does add up, so it is worth it, but it really is a pain.

I actually never thought to approach BrickLink directly. I'll have to do that.

Thanks all!

[legodavee123]

Wouldn't it be a good idea to approach the owners of Bricklink and talk to them about your proposed idea. They may work with you to incorporate it into the site ?

Pretty difficult. Since Dan passed on, his family has inherited the website, and I'm not sure anyone that's still there is able to do any difficult code work. I'm not sure if there's anyone doing any coding at BL at the moment-- it's possible, but I'm not aware of anyone doing that...

DaveE

[RfAurora]

Wouldn't it be a good idea to approach the owners of Bricklink and talk to them about your proposed idea. They may work with you to incorporate it into the site ?

Pretty difficult. Since Dan passed on, his family has inherited the website, and I'm not sure anyone that's still there is able to do any difficult code work. I'm not sure if there's anyone doing any coding at BL at the moment-- it's possible, but I'm not aware of anyone doing that...

DaveE

It's not difficult at all. If you haven't yet, read the notification under the login/userid box in the top right corner of Bricklink. Not only is it a memorial to Dan, but it also gives information on who is maintaining the website, and down at the bottom of the page is a link to Northstar Computer Systems where the new moderator works.

[Nick2253]

I got back an official reply from Bricklink:

We don't have anything available at this time. We are considering doing this for the future, but it is not an immediate project.

I guess that answers that. My next option is to create a web-trawling script that will index all the store for me. It will take time, and unfortunately won't be able to update super rapidly, but should be good enough for what I need.

[legodavee123]

It's not difficult at all. If you haven't yet, read the notification under the login/userid box in the top right corner of Bricklink. Not only is it a memorial to Dan, but it also gives information on who is maintaining the website, and down at the bottom of the page is a link to Northstar Computer Systems where the new moderator works.

Maintaining != Coding.

Basically, yes, there are a lot of people who are maintaining the site. They're using the interfaces that Dan wrote to keep the database current, and deal with user problems using those same interfaces. Same way it's been going for years. Similarly, if there are any network or operating system issues, they've got staff at NorthStar to deal with it.

But from what I've heard, new development is difficult, correct me if I'm wrong. If, say, they wanted to implement some new feature to the website, that means digging into the code that Dan wrote, and (last I heard) there wasn't anyone who was familiar enough with the code to do that.

DaveE

[TheCreativeBrickCoUk]

Hi,

I have written a program which does what you require although it doesn't take into account shipping costs - which can make quite a difference to where the the best prices come from. It basically a web-scraping application - I don't run it
very often - perhaps once a month - and I don't require it to cover all of the BL site (although it could).

The following text will give you an idea of what is does for a particular Lego set inventory.
(In this case limited to UK suppliers)

[SAMPLEOUTPUTSTART]

Part 3455, Brick, Arch 1 x 6 (Black) - 4 Required
Found All (4) at Bricks For U UK - B7 (UK) : $0.03 (Used)
Running Totals : 0.92

Part 3023, Plate 1 x 2 (Light Bluish Gray) - 8 Required
Found Some (4) at BricksNParts - BP (UK) : $0.02 (Used)
Found Remainder (4) at Brick Babe - BB (UK) : $0.03 (Used)
Running Totals : 9.10

Part 3623, Plate 1 x 3 (Light Bluish Gray) - 4 Required
Found All (4) at A Mix Of Bricks - MK (UK) : $0.03 (Used)
Running Totals : 9.22

Part 3666, Plate 1 x 6 (Light Bluish Gray) - 15 Required
Found Some (1) at Bricks For U UK - B7 (UK) : $0.04 (Used)
Found Some (10) at Bricks_4_Sale - B9 (UK) : $0.05 (Used)
Found Remainder (4) at Brick Babe - BB (UK) : $0.05 (Used)
Running Totals : 9.99

Summary:

Short By 153 Pieces
Total Pieces : 902
Total Price : $48.08501 (£26.86313)

[SAMPLEOUTPUTEND]



I'm afraid the code is fine for my needs but isn't in any sort of state which I would be happy about releasing.

As a website owner myself I know what a pain scraping programs can be so as I've indicated I only run it every few weeks and it is very 'gentle' - pausing for 10 seconds between each page retrieval - which makes it slow - but kind to BL.

What would be great would be a datafeed or datadump from BL - although they may place too high a value on their data to consider this...

Rob

TheCreativeBrickCoUk

http://www.thecreativebrick.co.uk