/usr/lib/info || ILS Benchmarks?

Front Page · News · Features
Software · Events · Standards
Ask Anything · Opinion · Diaries
Reviews . MLP . Everything

ILS Benchmarks?
By ksclarke, Section Ask Anything Posted on Fri Jul 11th, 2003 at 06:45:59 AM EST

I wonder if anyone is aware of a set of tests for catalogs that would test their response times on a range of possible queries (from simple author or title searches to more complex multi-field testing). While these tests would only cover known item searches, it would be interesting, for instance, to see how well one library catalog performs when compared to another with the same record set. One might want to load up one of the open source ILSes with the same set of records s/he has in the local proprietary database, for instance, and compare the results.

I realize a lot would depend on the type of indexes created by both. An effort should be made to make sure they are the same but, if they aren't, it would be interesting to learn one of the ILS choices could not create an index to make available a particular type of search. I guess what I'm curious about is the range of search possibilities that a patron might want to use. Such a tool might also be a (small) factor in considering which ILS to purchase.
As to how it would do the query I do not know... would it use the ILS' Web interface, or JDBC, or Z39.50?
Anyone know of any such thing? A possible query collection would be good, but software that does this would be even better. I would expect software to make multiple attempts at each query to try to remove random blips that might slow down the retrieval.
Hmmm, maybe query sets would also need to be domain specific? I haven't given this a lot of thought, just wanted to throw it out there...
Not even sure such a thing really would produce information that was useful in comparing different systems (maybe there are too many variables since there is no standard way to access a library catalog)... just thinking such a comparison would be nice...
FWIW,
Kevin

< Will Microsoft control our information access? (5 comments) | Update to OpenURL Patent Story (3 comments) >

Menu

· submit story
· create account
· faq
· search
· recommended reading
· editorial guide
· masthead

Login

Make a new account

Username:

Password:

Related Links

· More on
· Also by ksclarke

ILS Benchmarks? | 5 comments (5 topical, 0 editorial, 0 pending) | Post A Comment

ILS Benchmarks (none / 0) (#1)
by kcoyle on Fri Jul 11th, 2003 at 10:58:31 AM EST
(User Info) http://www.kcoyle.net

We've developed complex benchmarking to test our new ILS, an Aleph 500 system. We took about 10,000 real searches from our old system, set the timing of them based on average times between searches, and have software that runs them against the new catalog. In spite of that, it's hard to know how accurately this predicts real use.

What has been very interesting in doing this is how greatly file design affects the response time for different types of searches (keyword vs. heading searches; truncation; proximity; etc.). It would be fascinating to run the same searches against a number of different file designs and different database sizes.

I'm referring to the U of California MELVYL system. The old catalog is at http://www.melvyl.ucop.edu; the new one is http://melvyl.cdlib.org.

[ Reply to This ]

Real searches by ksclarke, 07/14/2003 12:17:34 PM EST (none / 0)

Performance benchmarks by kcoyle, 07/17/2003 11:30:09 AM EST (none / 0)

Benchmarks & Blueprints (none / 0) (#2)
by art on Fri Jul 11th, 2003 at 11:28:37 AM EST
(User Info) http://www.uwindsor.ca/library/leddy/people/art

Performance benchmarks can be a minefield. Even the TPC benchmarks have had a somewhat torturous journey and the dust still hasn't settled on them. Sometimes performance also involves a double-sided metric depending on the scale of use. The JBoss - PostNuke work is a case in point. For a small number of users, the caching and pooling benefits of JBoss probably don't outweigh the java overhead, but for heavy use it runs circles around PostNuke. Another aspect could concern how easily the system(s) can be clustered, or run on a grid. The hardware itself, especially if using Linux, is not a huge cost, a system that might be slower on one server looks much more attractive if it can be ramped up with cheap hardware.

Functional benchmarks might be more attainable, the Bath Profile, for example, has the notion of a minimum threshold for compliance. At one point there was talk of a MARC profile for Z39.50 and there was a gradual realization that the ILS options at the time seemed to be falling short for searching arbitrary MARC field/subfield/indicator combinations. It has always bothered me that libraries invest a huge effort into a format that our systems have never been able to fully exploit. I think it's also important to be able to arbitrarily add metadata for searching, RDF is a natural for this, since the elements needed for display may be radically different than what is needed for retrieval. Too many IR systems try to make the same data model do both.

Wow, if there's an award for "most unhelpful" answer then I must be pulverising the competition by now. I guess a more useful answer is that a library version of the Pet Store blueprint would be nice for this kind of thing. The other advantage of a blueprint is that it helps show the commonalities between systems, a library system blueprint would have some overlap with others.

[ Reply to This ]

Most unhelpful? by ksclarke, 07/14/2003 12:23:28 PM EST (none / 0)

ILS Benchmarks? | 5 comments (5 topical, 0 editorial, 0 pending) | Post A Comment

front page | submit story | create account | faq | search