Issue Details (XML | Word | Printable)

Key: MBS-5766
Type: Task Task
Status: Closed Closed
Resolution: Fixed
Priority: Normal Normal
Assignee: Frederik "Freso" S. Olesen
Reporter: nikki
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
MusicBrainz Server

Open up /release/ and /work/ in robots.txt

Created: 21/Jan/13 10:18 PM   Updated: 11/Feb/13 11:40 AM   Resolved: 11/Feb/13 11:40 AM
Component/s: None
Affects Version/s: None
Fix Version/s: 2013-02-11



Sort Order: Ascending order - Click to sort in descending order

Aurélien Mino added a comment - 22/Jan/13 07:34 AM

I also think url is pointless.


Frederik "Freso" S. Olesen added a comment - 23/Jan/13 05:37 AM

I've been thinking about /url/ since the meeting, and especially since we cannot search through URLs via MusicBrainz, I think this might actually be useful. Ie., to have them unmasked. It may be possible (once MusicBrainz has been indexed with the new paths) to e.g. search with site:musicbrainz.org/url/.


nikki added a comment - 23/Jan/13 11:51 AM

You don't need to open up /url/ to do that. URLs are always linked to an entity, so will (if recordings are indexed) always be part of another indexed page. e.g. http://www.google.de/search?q=site%3Amusicbrainz.org+http%3A%2F%2Fwww.youtube.com%2Fmorningmusumechannel Perhaps not quite as obvious, but I suspect the number of people who think to search for URLs that way will be rather low in the first place.

I think I'd rather see 1.5 million more interesting pages indexed...


Oliver Charles added a comment - 23/Jan/13 12:39 PM

In the long term I'd like /url to be opened up, because I still haven't seen a convincing reason otherwise. People are making all sorts of assumptions about search indexers - such as if we open /url it will suddenly decide to index nothing but /url, but I highly doubt that is the case. However, I can't back up my refutal of that claim, so I'm fine with just opening /release and /work for now, and /url can come later.


nikki added a comment - 23/Jan/13 01:07 PM

I'm not claiming it will suddenly decide to index nothing but /url/ any more than you're claiming it will not do any /url/ pages until it's done everything else. It just seems logical that if it spends some of its time indexing /url/, that's time it's not spending indexing pages with more interesting content.


Oliver Charles added a comment - 23/Jan/13 01:13 PM

any more than you're claiming it will not do any /url/ pages until it's done everything else

I know, that's why I said "I can't back my claim up either" and suggested we continue without /url.


Robert Kaye added a comment - 23/Jan/13 09:08 PM

nikki said:
>I think I'd rather see 1.5 million more interesting pages indexed...

This is a great argument. With that in mind, lets leave /url off for now. In a few months, lets revisit this and see if we want to do this.


Frederik "Freso" S. Olesen added a comment - 23/Jan/13 09:22 PM

I updated the pull request.