CRC-indexing on files (allows renaming shared files)

OneSwarm Forum » Feature Requests

CRC-indexing on files (allows renaming shared files)

(18 posts) (9 voices)

Started 4 years ago by airship
Latest reply from lah

Tags:

No tags yet.

airship
Member

I would like to bring in som smart crc-hashing for each file in shares. A type of linked connection between all swarms/torrents you share with a crc-like-hash-number, that points out to which file in which torrent this number exists for.

In this way you could search for a file and start your download. And if your "source" gets offline then OneSwarm could search for the crc-like-hash-number in the entire network, and other sources (that have renamed that exact file) can be marked as the same file, and the download can continue from a source in a "new-other-torrent-but-still-the-same".

This came up when noticing that my friends could not resume a file that I have renamed. The file is unmodified, but still OneSwarm doesnt "regognize it".

This type of hashing is found in most file-sharing software today. I would really like to see it in OneSwarm

Cheers!

Posted 4 years ago #
fesknet
Member

I think that this should be in Oneswarm I have many non limit friends that rename there download when they are done

Posted 4 years ago #
Natanael_L
Member

Sure, but CRC's suck for this. 8 characters for a checksum!?
SHA2+Tiger should be used, even though the checksum would be looong (+200 characters? don't know the exact length).

Posted 3 years ago #
alpahc
Member

A quick test tells me it has been fixed, at least in the latest dev version. Can anyone confirm this ? (Piatek ? :))

Posted 3 years ago #
jivvz
Member

this will probably later be implemented in one way or another.. I heard them talk about File Fingerprints... which would let oneswarm merge sources that are very similar together. for examples files that are renamed.. or mp3s which id3 tags have changed.
I'm very eager for that kind of feature :)

Posted 3 years ago #
alpahc
Member

>this will probably later be implemented

Well it *seems* it has been done already... No confirmation though...

Posted 3 years ago #
2plus2equals4
Member

I am still finding multiple version of same spelling size and hash coming up in searches... Can I search by hash instead of filename?

What about a way to have verified downloads instead of the randomness of using filename search. A hash link like emule or a new torrent ie .p2p xml file with hash meta info etc for swarm? Some way to create verified links for in forums or other community share methods so you know you have the original version of file and main/biggest swarm.

Posted 3 years ago #
piatek
Administrator

We do support links to download things by hash (e.g., try linking to a swarm you've downloaded in a chat window -- although any webpage could use this)

Or are you thinking of something else? Thanks!

Posted 3 years ago #
2plus2equals4
Member

Thank-You for pointing this out. I had missed it due to using jabber not in 1S chat :)

Question, is that hash it spits out based on bittorrent thus bound to file name? Resulting in a changed hash for each rename? Or is it the file hash that is also kept reffered to by isdal in other thread...

isdal: " Since we rely on the bittorrent protocol for the data transfers (and in bittorrent the file name is included in the hash) this would require quite a lot of work to fix. But, we anticipated this when we designed the hashing, so we keep the hash of the file as well. We plan to enable searching based on the sha1 and ed2k hash of the files but it will take some time to implement."

I envision and wish for a hash link based on file SHA1, tiger or better hash that can be clickable and detected by 1S when clicked for easy sharing with people that arent unlimited friends in in program chat enabled. Would also be great for indexing sites when 1S grows.

Posted 3 years ago #
Natanael_L
Member

"I envision and wish for a hash link based on file SHA1, tiger or better hash that can be clickable and detected by 1S when clicked for easy sharing with people that arent unlimited friends in in program chat enabled. Would also be great for indexing sites when 1S grows. "

I've suggested this months ago as one of the first. I don't know who thought about it first, though. :)

These links could replace regualar torrent files in the future, so TPBv2 would only contain pure links with hashes instead of entire torrent files. xD

Posted 3 years ago #
lr
Member

How is made the hash in TPBv2 ?

.torrent files may (optional) contains md5sum for each file, and list of sha1 for each file (one per piece), and length of the file.

If we wan't to use sha1 of the complete file, we can only share with people that have the complete file.
If we wan't to use md5sum of the complete file, we can only share with people that have the complete file or the md5 information in the torrent.

What do you think about use sha1(list of sha1 pieces of a file). It's easy to calculate it for all from the existing torrent files ?
And use optionaly sha1, md5 and length of the complete file.

(sorry for my english)

Posted 3 years ago #
lr
Member

Is it a bad idea ?

I search about this, and I found that it is named a Hash list : http://en.wikipedia.org/wiki/Hash_list

Posted 3 years ago #
airship
Member

Seems like sha1-hashing is now being implemented in 0.6.7. And as long the searches return both the torrent information (with information about parts of files) combined with a sha1 hash.. then Oneswarm will be able to deliver what I asked for when I started this thread.

But if anyone want to discuss on how to go from here (nothing is perfect) then this thread is an interesting start:
http://forum.oneswarm.org/topic/1214

Posted 3 years ago #
Natanael_L
Member

You don't need the entire file - bittorrent clients use that to verify that *each part* is correct continously instead of waiting until the download is done to check that the files is not damaged.

But since OneSwarm is built the way it is with encrypted communication, files can not really be damaged that way - a data package can either be lost or messed up and thus *discarded* and then sent again.

Only verifying that the file is complete and undamaged when it is done is manageable.

Posted 3 years ago #
lah
Member

Natanael_L: Think Your wrong here. 1s download parts from different sources and a source can be corrupted. It's not essential to know witch parts are broken but it's really a great advantage over just knowing the complete swarm is broken.

What I understand 1s encrypt communication between nodes, but not end to end, so probably every node can corrupt data! Thats in a way good as it opens the possibility to cache data as a future improvement.

It's an upcoming feature for 'private sharing' that will support end to end encryption. That's all good but I don't agree on the heuristics for enabling it. It should be crystal clear - a dedicated checkbox or something with a huge warning if You deactivate it. So the risk of sharing private stuff by mistake is minimal.

Posted 3 years ago #
Natanael_L
Member

lah: Sure, a node could mess it up and force a restart of the download.
Then "piece hashes" might be needed.

As of right now I think that it's not a priority, but sure, we might need it in the future for huge files (anything over 2-500 MB?) when a lot of people use it and want everything to just work without a thousand re-downloads.

Posted 3 years ago #
Natanael_L
Member

New idea: Each file can have a description file, like this:

Filename: Test.mp3
SHA256 hash: 0123456789ABCDEF0123456789ABCDEF
Description file: (Test.mp3- ?) 0123456789ABCDEF0123456789ABCDEF.1Sdesc

That file would then contain checksums for the file parts, like a torrent file. Before a download starts of a file of +10 MB , that file's description file will be downloaded first.

Posted 3 years ago #
lah
Member

Parts hash is also good to make sure the network is used for useful purpose and to minimize the impact of dirt inserting attacks on the network.

Yes. Parts hash in a separate resource might be good. You can download it in parallel - You don't really need it unless the file hash mismatch. It should however always be downloaded (and probably be checked for integrity) so it is available on the network if someone needs it.

Posted 3 years ago #

RSS feed for this topic

Reply

You must log in to post.