HuntnGather – Diff between revs 31 and 37
?pathlinks?
Rev 31 | Rev 37 | |||
---|---|---|---|---|
1 | Short: File indexing and search utilities. |
1 | Short: File indexing and search utilities. |
|
2 | Author: Wizardry and Steamworks |
2 | Author: Wizardry and Steamworks |
|
3 | Uploader: "Wizardry and Steamworks" <office@grimore.org> |
3 | Uploader: "Wizardry and Steamworks" <mail@grimore.org> |
|
4 | Type: util/dir |
4 | Type: util/dir |
|
5 | Replaces: util/dir/HuntnGather.lha |
5 | Replaces: util/dir/HuntnGather.lha |
|
6 | Version: 1.7.3 |
6 | Version: 1.7.4 |
|
7 | Architecture: m68k-amigaos |
7 | Architecture: m68k-amigaos |
|
8 | |
8 | |
|
9 | Hunt & Gather - File search and indexing utilities. |
9 | Hunt & Gather - File search and indexing utilities. |
|
10 | (or Hunt'n'Gather bro, ghetto style) |
10 | (or Hunt'n'Gather bro, ghetto style) |
|
11 | |
11 | |
|
12 | © Copyright 2021 by Wizardry and Steamworks |
12 | © Copyright 2021 by Wizardry and Steamworks |
|
13 | |
13 | |
|
14 | Released under the MIT License, all rights reserved. |
14 | Released under the MIT License, all rights reserved. |
|
15 | |
15 | |
|
16 | -=:[ Changes ]:=- |
16 | -=:[ Changes ]:=- |
|
17 | |
17 | |
|
18 | TBA: |
18 | 20211105: |
|
- | 19 | * Use a single binary for all CPUs. |
||
- | 20 | * Refocus the documentation to match the latest changes. |
||
19 | * Testing with MuForce. |
21 | * MuForce and MuGuardianAngel memory usage stress testing. |
|
20 | * Tested using large entire drives and eliminated some bugs. |
22 | * Large storage (cca. 8k directories / 80k files) stress testing. |
|
21 | * Use locale for string comparisons. |
23 | * Use locale for string comparisons on the Amiga. |
|
22 | * Gather now requires either of -a (add), -r (remove), -c (create). |
24 | * "Gather" now requires either of -a (add), -r (remove), -c (create). |
|
23 | * Allow specifying multiple paths when gathering. |
25 | * Allow multiple paths when issuing the "Gather" command. |
|
24 | |
26 | |
|
25 | -=:[ Introduction ]:=- |
27 | -=:[ Introduction ]:=- |
|
26 | |
28 | |
|
27 | |
29 | |
|
28 | Hunt and Gather are two utiltities for indexing and then searching |
30 | "Hunt" and "Gather" are two utiltities for indexing and then searching |
|
29 | fileswithin a drive or directory designed to speed up searching files. |
31 | fileswithin a drive or directory designed to speed up searching files. |
|
30 | |
32 | |
|
31 | The Gather utility is meant to index any path and generate a search |
33 | The "Gather" utility is meant to index any path and generate a search |
|
32 | database. The Hunt utility will then open the database generated by |
34 | database. The "Hunt" utility will then open the database generated by |
|
33 | Gather and look for files matching the string provided to Hunt as |
35 | Gather and look for files matching the string provided to "Hunt" as |
|
34 | parameter. |
36 | parameter. |
|
35 | |
37 | |
|
36 | The utility was designed to check large collections of icons but the |
38 | The utility was designed to check large collections of icons but the |
|
37 | indexing and the fast search results proved indispensable enough for a |
39 | indexing and the fast search results proved indispensable enough for a |
|
38 | self-standing program. |
40 | self-standing program. |
|
39 | |
41 | |
|
40 | -=:[ Design ]:=- |
42 | -=:[ Design ]:=- |
|
41 | |
43 | |
|
42 | Hunt and Gather are designed with constant memory usage in order to be |
44 | "Hunt" and "Gather" are designed with constant memory usage in order |
|
43 | suitable for all Amiga models. Namely, the Gather utility will search |
45 | to be suitable for all Amigas. Namely, the "Gather" utility will |
|
44 | all files in a given path, sort the files in ascending order by using |
46 | search all files in a given path, sort the files in ascending order by |
|
45 | an external merge sort (tailored down to a 256KiB memory limit). |
47 | using an external file-based merge sort. |
|
46 | |
48 | |
|
47 | Conversely, Hunt uses brute force to search for files but by reading |
49 | Conversely, "Hunt" searchs files by reading lines from the database |
|
48 | lines from the database without loading the entire database in RAM. |
- | ||
49 | Perhaps ulterior versions of Hunt might partition the database file |
- | ||
50 | just like Gather does and then build Tries in oder to speed up finding |
50 | without loading the entire database in RAM or by searching files |
|
51 | files on the filesystem. |
51 | again. |
|
52 | |
52 | |
|
53 | The project adheres to the ANSI C standard and Amiga-centric semantics |
53 | The project adheres to the ANSI C standard and Amiga-centric semantics |
|
54 | are compiled conditionally (in case the "___AmigaOS__" macro is |
54 | are compiled conditionally (in case the "___AmigaOS__" macro is |
|
55 | defined at compile time).Otherwise, Hunt & Gather should run under any |
55 | defined at compile time). Otherwise, "Hunt" and "Gather" should run |
|
56 | platform that benefits from an ANSI C compiler. |
56 | under any platform that benefits from an ANSI C compiler. |
|
- | 57 | |
||
- | 58 | The project is developed from scratch on a real Amiga using StormC. |
||
57 | |
59 | |
|
58 | -=:[ Usage ]:=- |
60 | -=:[ Usage ]:=- |
|
59 | |
61 | |
|
60 | First the Gather utility is used to index a path: |
62 | The "Gather" utility is used to index a path. The following command: |
|
61 | |
63 | |
|
62 | |
64 | |
|
- | 65 | Gather -c RAM: |
||
63 | Gather RAM: |
66 | |
|
64 | |
67 | |
|
65 | |
68 | will create a file in the S: directory named "gather.db". "Gather"is |
|
66 | which will create a file in the S: directory named "gahter.db". While |
69 | verbose by default and will show the user what the utility is doing |
|
67 | Gather is running, the utility will display the number of indexed |
70 | but the behaviour can be changed with the "-q" (quiet) flag that will |
|
68 | directories and files on the command line. |
71 | make "Gather" print only errors. |
|
69 | |
72 | |
|
70 | In order to look for a file, the Hunt utility is invoked with an |
73 | In order to look for a file, the "Hunt" utility is then invoked with |
|
71 | Amiga search pattern: |
74 | an AmigaOS search pattern, for instance, the pattern "#?test#?: |
|
72 | |
75 | |
|
73 | |
76 | |
|
74 | Hunt #?test#? |
77 | Hunt #?test#? |
|
75 | |
78 | |
|
- | 79 | |
||
76 | |
80 | "Hunt" will then search the database previously generated by the |
|
- | 81 | "Gather" utility and will print out all the paths corresponding to the |
||
- | 82 | files matching the supplied pattern. |
||
77 | in this case, "#?test#?", that will be compared to all the files |
83 | |
|
78 | indexed previously by Gather. In case any of the files previously |
84 | In the previous example, in case any of the files previously indexed |
|
79 | indexed by Gather contain the term "test", then the Hunt utility will |
85 | by "Gather" contain the term "test", then the "Hunt" utility will |
|
- | 86 | display the path to the file. |
||
- | 87 | |
||
- | 88 | At some point you might decide to add some other path to the search |
||
- | 89 | database as well. In that case, "Gather" would be invoked with the |
||
- | 90 | "-a" option instead of "-c" in order to add the files: |
||
- | 91 | |
||
- | 92 | Gather -a HDH0:Icons/ |
||
- | 93 | |
||
- | 94 | "Gather" will then index the additional directory and add the new |
||
- | 95 | files to the database. Adding a path to the index database will |
||
- | 96 | require that "Gather" sorts the database again such that after adding |
||
- | 97 | the new files, "Gather" will proceed with sorting. |
||
- | 98 | |
||
- | 99 | Lastly, the "-r" parameter can be used with "Gather" to remove paths |
||
- | 100 | that have been previously indexed. Let's say that you have indexed the |
||
- | 101 | following paths with "Gather": |
||
- | 102 | |
||
- | 103 | RAM: |
||
- | 104 | HDH0:Icons/ |
||
- | 105 | |
||
- | 106 | but now you would like to remove the "RAM:" path and all the files |
||
- | 107 | indexed below that path. In that case, you would issue a "Gather" |
||
- | 108 | command with the "-r" parameter: |
||
- | 109 | |
||
- | 110 | Gather -r RAM: |
||
- | 111 | |
||
- | 112 | and "Gather" will remove all files matching the "RAM:" path. Removing |
||
- | 113 | a path with the "-r" parameter does not take a long time compared to |
||
80 | display the path to the file. |
114 | adding files to the database. |
|
81 | |
115 | |
|
82 | -=:[ Gather ]:=- |
116 | -=:[ Gather ]:=- |
|
- | 117 | |
||
- | 118 | "Gather" requires that one of the following parameters is specified: |
||
- | 119 | * -a (add files to an already existing database), |
||
- | 120 | * -r (remove files from an already existing database), |
||
- | 121 | * -c (delete the previous database file and create a new database). |
||
83 | |
122 | |
|
84 | The Gather utility takes one single parameter representing the path |
123 | The "Gather" utility takes several paths as parameters representing |
|
- | 124 | the paths to be indexed; for example, all the following paths are |
||
85 | to be indexed; for example, all the following paths are valid: |
125 | valid: |
|
86 | |
126 | |
|
87 | |
127 | |
|
88 | RAM: |
128 | RAM: |
|
89 | DH0:System/ |
129 | DH0:System/ |
|
90 | |
130 | |
|
91 | |
131 | |
|
- | 132 | When the "Gather" utility runs, a database is created at "S:gather.db" |
||
- | 133 | containing all the found files. |
||
- | 134 | |
||
- | 135 | "Gather" is also happy to work with a different database file other |
||
92 | When Gather runs, a database is created at "S:gather.db" by |
136 | than the default database at "S:gather.db" by passing the "-d" |
|
93 | overwriting the previous database. For best results, Gather should |
137 | parameter when "Gather" is invoked. For instance, the following |
|
- | 138 | command invocation will create the database file at "T:gather.db" and |
||
- | 139 | index the paths "RAM:" and "HDH0:Icons": |
||
- | 140 | |
||
- | 141 | Gather -d T:gather.db RAM: HDH0:Icons |
||
- | 142 | |
||
94 | run periodically and should scan a path that is most frequently |
143 | Conversely, the "Hunt" utility can then be used to search specified |
|
- | 144 | database files: |
||
- | 145 | |
||
- | 146 | Hunt -d T:gather.db #?test?# |
||
- | 147 | |
||
- | 148 | The previous "Hunt" command will search a database file located at |
||
95 | searched for files. |
149 | "T:gather.db" for all files matching the pattern "#?test?". |
|
96 | |
150 | |
|
97 | -=:[ Hunt ]:=- |
151 | -=:[ Hunt ]:=- |
|
98 | |
152 | |
|
99 | Hunt is the counterpart to Gather and will search the database at |
153 | "Hunt" is the counterpart to "Gather" and will search a given database |
|
100 | "S:gather.db" for files matching the terms passed to Hunt on the |
154 | generated by the "Gather" utility for files matching the terms passed |
|
101 | command line. |
155 | to "Hunt" on the command line. |
|
102 | |
156 | |
|
103 | For instance: |
157 | For instance: |
|
104 | |
158 | |
|
105 | Hunt #?test#? |
159 | Hunt #?test#? |
|
106 | |
160 | |
|
107 | will search all files in the Gather database "S:gather.db" for the |
161 | will search all files in the "Gather" database "S:gather.db" for the |
|
108 | term "test". If any file within the database partially matches the |
162 | term "test". If any file within the database partially matches the |
|
109 | term "test", then Hunt will display the path on the command line. |
- | ||
110 | |
- | ||
- | 163 | term "test", then "Hunt" will display the path on the command line. |
||
- | 164 | |
||
- | 165 | "Hunt" uses AmigaOS pattern for matching the file names on AmigaOS. |
||
- | 166 | |
||
- | 167 | -=:[ Notes ]:=- |
||
- | 168 | |
||
- | 169 | * The "Gather" utility will be slow and that is the intended |
||
- | 170 | behaviour: slow indexing with "Gather", fast searching with "Hunt". |
||
- | 171 | |
||
- | 172 | * Temporary files might end up created in the same location where the |
||
- | 173 | "Gather" utility is invoked. Traditionally the temporary directory |
||
- | 174 | on AmigaOS is mainted in RAM but "Gather" cannot use RAM since it |
||
- | 175 | intends to index very large hierarchies. Fortunately, "Gather"will |
||
- | 176 | delete the temporary files once "Gather" is done indexing. |
||
- | 177 | |
||
- | 178 | Nevertheless, in case you intend to index a large filesystem |
||
- | 179 | hierarchy please make sure that you invoke "Gather" from a directory |
||
- | 180 | that is able to hold large temporary files. |
||
- | 181 | |
||
- | 182 | * The output of the "Hunt" utility can be combined with the pipe |
||
- | 183 | operator (in newer AmigaOS releases) or the PIPE: handler on older |
||
- | 184 | AmigaOS releases in order to to perform some action on the found |
||
- | 185 | files. For example, using Thomas Radtke's "from" utility located at: |
||
- | 186 | |
||
- | 187 | http://aminet.net/package/util/batch/from |
||
- | 188 | |
||
- | 189 | and the Workbench 3.2 "MD5Sum" utility, you could print out the MD5 |
||
- | 190 | hashes of all files indexed by "Gather" ending in "#?.library: |
||
- | 191 | |
||
- | 192 | Hunt #?.library | from - md5sum $1 |
||
- | 193 | |
||
- | 194 | Or you could generate a list of versions of all libraries indexed |
||
- | 195 | with the "Gahter" utility: |
||
- | 196 | |
||
111 | Hunt uses Amiga-style pattern for matching the file names. |
197 | Hunt #?.library | from - version $1 |
|
112 | |
198 | |
|
113 | -=:[ Source ]:=- |
199 | -=:[ Source ]:=- |
|
114 | |
200 | |
|
115 | The project is open sourced and licensed under MIT. The source code |
201 | The project is open sourced and licensed under MIT. The source code |
|
116 | is included in the AmiNET release or can be checked out via subversion |
202 | is included in the AmiNET release or can be checked out via subversion |
|
117 | from the Wizardry and Steamworks repository: |
203 | from the Wizardry and Steamworks repository: |
|
118 | |
204 | |
|
119 | svn co http://svn.grimore.org/HuntnGather |
205 | svn co http://svn.grimore.org/HuntnGather |
|
- | 206 | |
||
- | 207 | StormC was used as the developer environment. |
||
120 | |
208 | |
|
121 | -=:[ Mentions ]:=- |
209 | -=:[ Mentions ]:=- |
|
122 | |
210 | |
|
123 | The code includes a shim for "getopt" in order to process command line |
211 | The code includes a shim for "getopt" in order to process command line |
|
124 | parameters on Amiga without changing the semantics. The shim is |
212 | parameters on AmigaOS just like one would on a POSIX sytem. The shim |
|
125 | created by Daniel J. Barrett, barrett@cs.umass.edu and is available on |
213 | is created by Daniel J. Barrett, barrett@cs.umass.edu and is |
|
126 | AmiNET: |
214 | available on AmiNET: |
|
127 | |
215 | |
|
128 | http://aminet.net/package/dev/misc/GetOpt-1.3 |
216 | http://aminet.net/package/dev/misc/GetOpt-1.3 |
|
129 | |
217 | |
|
130 | -=:[ Contact ]:=- |
218 | -=:[ Contact ]:=- |
|
131 | |
219 | |
|
132 | E-Mail(tor): |
220 | E-Mail(tor): |
|
133 | |
221 | |
|
134 | office@3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion |
222 | office@3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion |
|
135 | |
223 | |
|
136 | Website: https://grimore.org/amiga/hunt_and_gather |
224 | Website: https://grimore.org/amiga/hunt_and_gather |
|
137 | Website(tor): |
225 | Website(tor): |
|
138 | |
226 | |
|
139 | http://3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion/ |
227 | http://3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion/ |
|
140 | |
228 | |
|
141 | Discord: https://discord.gg/k9kyDsa |
229 | Discord: https://discord.gg/k9kyDsa |
|
142 | |
230 | |
|
143 | |
231 | |