HuntnGather – Diff between revs 37 and 44

Subversion Repositories:
Rev:
Only display areas with differences – Ignore whitespace
Rev 37 Rev 44
1 Short: File indexing and search utilities. 1 Short: File indexing and search utilities.
2 Author: Wizardry and Steamworks 2 Author: Wizardry and Steamworks
3 Uploader: "Wizardry and Steamworks" <mail@grimore.org> 3 Uploader: "Wizardry and Steamworks" <mail@grimore.org>
4 Type: util/dir 4 Type: util/dir
5 Replaces: util/dir/HuntnGather.lha 5 Replaces: util/dir/HuntnGather.lha
6 Version: 1.7.4 6 Version: 1.7.5
7 Architecture: m68k-amigaos 7 Architecture: m68k-amigaos
8   8  
9 Hunt & Gather - File search and indexing utilities. 9 Hunt & Gather - File search and indexing utilities.
10 (or Hunt'n'Gather bro, ghetto style) 10 (or Hunt'n'Gather bro, ghetto style)
11   11  
12 © Copyright 2021 by Wizardry and Steamworks 12 © Copyright 2021 by Wizardry and Steamworks
13   13  
14 Released under the MIT License, all rights reserved. 14 Released under the MIT License, all rights reserved.
15   15  
16 -=:[ Changes ]:=- 16 -=:[ Changes ]:=-
17   17  
18 20211105: -  
19 * Use a single binary for all CPUs. -  
20 * Refocus the documentation to match the latest changes. -  
21 * MuForce and MuGuardianAngel memory usage stress testing. -  
-   18 20211114:
22 * Large storage (cca. 8k directories / 80k files) stress testing. 19  
23 * Use locale for string comparisons on the Amiga. -  
24 * "Gather" now requires either of -a (add), -r (remove), -c (create). -  
25 * Allow multiple paths when issuing the "Gather" command. 20 * Rewrite and preparations for pooling memory the Amiga way.
26   21  
27 -=:[ Introduction ]:=- 22 -=:[ Introduction ]:=-
28   23  
29   24  
30 "Hunt" and "Gather" are two utiltities for indexing and then searching 25 "Hunt" and "Gather" are two utiltities for indexing and then searching
31 fileswithin a drive or directory designed to speed up searching files. 26 fileswithin a drive or directory designed to speed up searching files.
32   27  
33 The "Gather" utility is meant to index any path and generate a search 28 The "Gather" utility is meant to index any path and generate a search
34 database. The "Hunt" utility will then open the database generated by 29 database. The "Hunt" utility will then open the database generated by
35 Gather and look for files matching the string provided to "Hunt" as 30 Gather and look for files matching the string provided to "Hunt" as
36 parameter. 31 parameter.
37   32  
38 The utility was designed to check large collections of icons but the 33 The utility was designed to check large collections of icons but the
39 indexing and the fast search results proved indispensable enough for a 34 indexing and the fast search results proved indispensable enough for a
40 self-standing program. 35 self-standing program.
41   36  
42 -=:[ Design ]:=- 37 -=:[ Design ]:=-
43   38  
44 "Hunt" and "Gather" are designed with constant memory usage in order 39 "Hunt" and "Gather" are designed with constant memory usage in order
45 to be suitable for all Amigas. Namely, the "Gather" utility will 40 to be suitable for all Amigas. Namely, the "Gather" utility will
46 search all files in a given path, sort the files in ascending order by 41 search all files in a given path, sort the files in ascending order by
47 using an external file-based merge sort. 42 using an external file-based merge sort.
48   43  
49 Conversely, "Hunt" searchs files by reading lines from the database 44 Conversely, "Hunt" searchs files by reading lines from the database
50 without loading the entire database in RAM or by searching files 45 without loading the entire database in RAM or by searching files
51 again. 46 again.
52   47  
53 The project adheres to the ANSI C standard and Amiga-centric semantics 48 The project adheres to the ANSI C standard and Amiga-centric semantics
54 are compiled conditionally (in case the "___AmigaOS__" macro is 49 are compiled conditionally (in case the "___AmigaOS__" macro is
55 defined at compile time). Otherwise, "Hunt" and "Gather" should run 50 defined at compile time). Otherwise, "Hunt" and "Gather" should run
56 under any platform that benefits from an ANSI C compiler. 51 under any platform that benefits from an ANSI C compiler.
57 52
58 The project is developed from scratch on a real Amiga using StormC. 53 The project is developed from scratch on a real Amiga using StormC.
59   54  
60 -=:[ Usage ]:=- 55 -=:[ Usage ]:=-
61   56  
62 The "Gather" utility is used to index a path. The following command: 57 The "Gather" utility is used to index a path. The following command:
63   -  
64   58  
65 Gather -c RAM: -  
66   59 Gather -c RAM:
67   60  
68 will create a file in the S: directory named "gather.db". "Gather"is 61 will create a file in the S: directory named "gather.db". "Gather"is
69 verbose by default and will show the user what the utility is doing 62 verbose by default and will show the user what the utility is doing
70 but the behaviour can be changed with the "-q" (quiet) flag that will 63 but the behaviour can be changed with the "-q" (quiet) flag that will
71 make "Gather" print only errors. 64 make "Gather" print only errors.
72   65  
73 In order to look for a file, the "Hunt" utility is then invoked with 66 In order to look for a file, the "Hunt" utility is then invoked with
74 an AmigaOS search pattern, for instance, the pattern "#?test#?: -  
75   -  
76   -  
-   67 an AmigaOS search pattern, for instance, the pattern "#?test#?:
77 Hunt #?test#? 68  
78   69 Hunt #?test#?
79   70  
80 "Hunt" will then search the database previously generated by the 71 "Hunt" will then search the database previously generated by the
81 "Gather" utility and will print out all the paths corresponding to the 72 "Gather" utility and will print out all the paths corresponding to the
82 files matching the supplied pattern. 73 files matching the supplied pattern.
83 74
84 In the previous example, in case any of the files previously indexed 75 In the previous example, in case any of the files previously indexed
85 by "Gather" contain the term "test", then the "Hunt" utility will 76 by "Gather" contain the term "test", then the "Hunt" utility will
86 display the path to the file. 77 display the path to the file.
87 78
88 At some point you might decide to add some other path to the search 79 At some point you might decide to add some other path to the search
89 database as well. In that case, "Gather" would be invoked with the 80 database as well. In that case, "Gather" would be invoked with the
90 "-a" option instead of "-c" in order to add the files: 81 "-a" option instead of "-c" in order to add the files:
91 82
92 Gather -a HDH0:Icons/ 83 Gather -a HDH0:Icons/
93 84
94 "Gather" will then index the additional directory and add the new 85 "Gather" will then index the additional directory and add the new
95 files to the database. Adding a path to the index database will 86 files to the database. Adding a path to the index database will
96 require that "Gather" sorts the database again such that after adding 87 require that "Gather" sorts the database again such that after adding
97 the new files, "Gather" will proceed with sorting. 88 the new files, "Gather" will proceed with sorting.
98 89
99 Lastly, the "-r" parameter can be used with "Gather" to remove paths 90 Lastly, the "-r" parameter can be used with "Gather" to remove paths
100 that have been previously indexed. Let's say that you have indexed the 91 that have been previously indexed. Let's say that you have indexed the
101 following paths with "Gather": 92 following paths with "Gather":
102 93
103 RAM: 94 RAM:
104 HDH0:Icons/ 95 HDH0:Icons/
105 96
106 but now you would like to remove the "RAM:" path and all the files 97 but now you would like to remove the "RAM:" path and all the files
107 indexed below that path. In that case, you would issue a "Gather" 98 indexed below that path. In that case, you would issue a "Gather"
108 command with the "-r" parameter: 99 command with the "-r" parameter:
109 100
110 Gather -r RAM: 101 Gather -r RAM:
111 102
112 and "Gather" will remove all files matching the "RAM:" path. Removing 103 and "Gather" will remove all files matching the "RAM:" path. Removing
113 a path with the "-r" parameter does not take a long time compared to 104 a path with the "-r" parameter does not take a long time compared to
114 adding files to the database. 105 adding files to the database.
115   106  
116 -=:[ Gather ]:=- 107 -=:[ Gather ]:=-
117   108  
118 "Gather" requires that one of the following parameters is specified: 109 "Gather" requires that one of the following parameters is specified:
119 * -a (add files to an already existing database), 110 * -a (add files to an already existing database),
120 * -r (remove files from an already existing database), 111 * -r (remove files from an already existing database),
121 * -c (delete the previous database file and create a new database). 112 * -c (delete the previous database file and create a new database).
122   113  
123 The "Gather" utility takes several paths as parameters representing 114 The "Gather" utility takes several paths as parameters representing
124 the paths to be indexed; for example, all the following paths are 115 the paths to be indexed; for example, all the following paths are
125 valid: -  
126   -  
127   -  
128 RAM: -  
-   116 valid:
-   117  
129 DH0:System/ 118 RAM:
130   119 DH0:System/
131   120  
132 When the "Gather" utility runs, a database is created at "S:gather.db" 121 When the "Gather" utility runs, a database is created at "S:gather.db"
133 containing all the found files. 122 containing all the found files.
134 123
135 "Gather" is also happy to work with a different database file other 124 "Gather" is also happy to work with a different database file other
136 than the default database at "S:gather.db" by passing the "-d" 125 than the default database at "S:gather.db" by passing the "-d"
137 parameter when "Gather" is invoked. For instance, the following 126 parameter when "Gather" is invoked. For instance, the following
138 command invocation will create the database file at "T:gather.db" and 127 command invocation will create the database file at "T:gather.db" and
139 index the paths "RAM:" and "HDH0:Icons": 128 index the paths "RAM:" and "HDH0:Icons":
140 129
141 Gather -d T:gather.db RAM: HDH0:Icons 130 Gather -d T:gather.db RAM: HDH0:Icons
142 131
143 Conversely, the "Hunt" utility can then be used to search specified 132 Conversely, the "Hunt" utility can then be used to search specified
144 database files: 133 database files:
145 134
146 Hunt -d T:gather.db #?test?# 135 Hunt -d T:gather.db #?test?#
147 136
148 The previous "Hunt" command will search a database file located at 137 The previous "Hunt" command will search a database file located at
149 "T:gather.db" for all files matching the pattern "#?test?". 138 "T:gather.db" for all files matching the pattern "#?test?".
150   139  
151 -=:[ Hunt ]:=- 140 -=:[ Hunt ]:=-
152   141  
153 "Hunt" is the counterpart to "Gather" and will search a given database 142 "Hunt" is the counterpart to "Gather" and will search a given database
154 generated by the "Gather" utility for files matching the terms passed 143 generated by the "Gather" utility for files matching the terms passed
155 to "Hunt" on the command line. 144 to "Hunt" on the command line.
156   145  
157 For instance: 146 For instance:
158   147  
159 Hunt #?test#? 148 Hunt #?test#?
160   149  
161 will search all files in the "Gather" database "S:gather.db" for the 150 will search all files in the "Gather" database "S:gather.db" for the
162 term "test". If any file within the database partially matches the 151 term "test". If any file within the database partially matches the
163 term "test", then "Hunt" will display the path on the command line. 152 term "test", then "Hunt" will display the path on the command line.
164 153
165 "Hunt" uses AmigaOS pattern for matching the file names on AmigaOS. 154 "Hunt" uses AmigaOS pattern for matching the file names on AmigaOS.
166   155  
167 -=:[ Notes ]:=- 156 -=:[ Notes ]:=-
168   157  
169 * The "Gather" utility will be slow and that is the intended 158 * The "Gather" utility will be slow and that is the intended
170 behaviour: slow indexing with "Gather", fast searching with "Hunt". 159 behaviour: slow indexing with "Gather", fast searching with "Hunt".
171 160
172 * Temporary files might end up created in the same location where the 161 * Temporary files might end up created in the same location where the
173 "Gather" utility is invoked. Traditionally the temporary directory 162 "Gather" utility is invoked. Traditionally the temporary directory
174 on AmigaOS is mainted in RAM but "Gather" cannot use RAM since it 163 on AmigaOS is mainted in RAM but "Gather" cannot use RAM since it
175 intends to index very large hierarchies. Fortunately, "Gather"will 164 intends to index very large hierarchies. Fortunately, "Gather"will
176 delete the temporary files once "Gather" is done indexing. 165 delete the temporary files once "Gather" is done indexing.
177 166
178 Nevertheless, in case you intend to index a large filesystem 167 Nevertheless, in case you intend to index a large filesystem
179 hierarchy please make sure that you invoke "Gather" from a directory 168 hierarchy please make sure that you invoke "Gather" from a directory
180 that is able to hold large temporary files. 169 that is able to hold large temporary files.
181   170  
182 * The output of the "Hunt" utility can be combined with the pipe 171 * The output of the "Hunt" utility can be combined with the pipe
183 operator (in newer AmigaOS releases) or the PIPE: handler on older 172 operator (in newer AmigaOS releases) or the PIPE: handler on older
184 AmigaOS releases in order to to perform some action on the found 173 AmigaOS releases in order to to perform some action on the found
185 files. For example, using Thomas Radtke's "from" utility located at: 174 files. For example, using Thomas Radtke's "from" utility located at:
186 175
187 http://aminet.net/package/util/batch/from 176 http://aminet.net/package/util/batch/from
188 177
189 and the Workbench 3.2 "MD5Sum" utility, you could print out the MD5 178 and the Workbench 3.2 "MD5Sum" utility, you could print out the MD5
190 hashes of all files indexed by "Gather" ending in "#?.library: 179 hashes of all files indexed by "Gather" ending in "#?.library:
191 180
192 Hunt #?.library | from - md5sum $1 181 Hunt #?.library | from - md5sum $1
193 182
194 Or you could generate a list of versions of all libraries indexed 183 Or you could generate a list of versions of all libraries indexed
195 with the "Gahter" utility: 184 with the "Gahter" utility:
196 185
197 Hunt #?.library | from - version $1 186 Hunt #?.library | from - version $1
198 187
199 -=:[ Source ]:=- 188 -=:[ Source ]:=-
200   189  
201 The project is open sourced and licensed under MIT. The source code 190 The project is open sourced and licensed under MIT. The source code
202 is included in the AmiNET release or can be checked out via subversion 191 is included in the AmiNET release or can be checked out via subversion
203 from the Wizardry and Steamworks repository: 192 from the Wizardry and Steamworks repository:
204 193
205 svn co http://svn.grimore.org/HuntnGather 194 svn co http://svn.grimore.org/HuntnGather
206 195
207 StormC was used as the developer environment. 196 StormC was used as the developer environment.
208 197
209 -=:[ Mentions ]:=- 198 -=:[ Mentions ]:=-
210   199  
211 The code includes a shim for "getopt" in order to process command line 200 The code includes a shim for "getopt" in order to process command line
212 parameters on AmigaOS just like one would on a POSIX sytem. The shim 201 parameters on AmigaOS just like one would on a POSIX sytem. The shim
213 is created by Daniel J. Barrett, barrett@cs.umass.edu and is 202 is created by Daniel J. Barrett, barrett@cs.umass.edu and is
214 available on AmiNET: 203 available on AmiNET:
215 204
216 http://aminet.net/package/dev/misc/GetOpt-1.3 205 http://aminet.net/package/dev/misc/GetOpt-1.3
217 206
218 -=:[ Contact ]:=- 207 -=:[ Contact ]:=-
219   208  
220 E-Mail(tor): 209 E-Mail(tor):
221   210  
222 office@3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion 211 office@3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion
223   212  
224 Website: https://grimore.org/amiga/hunt_and_gather 213 Website: https://grimore.org/amiga/hunt_and_gather
225 Website(tor): 214 Website(tor):
226   215  
227 http://3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion/ 216 http://3wymlmcsvxiaqzmbepsdawqpk6o2qsk65jhms72qqjulk5u4bgmvs3qd.onion/
228   217  
229 Discord: https://discord.gg/k9kyDsa 218 Discord: https://discord.gg/k9kyDsa
230 219
231   220