Skip to content

Commit eaf7e31

Browse files
authored
Search chain and replace plugin (#115)
* Some initial work on search sequences/chains + unique search id * Flesh out edit search chain dialog * Cleanup - Simplify editor dialog setup with gridbagsizer - Switch to listctrl (wxFormBuilder is freezing up with ListBox on OS X) - Clean up some deprecations. * Localize edit chain dialog * Dialog to manage saved chains (incomplete) * Finish up chain management * Go back to ListBox with a workaround to fix tabbing * Actual ability to chain multiple saved regex patterns * Update tests * Fix lint * Localize search chain dialog * Show literal status in save search dialog * Perform validation on chains before search Chains should check if all chained searches exist, and that the search compiles. Also clean up related code and make it more compact. We were also always setting regex version flags which doesn’t make much if regex is disabled. * Make chain wxCombo a wxChoice and remove divider * When chains is enabled, change default focus * Update comment * Lint * Rumcore doesn't decode strings directly Rumcore will no longer decode strings to simplify the internal code. If someone is using the API, they should decode it to Unicode or pass it as a byte string for binary. Only files will be decoded/encoded on input/output. Technically everything is included so a user could detect encoding. But as the API is not documented right now, I won’t worry about it, but in the future, maybe we can expose an easy detect function that does everything the that happens for files. * Fix replace related issues * Revert: ensure we always return 1 record Forgot that I require the API to always return 1 file record (or buffer record in the case of a buffer replace). Revert earlier change. and ensure all loops abort. * Fix search abort not working * Update changelog * Update dictionary with spelling * Replace search button text when aborting * Adjust putty patterns to work locally * Git putty to work locally and on Travis * Use existing "search for" object for chains * Add replace plugin support + fixes Add replace plugin support and fix issue where drop down list didn’t hide the limit panel. Also don’t display replace message with aborting replace. Also run flake8 in tox differently so we can avoid weird flake8-putty patterns. * Add .codecov.yml * Don't let plugin refs get garbage collected We had to pass the whole module in to keep plugin function working because the module was getting garbage collected. We don’t want to store the plugin in sys.module, so track in a local structure and delete when we are done with it. * Don't resize panel on search * Fix path check * Add format to settings and history file * Visual tweaks - Adjust layouts after chainging label names - Don't adjust height when only graying out controls - Disable and enable "replace plugin" directory picker when applicable. * Handle special cases - Disable replace plugin checkbox and save search button when chain mode is enabled. - When loading search, run event for chain mode, regex search, and/or replace plugin when toggling their respective checkbox manually. * Setup argument to avoid redirect * Add support for replace plugin in tester * Literal and Unicode changes Literal will allow the Unicode flag and will allow full case in Regex V0. Tester will now allow testing with literal to allow pairing literal with replace plugin. * Patch gui and ensure literal flag is handled in tester * Add some missing localization * Update some documentation - Add recent changes to changelog - Add missing statement about POSIX flag * Fix GUI inconsistencies Visual inconsistencies in regard to text box size relative to siblings, alignment of labels, etc. Also, content text box in tester dialog will now allow entering tab characters on all platforms instead of navigating to next control. * Tweak date and time control On each OS, the date and time object is a little different with how it fits. Try and ensure the date and time control take the full height of the row they are on. Format the time spinner to center better and remove the gap to give a clear indication it is attached to the time control. Hopefully this look sfine on macOS and Linux as well. * Tweak time ctrl * Don't expand spinner (distorts on Ubuntu) * Fix deprecation in messages lib * Update saved project * Some doc updates * Fix spelling failure * Add optional hint to add search diag Add a hint to say comment is optional opposed to having it in the label. Also tweak some of the add search labels to be shorter. * Rumcore tweaks Denote RummageFileContent as a protected class. Move Search object with Replace plugin object. Add on_init method to Replaceplugin object. * Update docs with more info on new features * Don't list regex as optional (just optional to use) * Fix spelling * Update docs New images, some re-wording and update theme CSS. * Bump rev
1 parent 7e87a73 commit eaf7e31

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+11878
-6344
lines changed

.codecov.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
comment: false
2+
3+
coverage:
4+
status:
5+
patch: false

.dictionary

+5
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
API
12
Automator
23
Backrefs
34
Changelog
@@ -22,7 +23,10 @@ multi
2223
NONINFRINGEMENT
2324
OS's
2425
OSD
26+
Plugin
2527
plugin
28+
plugins
29+
POSIX
2630
PyMdown
2731
Regex
2832
regex
@@ -35,5 +39,6 @@ symlink
3539
taskbar
3640
timestamp
3741
Tox
42+
tuple
3843
UI
3944
wxPython

.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,5 @@ pyinstaller
4848
site/*
4949

5050
.cache
51+
52+
*.patch

README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,11 @@ Rummage
1010

1111
Rummage is a cross platform search and replace tool written in Python. Rummage crawls directories and searches for specified patterns (either regular expression or literal) and can optionally replace those targets with desired text.
1212

13-
Rummage was inspired by the Windows tool [grepWin][grepwin]. At the time, it was difficult to find a decent GUI search and replace tool macOS and Linux. Rummage was created to fill that void and is available on Linux, macOS, and Windows.
13+
Rummage was inspired by the Windows tool [grepWin](http://stefanstools.sourceforge.net/grepWin.html). At the time, it was difficult to find a decent GUI search and replace tool macOS and Linux. Rummage was created to fill that void and is available on Linux, macOS, and Windows.
1414

15-
Rummage is written in Python and is currently available on Python 2.7 and Python 3.4+. Rummage by default uses the Python's Re regular expression engine, but you can also use the fantastic, feature rich [Regex][regex] search engine and do interesting things like fuzzy searching and more.
15+
Rummage is written in Python and is currently available on Python 2.7 and Python 3.4+. Rummage by default uses the Python's Re regular expression engine, but you can also use the fantastic, feature rich [Regex](https://pypi.python.org/pypi/regex) search engine and do interesting things like fuzzy searching and more.
1616

17-
When specifying a specific file encoding, Rummage is quick enough to search large projects easily. Optionally you can let Rummage detect encoding, but because the encoding detection is done by the pure Python package [Chardet][chardet], it will run significantly slower. Encoding detection is the biggest bottleneck. This shouldn't be a problem in small projects, but it will become quite noticeable in large projects.
17+
When specifying a specific file encoding, Rummage is quick enough to search large projects easily. Optionally you can let Rummage detect encoding, but because the encoding detection is done by the pure Python package [Chardet](https://pypi.python.org/pypi/chardet, it will run significantly slower. Encoding detection is the biggest bottleneck. This shouldn't be a problem in small projects, but it will become quite noticeable in large projects.
1818

1919
## Screenshots
2020

docs/src/markdown/changelog.md

+22
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,27 @@
11
# Changelog
22

3+
## 2.1.0
4+
5+
> Jul X, 2017
6+
7+
- **NEW**: Add new chained search feature.
8+
- **NEW**: Add new replace plugin support.
9+
- **NEW**: Restructure internal API to support chained search.
10+
- **NEW**: Saved searches will now require a unique name and an optional comment. Old legacy saves will be converted on first access. The old legacy name will be the comment, and a unique name will be generated from the comment.
11+
- **NEW**: Saved search names and comments can be edited from the "Load Search" panel.
12+
- **NEW**: Internal API no longer will guess and decode strings, only files. It is expected that the caller handles encoding of string buffers. A Unicode buffer will be searched as usual, and a binary string buffer will be treated as binary.
13+
- **NEW**: Tester dialog will now process literal searches as well.
14+
- **NEW**: Literal searches will now utilize the Unicode related flags. This is particularly notable if using the Regex module and wanting to have full case-folding applied in case-insensitive matches.
15+
- **FIX**: Visual inconsistencies in regard to text box size relative to siblings, alignment of labels, etc.
16+
- **FIX**: Content text box in tester dialog will now allow entering tab characters on all platforms instead of navigating to next control.
17+
- **FIX**: Fix wxPython deprecation noise in the console.
18+
- **FIX**: Fix some binary related replace issues.
19+
- **FIX**: Fix search not aborting.
20+
- **FIX**: Fix issue where selecting file from drop down list didn't hide limit panel.
21+
- **FIX**: Don't display replace message when aborting replace.
22+
- **FIX**: Fix limit panel hide logic so it doesn't show a the hidden panel on search.
23+
- **FIX**: POSIX flag not generating refresh in tester dialog.
24+
325
## 2.0.5
426

527
> Jul 2, 2017
12.3 KB
Loading
65.1 KB
Loading

docs/src/markdown/images/chains.png

47.7 KB
Loading
-868 Bytes
Loading
18.3 KB
Loading
-51 Bytes
Loading
14.6 KB
Loading
27.1 KB
Loading
16.4 KB
Loading
7.1 KB
Loading
13.8 KB
Loading
-11.7 KB
Loading
12.6 KB
Loading

docs/src/markdown/installation.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,13 @@
44

55
Rummage has a few requirements when installing. These will all be taken care of when installing via `pip`.
66

7-
Name | Required | Details
8-
------------------------------ | -------- | -------
9-
[`backrefs` 1.0.1+][backrefs] | Yes | Used to extend the `re` or `regex` regular expression engine with additional back references.
10-
[`gntp`][gntp] | Yes | Used to send notifications to Growl via the the Growl Notification Transport Protocol for all platforms (macOS, Windows, and Linux).
11-
[`chardet`\ 3.0.4+][chardet] | Yes | Used for file encoding guessing when an encoding is not specified.
12-
[`wxPython`\ 4.0.0a3+][wxpython] | Yes | The new wxPython 4.0.0 is required for for Rummage to run in Python 2 and Python 3. Classic wxPython support has unfortunately be dropped.
13-
[`regex`\ 2015.07.19+][regex] | No | **regex** is completely optional, but it will be installed automatically if using `pip`. It is a great regular expression engine that adds some nice features such as fuzzy searching, nested char sets, better Unicode support, and more.
7+
Name | Details
8+
-------------------------------- | -------
9+
[`backrefs` 1.0.1+][backrefs] | Used to extend the `re` or `regex` regular expression engine with additional back references.
10+
[`gntp`][gntp] | Used to send notifications to Growl via the the Growl Notification Transport Protocol for all platforms (macOS, Windows, and Linux).
11+
[`chardet`\ 3.0.4+][chardet] | Used for file encoding guessing when an encoding is not specified.
12+
[`wxPython`\ 4.0.0a3+][wxpython] | The new wxPython 4.0.0 is required for for Rummage to run in Python 2 and Python 3. Classic wxPython support has unfortunately be dropped.
13+
[`regex`\ 2015.07.19+][regex] | **regex** is usage is completely optional, but it is included for those who wish to use it. Regex is a great regular expression engine that adds some nice features such as fuzzy searching, nested char sets, better Unicode support, and more.
1414

1515
!!! warning "Linux Prerequisites"
1616
In traditional Linux fashion, there is a little extra work that needs to be done prior to installing. Linux requires some prerequisites so that it can build wxPython during installation.

docs/src/markdown/usage.md

+139-2
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,8 @@ Toggle | Description
8888
--------------------------- | -----------
8989
Best\ fuzzy\ match | If performing a fuzzy match, the *best* fuzzy match will be used.
9090
Improve\ fuzzy\ fit | Makes fuzzy matching attempt to improve the fit of the next match that it finds.
91-
Unicode\ word\ breaks | Will use proper Unicode word breaks when Unicode is enabled. This differs from Re's default.
91+
Unicode\ word\ breaks | Will use proper Unicode word breaks and line separators when Unicode is enabled. See Regex documentation for more info.
92+
Use\ POSIX\ matching | Use the POSIX standard for regular expression, which is to return the leftmost longest match.
9293
Search\ backwards | Search backwards. The result of a reverse search is not necessarily the reverse of a forward search.
9394
Format\ style\ replacements | Replace pattern will use [Python's string replace format][format-string] for replace. `#!python "{1[-1]} {1[-2]} {1[-3]}"` etc.
9495
Full\ case-folding | Use full case folding. For Regex `V0` only as it is enabled by default for `V1`.
@@ -103,12 +104,14 @@ Boolean\ match | Will check each file up until the first match and will
103104
Count\ only | Will just count the number of matches in the file and will not display line context information. This has no effect when applying replaces.
104105
Create\ backups | On replace, files with matches will be backed up before applying the replacements; backup files will have the `.rum-bak` extension.
105106
Force\ <encoding> | Forces all files to be opened with the specified encoding opposed to trying to detect the encoding. Encoding is hard and slow, so this is the preferred method for fast searches. On failure, binary will be used instead.
107+
Use\ chain\ search | Puts Rummage into ["search chain" mode](#search-chains). When in "search chain" mode, rummage will only use saved search chains for search and replace.
108+
Use\ replace\ plugin | When enabled, Rummage will use a [replace plugin](#replace-plugins) instead of a replace pattern in order to do more advanced replaces.
106109

107110
### Regular Expression Tester
108111

109112
![Regex Tester](/images/regex_tester.png)
110113

111-
Rummage comes with a simple regular expression tester. It has a simple multi-line text box for content to search, and another multi-line box that will show the final results after the find and replace. Below that you will find two text input boxes for the find pattern and the replace pattern. Lastly, all related regular expression flag toggles will be found under the patterns.
114+
Rummage comes with a simple regular expression tester (but you can also test literal patterns if desired). It has a simple multi-line text box to place content to search, and another multi-line box that will show the final results after the find and replace are applied. Below that you will find two text input boxes for the find pattern and the replace pattern. Lastly, all search and replace flag toggles are found under the patterns.
112115

113116
To use the tester, simply enter the content to search, set your desired toggles, and input your find and replace pattern. As you change your pattern or change your toggles, matches will be updated and highlighted, and the result box will be updated.
114117

@@ -120,12 +123,146 @@ Regular expressions can be very complex, and sometimes you might want to save th
120123

121124
When you have a pattern configured that you want to save, simply click the `Save Search` button, and a dialog will pop up asking you to name the search. When done, click the `Save` button on the dialog and your search patterns and toggles will be saved.
122125

126+
You'll notice that there are two input boxes. The first requires a unique name (only word characters, underscores, and hyphens are allowed). The second is an optional comment in case you wish to elaborate on what the pattern is for.
127+
128+
Underneath the inputs will be the actual search settings being saved. All of the search settings will be in read only controls.
129+
123130
![Save Search](/images/save_search.png)
124131

125132
To load a pattern that was saved previously, click the `Load Search` button. You will be presented with a dialog showing all your saved searches. Highlight the pattern you want to load and click the `Load` button. Your pattern and toggles will be populated in the main dialog.
126133

134+
If you wish to edit the name or comment of a search, you can double click the entry or click the "Edit" button.
135+
127136
![Load Search](/images/load_search.png)
128137

138+
### Search Chains
139+
140+
There are times you may have a task that requires you to do multiple find and replaces that are all related, but are too difficult to represent as a single find and replace. This is where search chains can be helpful.
141+
142+
Search chains are essentially a sequence of multiple [saved search and replace patterns](#saving-and-loading-regular-expressions). You can create a search chain by clicking the `Search Chains` button which will bring up the search change manager.
143+
144+
![Chain Button](/images/chain_button.png)
145+
146+
Here you can create or delete search chains.
147+
148+
![Search Chains](/images/chains.png)
149+
150+
To use search chains you must put Rummage in "search chain" mode by selecting the check box named `Use search chains` in the main window. When "search chain" mode is enabled, all controls that don't apply to search chains will be disabled, and the search box will be replaced with a drop down for selecting created chains. When a search is performed, Rummage will iterate over each file with all the saved searches in the chain.
151+
152+
![Chain Select](/images/chain_mode.png)
153+
154+
### Replace plugins
155+
156+
Regular expressions are great, but some times regular expressions aren't enough. If you are dealing with a replace task that requires logic that cannot be represented in a simple replace pattern, you can create a "replace plugin".
157+
158+
Simply create a Python script with a Replace class derived from the `ReplacePlugin` class found in `rumcore` at: `#!py from rummage.rummage import rumcore`. The plugin file must include a function called `get_replace` that returns the needed class.
159+
160+
```py
161+
class ReplacePlugin(object):
162+
"""Rummage replace plugin."""
163+
164+
def __init__(self, file_info, flags):
165+
"""Initialize."""
166+
167+
self.file_info = file_info
168+
self.flags = flags
169+
self.on_init()
170+
171+
def on_init(self):
172+
"""Override this function to add initialization setup."""
173+
174+
def get_flags(self):
175+
"""Get flags."""
176+
177+
return self.flags
178+
179+
def get_file_name(self):
180+
"""Get file name."""
181+
182+
return self.file_info.name
183+
184+
def is_binary(self):
185+
"""Is a binary search."""
186+
187+
return self.file_info.encoding.encode == 'bin'
188+
189+
def is_literal(self):
190+
"""Is a literal search."""
191+
192+
return self.flags & LITERAL
193+
194+
def replace(self, m):
195+
"""Make replacement."""
196+
197+
return m.group(0)
198+
```
199+
200+
The `file_info` property is a named tuple providing information about the current file such as name, size, creation date, etc.
201+
202+
```py
203+
class FileInfoRecord(namedtuple('FileInfoRecord', ['id', 'name', 'size', 'modified', 'created', 'encoding'])):
204+
"""A record for tracking file info."""
205+
```
206+
207+
The `flags` property seen above contains only Rummage search related flags (the flags are abstracted at this level and are converted to the appropriate regular expression flags later).
208+
209+
```py
210+
# Common regular expression flags (re|regex)
211+
IGNORECASE = 0x1 # (?i)
212+
DOTALL = 0x2 # (?s)
213+
MULTILINE = 0x4 # (?m)
214+
UNICODE = 0x8 # (?u)
215+
216+
# Regex module flags
217+
ASCII = 0x10 # (?a)
218+
FULLCASE = 0x20 # (?f)
219+
WORD = 0x40 # (?w)
220+
BESTMATCH = 0x80 # (?b)
221+
ENHANCEMATCH = 0x100 # (?e)
222+
REVERSE = 0x200 # (?r)
223+
VERSION0 = 0x400 # (?V0)
224+
VERSION1 = 0x800 # (?V1)
225+
FORMATREPLACE = 0x1000 # Use {1} for groups in replace
226+
POSIX = 0x2000 # (?p)
227+
228+
# Rumcore search related flags
229+
LITERAL = 0x10000 # Literal search
230+
```
231+
232+
To use replace plugins, simply check the `Use plugin replace` check box in the main dialog.
233+
234+
![Enable Replace Plugin](/images/plugin_toggle.png)
235+
236+
The main dialog's `Replace with` text box will become the `Replace plugin` text box with an associated file picker. Here you can point to your replace plugin file.
237+
238+
![Enable Replace Plugin](/images/plugin_input.png)
239+
240+
241+
??? settings "Example Plugin"
242+
In the example below, we have a replace plugin that replaces the search result with the name of the file. It is assumed this is not a binary replace.
243+
244+
```py
245+
from __future__ import unicode_literals
246+
from rummage.rummage import rumcore
247+
import os
248+
249+
250+
class TestReplace(rumcore.ReplacePlugin):
251+
"""Replace object."""
252+
253+
def replace(self, m):
254+
"""Replace method."""
255+
256+
name = os.path.basename(self.get_file_name())
257+
return name
258+
259+
260+
def get_replace():
261+
"""Get the replace object."""
262+
263+
return TestReplace
264+
```
265+
129266
## Limit Search Panel
130267

131268
![Limit Search Panel](/images/limit_search_panel.png)

docs/theme/extra-6068b97e47.css

-1
This file was deleted.

docs/theme/extra-9b4c72abee.css

+1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)