Mark, can one execute this thread dump command
bin/nifi.sh dump thread-dump.txt
while nifi continues to try and start up, or in cases where nifi is actively running? Or does it result in termination of the running nifi instance?
Here is what I tried and what finally worked last night:
1. I deleted all prov.gz files from my provenance_repository
2. when I still failed on start up, I deleted all under my provenance_repository
3. still failed, so I ensured nifi was entirely shut down and then I deleted all swap files from flowfile_repository/swap
4. still failed. We continued to get GC out of memory errors in the nifi-app.log. We added memory to our nifi server and bumped up -Xmx to 24576m
If anyone else reads this let me emphasize that I am no nifi sys admin expert, and these steps are not without drawbacks re: provenance and flow. I leave it to the experts to offer better guidance. All I can offer in my defense is this: I tried a sequence of steps in progression from low cost to higher cost. When those didn't work, I fell back on brute force, waived a white flag in surrender, and threw more memory at the problem to *hopefully* allow it to get through start up. I had to get our production workflows up and running again.
Thank you again to all for your advice. I'd welcome feedback. -Jim