So-
Since Splunk version 8.0 was dropped earlier today, I decided to upgrade my DEV environment to start playing with the latest additions and changes.
Links and References
The problem
However- after running the upgrade, I quickly ran into this issue, found in web_service.log, preventing SplunkWeb from starting properly.
web_service.log
2019-10-22 10:45:25,928 INFO [5daf2415ae7f3323106710] __init__:174 - Using default logging config file: /opt/splunk/etc/log.cfg
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk level=INFO
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver level=INFO
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.controllers level=INFO
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.lib level=WARN
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.pdfgen level=INFO
2019-10-22 10:45:25,935 INFO [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.archiver_restoration level=INFO
2019-10-22 10:45:26,259 ERROR [5daf2415ae7f3323106710] root:769 - Unable to start splunkweb
2019-10-22 10:45:26,259 ERROR [5daf2415ae7f3323106710] root:770 - No module named 'UserDict'
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in <module>
from splunk.appserver.mrsparkle.controllers.top import TopController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in <module>
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 25, in <module>
from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 22, in <module>
from splunk.appserver.mrsparkle.lib import module
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 465, in <module>
moduleMapper = ModuleMapper()
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 83, in __init__
self.installedModules = self.getInstalledModules()
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 28, in helper
return f(*a, **kw)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 448, in getInstalledModules
mods = self.getModuleList(root)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 37, in helper
return f(*a, **kw)
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 223, in getModuleList
mod = __import__(modname)
File "/opt/splunk/etc/apps/search_activity/appserver/modules/D3DynamicTree/D3DynamicTree.py", line 5, in <module>
import UserDict
ModuleNotFoundError: No module named 'UserDict'
So, the first troubleshooting step I will perform- is to reinstall the .rpm, just in case.
sudo rpm --reinstall ~/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86_64.rpm splunk start splunk status
Of course- that did not resolve the issue. The biggest issue which stands out in the logs, is “ModuleNotFoundError: No module named ‘UserDict'”
So- since I see references to the search_activity app, I removed the app and restarted Splunk.
Voila, Problem resolved. Perhaps somebody will find some use in this article.
